Bug#846396: screen: display corruption with bce due to wide character

[Vincent Lefevre]

On 2017-06-29 00:00:59 +0200, Axel Beckert wrote:
> Your reasoning sounds valid, so I'll close this bug report with
> 4.6.0-1 (which I'm currently preparing). Please reopen if you still
> can reproduce the issue.

I've installed screen 4.6.0-1 from experimental, and I confirm
that this bug is fixed. Thanks.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#846396: screen: display corruption with bce due to wide character

[Axel Beckert]

Control: forwarded -1 https://savannah.gnu.org/bugs/index.php?50044
Control: tag -1 + pending

Hi Vincent,

Vincent Lefevre wrote:
> According to a user who ran into the same issue, the cause is the
> obsolete Unicode lookup table:
> 
>   https://savannah.gnu.org/bugs/index.php?50044

I see. Wouldn't have guessed that so thanks for that hint.

> I didn't have the time to test the patch. But GNU Screen 4.6.0, which
> has just been released, has:
> 
>   * Update Unicode wide tables to 9.0
> 
> Thus, if this is really the same issue, this bug should now be fixed
> upstream.

Your reasoning sounds valid, so I'll close this bug report with
4.6.0-1 (which I'm currently preparing). Please reopen if you still
can reproduce the issue.

Regards, Axel
-- 
 ,''`.  |  Axel Beckert , http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-|  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE

Bug#846396: screen: display corruption with bce due to wide character

[Vincent Lefevre]

Control: tags -1 - moreinfo + fixed-upstream
Control: found -1 4.5.1-4

According to a user who ran into the same issue, the cause is the
obsolete Unicode lookup table:

  https://savannah.gnu.org/bugs/index.php?50044

I didn't have the time to test the patch. But GNU Screen 4.6.0, which
has just been released, has:

  * Update Unicode wide tables to 9.0

Thus, if this is really the same issue, this bug should now be fixed
upstream.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#846396: screen: display corruption with bce due to wide character

[Vincent Lefevre]

This is getting worse and worse, as more and more messages (not spam)
use wide characters. I've attached a small part of the index obtained
just after Page Up - Page Down.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#846396: screen: display corruption with bce due to wide character

[Vincent Lefevre]

Hi,

On 2016-12-01 02:00:59 +0100, Axel Beckert wrote:
> JFTR: IMHO this issue has not "a major effect on the usability of
> the package" (c.f. https://www.debian.org/Bugs/Developer#severities),
> but since I'm rather sick of playing severity ping-pong, I'll leave it
> as is, at least for now.

By silently giving wrong information to the user, it can cause
data loss (and it really did with a similar bug also yield a
content shift). So, it should actually be "grave".

> Vincent Lefevre wrote:
> > > Nevertheless, there is an even partially persistent workaround: Press
> > > Ctrl-L inside mutt and the problem vanishes even until after the next
> > > detaching and reattaching. Only restarting mutt inside the screen
> > > session and then detaching and reattaching again causes it to show up
> > > again. Hence downgrading the severity.
> > 
> > This is incorrect.
> 
> Well, then we either see different issues or this is even more a
> corner case which requires more variables to have the right values
> (font availability maybe? init system maybe? sysvinit here.) than we
> know so far.

Perhaps font availability. Or terminfo settings. I doubt concerning
the init system.

Also, you need more than the simple test case to produce corruption
without quitting the current screen window.

> > With my incoming mailbox, the bug is reproducible
> > after doing Ctrl-L. For instance, Page Up + Page Down yields display
> > issues, e.g.
> > 
> >2931 N   2016-11-30 ...
> >2932 2016-11-30 ...
> >9337 N + 2016-11-30 ...
> >2934 N + 2016-11-30 ...
> > 
> > Also, switching screen [obvious stuff deleted] also yields upward
> > shift.
> 
> Not here either. Ctrl-L works fine here permanently until I quit
> mutt — in both, urxvt and gnome-terminal.

Even when creating a new window and switching to the first one?

> So it's not even that the symtomps are slightly different inside an
> urxvt since we see even different behaviour inside gnome-terminal. And
> as mentioned above, I'm not even able to reproduce these issues in
> uxterm at all (as I don't see the wide character there despite it
> should be possible as it is in urxvt.

Actually that's strange that on your machine, uxterm yields a
different behavior than the other terminals. It seems a local
issue. It seems that your uxterm doesn't support wide characters.

Note that I get a similar behavior with xterm, uxterm, urxvt,
GNOME Terminal and aterm. The behavior is OK with mlterm, but
it doesn't support wide characters, i.e. the U+1F534 character
takes only 1 column; but this mlterm behavior yields other display
problems in other contexts, like with zsh and Mutt (where wrapping
and clipping isn't done at the right place) because applications
assume 2 columns.

> So I'm no more sure that we actually look at the same issue.

I think that's the same, but this can more or less yield
undeterministic behavior depending on the context. And you
need a terminal that supports wide characters.

> Do you by chance know which font you use in uxterm?

AFAIK, it is "fixed". It seems to be the same as "6x13".
At least, the problem is reproducible with it. Well, this
may be more complex than that, since xterm uses a different
font for wide text:

  -fw font
   This option specifies the font to be used for displaying wide
   text.  By default, it will attempt to use a font twice as wide
   as the font that will be used to draw normal text.  If no
   double-width font is found, it will improvise, by stretching
   the normal font.  This corresponds to the wideFont resource.

> > One major problem is that due to the shift, the arrow can be on the
> > wrong message (the arrow is at the right place, but the contents had
> > been shifted upwards), and if I type 'd' to delete the message, the
> > D mark appears on the wrong message for the same reason.
> 
> It would be an issue if you can't fix that with Ctrl-L. But even if it
> comes back more often for you than for me, Ctrl-L seems to help at
> least immediately as I understand your mail.

That's too unpredictable. I recall that the display corruption is
not always visible at first sight, and it may be too late when it
is noticeable. So, one would need an automatic Ctrl-L after each
character sent to the terminal (well, that's worse as it hides
Mutt error messages).

> E.g. the attached mbox with a single spam message causes the issues
> you described for me (even without a wide character as far as I can
> see), but only with screen in Wheezy and no more with screen 4.4.0-6
> from Sid. So I'm actually a little bit surprised that still see this
> issues in Sid with the same intensity as I see them only on Wheezy.

Note that Mutt has changed in sid after I reported bugs that
yielded display issues in GNU screen in particular (but also
even without screen, though these were mainly wrapping / clipping
issues). The cause was characters with wcwidth = 0, such as LTR
and RTL marks and some other special "characters". But thi

Bug#846396: screen: display corruption with bce due to wide character

[Axel Beckert]

Control: tag -1 - confirmed + moreinfo

Hi Vincent,

JFTR: IMHO this issue has not "a major effect on the usability of
the package" (c.f. https://www.debian.org/Bugs/Developer#severities),
but since I'm rather sick of playing severity ping-pong, I'll leave it
as is, at least for now.

Vincent Lefevre wrote:
> > Nevertheless, there is an even partially persistent workaround: Press
> > Ctrl-L inside mutt and the problem vanishes even until after the next
> > detaching and reattaching. Only restarting mutt inside the screen
> > session and then detaching and reattaching again causes it to show up
> > again. Hence downgrading the severity.
> 
> This is incorrect.

Well, then we either see different issues or this is even more a
corner case which requires more variables to have the right values
(font availability maybe? init system maybe? sysvinit here.) than we
know so far.

> With my incoming mailbox, the bug is reproducible
> after doing Ctrl-L. For instance, Page Up + Page Down yields display
> issues, e.g.
> 
>2931 N   2016-11-30 ...
>2932 2016-11-30 ...
>9337 N + 2016-11-30 ...
>2934 N + 2016-11-30 ...
> 
> Also, switching screen [obvious stuff deleted] also yields upward
> shift.

Not here either. Ctrl-L works fine here permanently until I quit
mutt — in both, urxvt and gnome-terminal.

So it's not even that the symtomps are slightly different inside an
urxvt since we see even different behaviour inside gnome-terminal. And
as mentioned above, I'm not even able to reproduce these issues in
uxterm at all (as I don't see the wide character there despite it
should be possible as it is in urxvt.

So I'm no more sure that we actually look at the same issue.

Do you by chance know which font you use in uxterm?

> One major problem is that due to the shift, the arrow can be on the
> wrong message (the arrow is at the right place, but the contents had
> been shifted upwards), and if I type 'd' to delete the message, the
> D mark appears on the wrong message for the same reason.

It would be an issue if you can't fix that with Ctrl-L. But even if it
comes back more often for you than for me, Ctrl-L seems to help at
least immediately as I understand your mail.

> Note that it may not be obvious that the contents have shifted
> upwards, in particular because in my normal config, the help line is
> not shown.

At least for me it always was obvious in the past (e.g. on Squeeze and
Wheezy), especially with mutt and wide characters. And there were
quite some of these bugs in the past…

Especially I'm very used to the fact that screen window switching to
mutt while having displayed the mail index with mails with subjects
containing wide character (usually spam in my case) yields such a
corruption as you describe it, even with uxterm and Ctrl-L only
helping once and not persistently — but I only experience these issues
nowadays with screen from Debian 7 Wheezy — and comparing Sid with
Wheezy on these kind of issues, Sid got much better for me. I also
tried a Screen session on Sid inside a gnome-terminal (as well as
uxterm), then SSHing into a Wheezy box and opening mutt with my spam
archive — which usually triggers such bugs in the screen version in
Wheezy, but screen seems to have improved at lot since then.

E.g. the attached mbox with a single spam message causes the issues
you described for me (even without a wide character as far as I can
see), but only with screen in Wheezy and no more with screen 4.4.0-6
from Sid. So I'm actually a little bit surprised that still see this
issues in Sid with the same intensity as I see them only on Wheezy.

Regards, Axel
-- 
 ,''`.  |  Axel Beckert , http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-|  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE


spam-which-causes-display-corruption-with-mutt-inside-screen-on-wheezy-but-not-sid.xz
Description: Binary data

Bug#846396: screen: display corruption with bce due to wide character

[Vincent Lefevre]

Control: severity -1 important

On 2016-11-30 23:44:20 +0100, Axel Beckert wrote:
> I was able to reproduce this issue inside an urxvt, but not inside an
> uxterm — there I just got a question mark inside a filled circle
> instead of a wide character.

Strange, I can reproduce it with uxterm (note that xterm with UTF-8
locales is my usual terminal) and GNOME Terminal.

> And if you type cursor down, you get two arrows, one below the line
> with no. 2, i.e. where the line should be, but isn't.

I get a single arrow, below the line no 2.

> Nevertheless, there is an even partially persistent workaround: Press
> Ctrl-L inside mutt and the problem vanishes even until after the next
> detaching and reattaching. Only restarting mutt inside the screen
> session and then detaching and reattaching again causes it to show up
> again. Hence downgrading the severity.

This is incorrect. With my incoming mailbox, the bug is reproducible
after doing Ctrl-L. For instance, Page Up + Page Down yields display
issues, e.g.

   2931 N   2016-11-30 ...
   2932 2016-11-30 ...
   9337 N + 2016-11-30 ...
   2934 N + 2016-11-30 ...

Also, switching screen (which I do quite often, this is one of the main
feature of GNU screen) also yields upward shift. One major problem is
that due to the shift, the arrow can be on the wrong message (the arrow
is at the right place, but the contents had been shifted upwards), and
if I type 'd' to delete the message, the D mark appears on the wrong
message for the same reason. This means that the message that will be
deleted is not the one I expected from the display (I already deleted
a wrong mail in the past due to a similar bug, which was this time a
character whose wcwdith was 0). Note that it may not be obvious that
the contents have shifted upwards, in particular because in my normal
config, the help line is not shown.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#846396: screen: display corruption with bce due to wide character

[Axel Beckert]

Control: tag -1 + confirmed upstream
Control: severity -1 normal

Hi Vincent,

Vincent Lefevre wrote:
> A wide character can corrupt the display, as shown by the following
> example with Mutt. The mailbox file "mbox" and config file "muttrc"
> are attached.

Thanks for the very detailed bug report and the attached examples.

I was able to reproduce this issue inside an urxvt, but not inside an
uxterm — there I just got a question mark inside a filled circle
instead of a wide character.

> I get:
> 
> 
>   1 Nov 30 a@x.invalid (   1) mail 1
> ->2 Nov 30 b@x.invalid (   1) XX mail 2
> 
> i.e. the contents have shifted upwards.
> 
> 5. Type the up arrow.
> 
> Nothing changes, i.e. the arrow still seems to be over mail 2,
> while the current message is mail 1.

And if you type cursor down, you get two arrows, one below the line
with no. 2, i.e. where the line should be, but isn't.

Nevertheless, there is an even partially persistent workaround: Press
Ctrl-L inside mutt and the problem vanishes even until after the next
detaching and reattaching. Only restarting mutt inside the screen
session and then detaching and reattaching again causes it to show up
again. Hence downgrading the severity.

Regards, Axel
-- 
 ,''`.  |  Axel Beckert , http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
  `-|  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE

Bug#846396: screen: display corruption with bce due to wide character

[Vincent Lefevre]

Package: screen
Version: 4.4.0-6
Severity: important

A wide character can corrupt the display, as shown by the following
example with Mutt. The mailbox file "mbox" and config file "muttrc"
are attached.

1. Start "screen" with "defbce on" in the .screenrc file.

2. Open this mailbox with: mutt -F muttrc -f mbox

On a 60-column terminal, I get:


q:Quit  d:Del  u:Undel  s:Save  m:Mail  r:Reply  g:Group  ?:
  1 Nov 30 a@x.invalid (   1) mail 1
->2 Nov 30 b@x.invalid (   1) XX mail 2

where XX (which corresponds to U+1F534 LARGE RED CIRCLE) depends
on the terminal, but takes 2 columns.

3. Detach the screen session.

4. Reattach the screen session.

I get:


  1 Nov 30 a@x.invalid (   1) mail 1
->2 Nov 30 b@x.invalid (   1) XX mail 2

i.e. the contents have shifted upwards.

5. Type the up arrow.

Nothing changes, i.e. the arrow still seems to be over mail 2,
while the current message is mail 1.

-- System Information:
Debian Release: stretch/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'stable-updates'), (500, 
'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.8.0-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=POSIX, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages screen depends on:
ii  libc6  2.24-7
ii  libpam0g   1.1.8-3.3
ii  libtinfo5  6.0+20160917-1

screen recommends no packages.

Versions of packages screen suggests:
pn  byobu | screenie | iselect  
ii  ncurses-term6.0+20160917-1

-- no debconf information
>From a@x.invalid Wed Nov 30 19:59:33 2016
From: a@x.invalid
Subject: mail 1
Date: Wed, 30 Nov 2016 19:58:44 +0100
MIME-Version: 1.0
Content-Type: text/plain
Status: RO

mail 1

>From b@x.invalid Wed Nov 30 20:06:58 2016
From: b@x.invalid
Date: Wed, 30 Nov 2016 20:06:53 +0100 (CET)
Subject: =?UTF-8?Q?=F0=9F=94=B4_mail_2?=
MIME-Version: 1.0
Content-Type: text/plain
Status: RO

mail 2

set arrow_cursor