Bug#840211: perlunicook man page does not display utf8 char beyond ascii

[Niko Tyni]

On Sat, Dec 03, 2016 at 12:21:39PM -0800, Russ Allbery wrote:
> Niko Tyni  writes:
> 
> > No, I don't think we've ever done that. I think it *should* be as simple
> > as patching ./installman to pass utf8 => 1 to Pod::Man (the Perl build
> > system doesn't use pod2man itself). There might be some issues with
> > cross-building but that's not really supported anyway.
> 
> Yeah, I think we're at the point where we should just go ahead and do this.

Agreed, so I already did it in the last upload, 5.24.1~rc4-1 :)

Sorry for not following up on the discussion.

> > I think my only problem with this is that upstreaming it properly would
> > be somewhat difficult due to the incompatibilities noted here. I suppose
> > I shouldn't let that be a blocker. Dominic, any opinion?
> 
> > (Possibly a manual Configure option like -Dmanutf8, maybe even
> > enabled by default on Linux, would be acceptable upstream. No idea how
> > an automatic Configure probe for the UTF8 compatibility would work.)
> 
> The upstreaming part is really my fault, since I've been intending for
> ages to start a thread on pod-people and perl5-porters about the best
> approach in general but haven't gotten to it (or, more to the point,
> haven't made sure I had the resources to participate in the thread and act
> on the results).  This is definitely on my to-do list to do.

Thanks. I decided to just go ahead and worry about upstreaming later
to make sure we get fixed man pages in stretch. So it's now on my to-do
list as well, but I'm very happy if you get there first :)
-- 
Niko

Bug#840211: perlunicook man page does not display utf8 char beyond ascii

[Russ Allbery]

Niko Tyni  writes:

> No, I don't think we've ever done that. I think it *should* be as simple
> as patching ./installman to pass utf8 => 1 to Pod::Man (the Perl build
> system doesn't use pod2man itself). There might be some issues with
> cross-building but that's not really supported anyway.

Yeah, I think we're at the point where we should just go ahead and do this.

> ISTR old concerns about Debian systems sharing manual pages (via NFS
> I guess) to other platforms with incompatible nroff implementations.
> But that seems like a silly corner case that we shouldn't care about.

Right, I'm pretty dubious that anyone is going to be doing this these
days, since package managers make this so much easier.  At the least, I
doubt anyone will be doing it with incompatible systems that aren't using
man-db.

> I think my only problem with this is that upstreaming it properly would
> be somewhat difficult due to the incompatibilities noted here. I suppose
> I shouldn't let that be a blocker. Dominic, any opinion?

> (Possibly a manual Configure option like -Dmanutf8, maybe even
> enabled by default on Linux, would be acceptable upstream. No idea how
> an automatic Configure probe for the UTF8 compatibility would work.)

The upstreaming part is really my fault, since I've been intending for
ages to start a thread on pod-people and perl5-porters about the best
approach in general but haven't gotten to it (or, more to the point,
haven't made sure I had the resources to participate in the thread and act
on the results).  This is definitely on my to-do list to do.

-- 
Russ Allbery (r...@debian.org)   

Bug#840211: perlunicook man page does not display utf8 char beyond ascii

[Niko Tyni]

On Fri, Nov 18, 2016 at 12:16:49PM -0800, Russ Allbery wrote:
> Dominique Dumont  writes:
> 
> > Looks like I was not clear. I was thinking of patching perl build system
> > to generate perl* man page (perlrun perlunicook...) as utf8 man page.
> 
> > That should not impact the generation of other man pages (i.e. not in
> > Debian perl package) using pod2man.
> 
> Oh, yes.  I thought that had already been done at some point by forcing
> the build system to pass -u, but possibly not.  I'm not sure how hard that
> is to do.

No, I don't think we've ever done that. I think it *should* be as simple
as patching ./installman to pass utf8 => 1 to Pod::Man (the Perl build
system doesn't use pod2man itself). There might be some issues with
cross-building but that's not really supported anyway.

ISTR old concerns about Debian systems sharing manual pages (via NFS
I guess) to other platforms with incompatible nroff implementations.
But that seems like a silly corner case that we shouldn't care about.

I think my only problem with this is that upstreaming it properly would
be somewhat difficult due to the incompatibilities noted here. I suppose
I shouldn't let that be a blocker. Dominic, any opinion?

(Possibly a manual Configure option like -Dmanutf8, maybe even
enabled by default on Linux, would be acceptable upstream. No idea how
an automatic Configure probe for the UTF8 compatibility would work.)
-- 
Niko

Bug#840211: perlunicook man page does not display utf8 char beyond ascii

[Russ Allbery]

Dominique Dumont  writes:

> Looks like I was not clear. I was thinking of patching perl build system
> to generate perl* man page (perlrun perlunicook...) as utf8 man page.

> That should not impact the generation of other man pages (i.e. not in
> Debian perl package) using pod2man.

Oh, yes.  I thought that had already been done at some point by forcing
the build system to pass -u, but possibly not.  I'm not sure how hard that
is to do.

-- 
Russ Allbery (r...@debian.org)   

Bug#840211: perlunicook man page does not display utf8 char beyond ascii

[Dominique Dumont]

On Friday, 18 November 2016 10:54:36 CET Russ Allbery wrote:
>> Considering [1], we may want to patch Debian's perl to produce utf8 man
> > pages even if upstream cannot, due to portability constraints.
> 
> That's not strictly true, which is the reason why I haven't already
> switched Linux systems over to generating UTF-8 by default.  This means
> that you cannot create man pages with pod2man on a Linux system and have
> them work on any other system, something that pod2man has, up until this
> point, always guaranteed.

Looks like I was not clear. I was thinking of patching perl build system to 
generate  perl* man page (perlrun perlunicook...) as utf8 man page. 

That should not impact the generation of other man pages (i.e. not in Debian 
perl package) using pod2man.

HTH
-- 
 https://github.com/dod38fr/   -o- http://search.cpan.org/~ddumont/
http://ddumont.wordpress.com/  -o-   irc: dod at irc.debian.org

Bug#840211: perlunicook man page does not display utf8 char beyond ascii

[Russ Allbery]

Dominique Dumont  writes:
> On Friday, 18 November 2016 09:34:03 CET Russ Allbery wrote:

>> Guillem did some more investigation, and sadly it turns out that raw
>> UTF-8 in man pages continues to break even completely current man
>> implementations on platforms like macOS.

> That's a constraint we do not have on Debian/unstable.

> Considering [1], we may want to patch Debian's perl to produce utf8 man
> pages even if upstream cannot, due to portability constraints.

That's not strictly true, which is the reason why I haven't already
switched Linux systems over to generating UTF-8 by default.  This means
that you cannot create man pages with pod2man on a Linux system and have
them work on any other system, something that pod2man has, up until this
point, always guaranteed.

This will break some uses of pod2man.  It's not uncommon for people to
write man pages in POD even for non-Perl packages, and ship release
tarballs with pregenerated man pages so that the installer doesn't have to
have Perl locally.  If those contain UTF-8, this will result in man pages
in the release tarball that aren't portable.

It may well be that this is acceptable breakage (I'm pretty much convinced
that it is), but it's not a decision with zero drawbacks.

-- 
Russ Allbery (r...@debian.org)   

Bug#840211: perlunicook man page does not display utf8 char beyond ascii

[Dominique Dumont]

On Friday, 18 November 2016 09:34:03 CET Russ Allbery wrote:
> Guillem did some more investigation, and sadly it turns out that raw UTF-8
> in man pages continues to break even completely current man
> implementations on platforms like macOS.

That's a constraint we do not have on Debian/unstable.

Considering [1], we may want to patch Debian's perl to produce utf8 man pages 
even if upstream cannot, due to portability constraints.  

All the best.

[1] https://wiki.debian.org/ReleaseGoals/utf-8
-- 
 https://github.com/dod38fr/   -o- http://search.cpan.org/~ddumont/
http://ddumont.wordpress.com/  -o-   irc: dod at irc.debian.org

Bug#840211: perlunicook man page does not display utf8 char beyond ascii

[Russ Allbery]

Niko Tyni  writes:

> I think upstream (Russ and iirc others on p5p too) have been stepping
> very carefully here due to compatibility issues with older nroff
> implementations and the like.

Guillem did some more investigation, and sadly it turns out that raw UTF-8
in man pages continues to break even completely current man
implementations on platforms like macOS.

> There's [rt.cpan.org #68741] upstream, quoting Russ in 2011: "I'm
> currently leaning towards outputing UTF-8 by default, but I'm kicking
> around the idea of trying to use the user's locale."

> Russ, any thoughts about the current status?

I still haven't done anything.  :(  It looks like any solution will
require generating non-portable man pages.  Maybe that doesn't matter any
more, and I should just default to generating UTF-8 man pages on any Linux
platform.

-- 
Russ Allbery (r...@debian.org)   

Bug#840211: perlunicook man page does not display utf8 char beyond ascii

[Niko Tyni]

severity 401089 normal
reassign 840211 perl-doc 5.24.1~rc3-3
forcemerge 840211 401089
forwarded 401089 https://rt.cpan.org/Public/Bug/Display.html?id=68741
thanks

On Sun, Oct 09, 2016 at 05:30:23PM +0200, Dominique Dumont wrote:
> Package: perl
> Version: 5.24.1~rc3-3
> Severity: normal
> 
> Dear Maintainer,
> 
> In perlunicook man page, non-ASCII utf-8 char are replaced by 'X'
> (which is a shame on a man page dealing with unicode and utf8 issue):

Indeed. This is also #401089, merging. A workaround is using 'perldoc'
instead of the man pages.

> I guess that this file was generated by pod2man without --utf8 option.

Yes.
 
> Could you please fix the generation of these man pages ?

I think upstream (Russ and iirc others on p5p too) have been stepping
very carefully here due to compatibility issues with older nroff
implementations and the like.

There's [rt.cpan.org #68741] upstream, quoting Russ in 2011: "I'm
currently leaning towards outputing UTF-8 by default, but I'm kicking
around the idea of trying to use the user's locale."

Russ, any thoughts about the current status?
-- 
Niko Tyni   nt...@debian.org

Bug#840211: perlunicook man page does not display utf8 char beyond ascii

[Dominique Dumont]

Package: perl
Version: 5.24.1~rc3-3
Severity: normal

Dear Maintainer,

In perlunicook man page, non-ASCII utf-8 char are replaced by 'X'
(which is a shame on a man page dealing with unicode and utf8 issue):

Compare:

  $ man perlunicook | grep -A 4 '$measure'
my $measure   = "Aangstroem";
my @Xsoft = qw( cp852 cp1251 cp1252 );
my @X = qw(   X );
my @X= qw( koi8-f koi8-u koi8-r );
my $motto = "X X X"; # FAMILY, GROWING HEART, DROMEDARY CAMEL


with:

  $ perldoc perlunicook | grep -A 4 '$measure'
 my $measure   = "Ångström";
 my @μsoft = qw( cp852 cp1251 cp1252 );
 my @ὑπέρμεγας = qw( ὑπέρ  μεγας );
 my @鯉= qw( koi8-f koi8-u koi8-r );
 my $motto = "👪 💗 🐪"; # FAMILY, GROWING HEART, DROMEDARY CAMEL

Turns out that /usr/share/man/man1/perlunicook.1.gz does not contain utf8 char.

I guess that this file was generated by pod2man without --utf8 option.

Could you please fix the generation of these man pages ?

All the best



-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (990, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386, armhf

Kernel: Linux 4.7.0-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages perl depends on:
ii  dpkg   1.18.10
ii  libperl5.245.24.1~rc3-3
ii  perl-base  5.24.1~rc3-3
ii  perl-modules-5.24  5.24.1~rc3-3

Versions of packages perl recommends:
ii  netbase  5.3
ii  rename   0.20-4

Versions of packages perl suggests:
ii  libterm-readline-gnu-perl   1.34-1+b1
ii  libterm-readline-perl-perl  1.0303-1
ii  make4.1-9
ii  perl-doc5.24.1~rc3-3

-- no debconf information