Re: [PossibleSpam] UTF-8 chars

[Omar Polo]

On 2023/12/16 20:12:16 +0100, Robert Palm  wrote:
> 
> Thanks again to all of you!
> 
> xterm is now working!!!
> 
> Still, xfce4-terminal doesn't seem to use this settings.
> 
> Don't dare to ask where that setting is digged.

I tried for curiosity and xfce4-terminal doesn't show the lock icon for
me either ( -- U+1F512) but japanese input (via fcitx) works which I
find curious.  I tried with several fonts (DejaVu, JuliaMono, IBM Plex,
Iosevka).

Re: [PossibleSpam] UTF-8 chars

[Robert Palm]

Thanks again to all of you!

xterm is now working!!!

Still, xfce4-terminal doesn't seem to use this settings.

Don't dare to ask where that setting is digged.


Quoting Omar Polo :


On 2023/12/16 12:08:03 +0100, Nowarez Market  wrote:


Using xterm,
then CTRL + Right click,
I add TrueType and I can display the lock:

https://5md.es/l/2cc972f

Seems you need a TrueType font do display the trick...


Yes, the default 'fixed' font has some 'fancy' character in it but not
much else.

The alternative is to run xterm with -fa 'Font Name' or something like
this to your ~/.Xdefaults (and then xrdb -load ~/.Xdefaults dependending
on your configuration)

XTerm*faceName: xft:DejaVu Sans Mono:pixelsize=10

also...


Robert Palm  wrote:

>
> Seems I have no success...
>
> Did a
>
> export LC_CTYPE=de_DE.UTF-8


How did you ran this?  Running *inside* the terminal emulator won't
change anything.  I don't know if xfce sets up this var by itself, but
usually you'd need it in your ~/.xsession file.

Re: [PossibleSpam] UTF-8 chars

[Omar Polo]

On 2023/12/16 12:08:03 +0100, Nowarez Market  wrote:
> 
> Using xterm, 
> then CTRL + Right click, 
> I add TrueType and I can display the lock:
> 
> https://5md.es/l/2cc972f
> 
> Seems you need a TrueType font do display the trick...

Yes, the default 'fixed' font has some 'fancy' character in it but not
much else.

The alternative is to run xterm with -fa 'Font Name' or something like
this to your ~/.Xdefaults (and then xrdb -load ~/.Xdefaults dependending
on your configuration)

XTerm*faceName: xft:DejaVu Sans Mono:pixelsize=10

also...

> Robert Palm  wrote:
> 
> > 
> > Seems I have no success...
> > 
> > Did a
> > 
> > export LC_CTYPE=de_DE.UTF-8

How did you ran this?  Running *inside* the terminal emulator won't
change anything.  I don't know if xfce sets up this var by itself, but
usually you'd need it in your ~/.xsession file.

Re: UTF-8 chars

[Capitan Cloud]

Who did ircnow Billy G ?

Re: UTF-8 chars

[jrmu]

This is the setup I use for Chinese and it works OK for me for
traditional Chinese and xterm:

https://wiki.ircnow.org/index.php?n=Openbsd.Uim

Fonts render properly and traditional Chinese input works.

-- 
jrmu
IRCNow (https://ircnow.org)

Re: [PossibleSpam] UTF-8 chars

[Nowarez Market]

Using xterm, 
then CTRL + Right click, 
I add TrueType and I can display the lock:

https://5md.es/l/2cc972f

Seems you need a TrueType font do display the trick...


== Nowarez Market


Robert Palm  wrote:

> 
> Seems I have no success...
> 

Re: [PossibleSpam] UTF-8 chars

[Robert Palm]

Seems I have no success...

Did a

export LC_CTYPE=de_DE.UTF-8

and use

DejaVu Sans Mono Book 12

as xterm font.

Can enter, e.g. Σ with CTRL - SHIFT u 03a3 (or simply copy it in xterm).

For 1f512 , i.e. , still only a blank.

Anything else you specified maybe ?

I have xfce-terminal 1.1.1



Quoting Omar Polo :


On 2023/12/15 15:33:44 -0600, "Jay F. Shachter"  wrote:


Centuries ago, Nostradamus predicted that Robert Palm would write  
on Fri Dec 15 15:10:58 2023:


>
> I am playing with UTF-8 characters and try to display, e.g. a lock
> symbol https://www.compart.com/en/unicode/U+1F512
>
> I use xfce and set the terminal default character encoding to UTF-8 in
> advanced settings.
>
> Still I cannot get it displayed in xterm or emacs, e.g. by simply
> trying to copy paste from the website :-/
>
> Emacs gives a box and xterm a blank.


This is probably 'just' a font issue.  For instance, I'm currently using
DejaVu Sans Mono on xterm and it renders the lock icon for me, while on
emacs Julia Mono doesn't.

Make sure you have a UTF8 locale set too.


Does your version of xterm support UTF-8?  Uxterm was originally the
version of xterm that supported UTF-8.  That was a long time ago; on
my system, xterm now also supports UTF-8.  Maybe on your system you
have to install uxterm.


xterm on OpenBSD definitely supports and has utf8 enabled.

Re: UTF-8 chars

[Robert Palm]

Thanks a lot for looking into my admittedly exotic question.

I will try the font op@ suggested, but wonder about noto (which was my pick).

Am 16. Dez. 2023, 04:44, um 04:44, Nowarez Market  schrieb:
>
>I neither can't see the char you show us but about the support of the
>unicode\Chinese in OpenBSD..
>
>I have the font Noto in my system and I own the following in the
>.xinitrc file:
>
>export LC_CTYPE=zh_CN.UTF-8
>
>export GTK_IM_MODULE=xim
>export QT_IM_MODULE=xim
>export GLFW_IM_MODULE=ibus
>export XMODIFIERS="@im=SCIM"
>/usr/local/bin/scim -d &
>
>xconsole -iconic &
>
>/usr/local/bin/startxfce4
>
>Indeed I can now do the following from xfce4-terminal:
>
>wiz$ pwd
>/home/wiz
>wiz$ ls
>
>wiz$ touch 你好.txt
>wiz$ ls
>你好.txt
>
>Please not that xfce4-terminal encoding is configured on UTF-8 but it
>can use any frontend font like "Monospace Regular 10" and at need seems
>to pick up the right font.
>
>I want to thank everyone worked on the unicode support at system level
>for the nice eandover, appreciated.
>
>
>== Nowarez Market
>
>
>
>Robert Palm  wrote:
>
>> I am playing with UTF-8 characters and try to display, e.g. a lock
>> symbol https://www.compart.com/en/unicode/U+1F512
>>
>> I use xfce and set the terminal default character encoding to UTF-8
>> in advanced settings.
>>
>> Still I cannot get it displayed in xterm or emacs, e.g. by simply
>> trying to copy paste from the website :-/
>>
>> Emacs gives a box and xterm a blank.
>>
>> Any hints? Thanks!

Re: UTF-8 chars

[Nowarez Market]

I neither can't see the char you show us but about the support of the
unicode\Chinese in OpenBSD..

I have the font Noto in my system and I own the following in the
.xinitrc file:

export LC_CTYPE=zh_CN.UTF-8

export GTK_IM_MODULE=xim
export QT_IM_MODULE=xim
export GLFW_IM_MODULE=ibus
export XMODIFIERS="@im=SCIM"
/usr/local/bin/scim -d &

xconsole -iconic &

/usr/local/bin/startxfce4

Indeed I can now do the following from xfce4-terminal:

wiz$ pwd
/home/wiz
wiz$ ls

wiz$ touch 你好.txt
wiz$ ls
你好.txt

Please not that xfce4-terminal encoding is configured on UTF-8 but it
can use any frontend font like "Monospace Regular 10" and at need seems
to pick up the right font.

I want to thank everyone worked on the unicode support at system level
for the nice eandover, appreciated.


== Nowarez Market

  

Robert Palm  wrote:

> I am playing with UTF-8 characters and try to display, e.g. a lock  
> symbol https://www.compart.com/en/unicode/U+1F512
> 
> I use xfce and set the terminal default character encoding to UTF-8
> in advanced settings.
> 
> Still I cannot get it displayed in xterm or emacs, e.g. by simply  
> trying to copy paste from the website :-/
> 
> Emacs gives a box and xterm a blank.
> 
> Any hints? Thanks!

Re: [PossibleSpam] UTF-8 chars

[Omar Polo]

On 2023/12/15 15:33:44 -0600, "Jay F. Shachter"  wrote:
> 
> Centuries ago, Nostradamus predicted that Robert Palm would write on Fri Dec 
> 15 15:10:58 2023:
> 
> > 
> > I am playing with UTF-8 characters and try to display, e.g. a lock  
> > symbol https://www.compart.com/en/unicode/U+1F512
> > 
> > I use xfce and set the terminal default character encoding to UTF-8 in  
> > advanced settings.
> > 
> > Still I cannot get it displayed in xterm or emacs, e.g. by simply  
> > trying to copy paste from the website :-/
> > 
> > Emacs gives a box and xterm a blank.

This is probably 'just' a font issue.  For instance, I'm currently using
DejaVu Sans Mono on xterm and it renders the lock icon for me, while on
emacs Julia Mono doesn't.

Make sure you have a UTF8 locale set too.

> Does your version of xterm support UTF-8?  Uxterm was originally the
> version of xterm that supported UTF-8.  That was a long time ago; on
> my system, xterm now also supports UTF-8.  Maybe on your system you
> have to install uxterm.

xterm on OpenBSD definitely supports and has utf8 enabled.

Re: [PossibleSpam] UTF-8 chars

[Jay F. Shachter]

Centuries ago, Nostradamus predicted that Robert Palm would write on Fri Dec 15 
15:10:58 2023:

> 
> I am playing with UTF-8 characters and try to display, e.g. a lock  
> symbol https://www.compart.com/en/unicode/U+1F512
> 
> I use xfce and set the terminal default character encoding to UTF-8 in  
> advanced settings.
> 
> Still I cannot get it displayed in xterm or emacs, e.g. by simply  
> trying to copy paste from the website :-/
> 
> Emacs gives a box and xterm a blank.
> 

Does your version of xterm support UTF-8?  Uxterm was originally the
version of xterm that supported UTF-8.  That was a long time ago; on
my system, xterm now also supports UTF-8.  Maybe on your system you
have to install uxterm.

Jay F. Shachter
6424 North Whipple Street
Chicago IL  60645-4111
(1-773)7613784   landline
(1-410)9964737   GoogleVoice
j...@m5.chicago.il.us
http://m5.chicago.il.us

"Quidquid latine dictum sit, altum videtur"

UTF-8 chars

[Robert Palm]

I am playing with UTF-8 characters and try to display, e.g. a lock  
symbol https://www.compart.com/en/unicode/U+1F512


I use xfce and set the terminal default character encoding to UTF-8 in  
advanced settings.


Still I cannot get it displayed in xterm or emacs, e.g. by simply  
trying to copy paste from the website :-/


Emacs gives a box and xterm a blank.

Any hints? Thanks!

Re: ls cant print higher UTF-8 chars

[Ingo Schwarze]

Hi Jon,

Jon S wrote on Wed, Jan 06, 2016 at 10:56:52AM +0100:

> In an attempt to get a better idea on how to make all non-ascii chars
> appear correctly in windows/samba, ssh and the local console I get the
> impression that UTF-8 is the charset that is mostly used in general and
> also most growing.

We agree with that.  UTF-8 and US-ASCII are the only charsets that
are supported in -current and in the upcoming OpenBSD 5.9, and there
are no plans to support any other charsets in the future.

> Using UTF-8 in samba and ssh on a OpenBSD 4.9 i386 generic machine
> that is about to retire

You should have retired that machine three years ago.  4.9 went EOL
on May 1, 2012.  I have no idea why anything worked or did not work
there.

> works as expected, tested with swedish and icelandish
> letters outside the 7bit ascii range.
> 
> However, the new machine to replace the 4.9-box, running OpenBSD 5.6

That makes no sense.  5.6 is not new at all.  It went EOL more than
two months ago, on October 18, 2015.

> on amd64 generic.mp #333, shows questionmarks when doing ls in ssh
> (using PuTTY set to UTF-8 in translation).

Yes, ls(1) replaces unknown characters by question marks when the
locale(1) is set to something that is not supported.  I'm no
longer sure what 5.6 supported because it's too old.

> The funny thing is that autocomplete using tab shows the expected
> letters.  pwd shows expected letters

Yes, the shells do less sanitation than ls(1).

> and even ls | cat shows correct letters.

No sure what that does in 5.6.

> Is this a known problem?
> Is there a solution to make ls print correct UTF-8?

Use -current.  I fixed ls(1) recently to correctly handle UTF-8.
On May 1, 2016, upgrade to OpenBSD 5.9 if you want to use -stable.
Do not forget to upgrade to OpenBSD 6.0 on November 1, 2016.
UTF-8 support is likely to improve further between 5.9 and 6.0.

Yours,
  Ingo

ls cant print higher UTF-8 chars

[Jon S]

Hello misc!

In an attempt to get a better idea on how to make all non-ascii chars
appear correctly in windows/samba, ssh and the local console I get the
impression that UTF-8 is the charset that is mostly used in general and
also most growing.

Using UTF-8 in samba and ssh on a OpenBSD 4.9 i386 generic machine that is
about to retire works as expected, tested with swedish and icelandish
letters outside the 7bit ascii range.

However, the new machine to replace the 4.9-box, running OpenBSD 5.6 on
amd64 generic.mp #333, shows questionmarks when doing ls in ssh (using
PuTTY set to UTF-8 in translation). The funny thing is that autocomplete
using tab shows the expected letters. pwd shows expected letters and even
ls | cat shows correct letters.

Is this a known problem? Is there a solution to make ls print correct UTF-8?

--
<>
Jon Sjöstedt 

Re: ls cant print higher UTF-8 chars

[Mike Burns]

On 2016-01-06 10.56.52 +0100, Jon S wrote:
> Is this a known problem? Is there a solution to make ls print correct UTF-8?

Use colorls from packages instead.

http://marc.info/?l=openbsd-misc=142539814225472=2

Re: ls cant print higher UTF-8 chars

[Martijn van Duren]

On 01/06/16 11:05, Mike Burns wrote:

On 2016-01-06 10.56.52 +0100, Jon S wrote:

Is this a known problem? Is there a solution to make ls print correct UTF-8?


Use colorls from packages instead.

http://marc.info/?l=openbsd-misc=142539814225472=2


Or wait for 5.9 to come out.

http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/bin/ls/

Re: ls cant print higher UTF-8 chars

[Christoph R. Murauer]

> Hello misc!
>
> In an attempt to get a better idea on how to make all non-ascii chars
> appear correctly in windows/samba, ssh and the local console I get the
> impression that UTF-8 is the charset that is mostly used in general
> and
> also most growing.
>
> Using UTF-8 in samba and ssh on a OpenBSD 4.9 i386 generic machine
> that is
> about to retire works as expected, tested with swedish and icelandish
> letters outside the 7bit ascii range.
>
> However, the new machine to replace the 4.9-box, running OpenBSD 5.6
> on
> amd64 generic.mp #333, shows questionmarks when doing ls in ssh (using
> PuTTY set to UTF-8 in translation). The funny thing is that
> autocomplete
> using tab shows the expected letters. pwd shows expected letters and
> even
> ls | cat shows correct letters.
>
> Is this a known problem? Is there a solution to make ls print correct
> UTF-8?

Yep, use colorls instead of ls. Works for german umlauts.

http://openports.se/sysutils/colorls

Re: ls cant print higher UTF-8 chars

[Christoph R. Murauer]

> Use -current.  I fixed ls(1) recently to correctly handle UTF-8.
> On May 1, 2016, upgrade to OpenBSD 5.9 if you want to use -stable.
> Do not forget to upgrade to OpenBSD 6.0 on November 1, 2016.
> UTF-8 support is likely to improve further between 5.9 and 6.0.
>
> Yours,
>   Ingo

Thanks for the work and the information.

uxterm is showing UTF-8 chars with errors?

[Thisis theone]

$ touch árvíztűrő tükörfúrógép
$ ls -lah
-rw---   1 user  user 0B Feb  8 18:20 ??rv??zt??r?? t??k??rf??r??g??p
$

I am using uxterm on OpenBSD 5.6. How can my uxterm show these accents in
this way? Why doesn't it displays it as it is?

Many thanks!

Re: uxterm is showing UTF-8 chars with errors?

[frantisek holop]

Thisis theone, 03 Mar 2015 16:55:
 $ touch árvíztűrő tükörfúrógép
 $ ls -lah
 -rw---   1 user  user 0B Feb  8 18:20 ??rv??zt??r?? t??k??rf??r??g??p
 $
 
 I am using uxterm on OpenBSD 5.6. How can my uxterm show these accents in
 this way? Why doesn't it displays it as it is?

look in the archives, there was a discussion not that
long ago.

$ ls |cat
árvíztűrő tükörfúrógép

-f
-- 
the word of the day is legs.  now spread the word!

Re: uxterm is showing UTF-8 chars with errors?

[Ted Unangst]

Thisis theone wrote:
 $ touch árvíztűrő tükörfúrógép
 $ ls -lah
 -rw---   1 user  user 0B Feb  8 18:20 ??rv??zt??r?? t??k??rf??r??g??p
 $
 
 I am using uxterm on OpenBSD 5.6. How can my uxterm show these accents in
 this way? Why doesn't it displays it as it is?

ls doesn't know about utf-8. it only prints basic ascii characters, and
replaces all other bytes with ?.

The problem is not in xterm (or the filesystem). If you run echo * you should
see the name echoed back correctly.

Re: uxterm is showing UTF-8 chars with errors?

[someone]

Thank you for the tricks! :) (Google already indexed it, so less people
will ask it in the future, lol)

Is this an old bug or just a feature?

I know it would be great if the world would only have 1 language: English,
but that will be about ~1000 years away.

http://www.wsj.com/articles/what-the-world-will-speak-in-2115-1420234648


On Tue, Mar 3, 2015 at 5:10 PM, Ted Unangst t...@tedunangst.com wrote:

 Thisis theone wrote:
  $ touch árvíztűrő tükörfúrógép
  $ ls -lah
  -rw---   1 user  user 0B Feb  8 18:20 ??rv??zt??r??
 t??k??rf??r??g??p
  $
 
  I am using uxterm on OpenBSD 5.6. How can my uxterm show these accents in
  this way? Why doesn't it displays it as it is?

 ls doesn't know about utf-8. it only prints basic ascii characters, and
 replaces all other bytes with ?.

 The problem is not in xterm (or the filesystem). If you run echo * you
 should
 see the name echoed back correctly.

Re: uxterm is showing UTF-8 chars with errors?

[Stefan Sperling]

On Tue, Mar 03, 2015 at 04:55:01PM +0100, Thisis theone wrote:
 $ touch árvíztűrő tükörfúrógép
 $ ls -lah
 -rw---   1 user  user 0B Feb  8 18:20 ??rv??zt??r?? t??k??rf??r??g??p
 $
 
 I am using uxterm on OpenBSD 5.6. How can my uxterm show these accents in
 this way? Why doesn't it displays it as it is?
 
 Many thanks!

This is because ls(1) filters output with isprint(3) and is not aware
of locales (i.e. it does not call setlocale(3)). Run pkg_add colorls
and alias ls=colorls if you need multi-byte ls output.

Please do not start a discussion about adding this feature to base ls(1)
unless you're willing to invest a non-trivial amount of time and energy
working on improved locale support for the entire OS.
It's already been discussed before.

Re: uxterm is showing UTF-8 chars with errors?

[Ted Unangst]

Stefan Sperling wrote:
 On Tue, Mar 03, 2015 at 04:55:01PM +0100, Thisis theone wrote:
  $ touch árvíztűrő tükörfúrógép
  $ ls -lah
  -rw---   1 user  user 0B Feb  8 18:20 ??rv??zt??r?? 
  t??k??rf??r??g??p
  $
  
  I am using uxterm on OpenBSD 5.6. How can my uxterm show these accents in
  this way? Why doesn't it displays it as it is?
  
  Many thanks!
 
 This is because ls(1) filters output with isprint(3) and is not aware
 of locales (i.e. it does not call setlocale(3)). Run pkg_add colorls
 and alias ls=colorls if you need multi-byte ls output.

As a shortcut, filtering out just esc will prevent most terminal damage? I'm
not sure what other characters can do, though... I vageuly recall that the
intersection of utf-8 and xterm controls is unknowable.

poc diff:

Index: util.c
===
RCS file: /cvs/src/bin/ls/util.c,v
retrieving revision 1.16
diff -u -p -r1.16 util.c
--- util.c  21 Nov 2013 15:54:45 -  1.16
+++ util.c  3 Mar 2015 16:56:15 -
@@ -51,7 +51,7 @@ putname(char *name)
int len;
 
for (len = 0; *name; len++, name++)
-   putchar((!isprint((unsigned char)*name)  f_nonprint) ? '?' : 
*name);
+   putchar((*name == 0x1b  f_nonprint) ? '?' : *name);
return len;
 }

Re: uxterm is showing UTF-8 chars with errors?

[someone]

pkg_add colorls
alias ls=colorls

This one did it, many thanks!!

On Tue, Mar 3, 2015 at 5:41 PM, Stefan Sperling s...@stsp.name wrote:

 On Tue, Mar 03, 2015 at 04:55:01PM +0100, Thisis theone wrote:
  $ touch árvíztűrő tükörfúrógép
  $ ls -lah
  -rw---   1 user  user 0B Feb  8 18:20 ??rv??zt??r??
 t??k??rf??r??g??p
  $
 
  I am using uxterm on OpenBSD 5.6. How can my uxterm show these accents in
  this way? Why doesn't it displays it as it is?
 
  Many thanks!

 This is because ls(1) filters output with isprint(3) and is not aware
 of locales (i.e. it does not call setlocale(3)). Run pkg_add colorls
 and alias ls=colorls if you need multi-byte ls output.

 Please do not start a discussion about adding this feature to base ls(1)
 unless you're willing to invest a non-trivial amount of time and energy
 working on improved locale support for the entire OS.
 It's already been discussed before.

Re: uxterm is showing UTF-8 chars with errors?

[Thomas Bohl]

Am 03.03.2015 um 17:58 schrieb Ted Unangst:

As a shortcut, filtering out just esc will prevent most terminal damage? I'm
not sure what other characters can do, though... I vageuly recall that the
intersection of utf-8 and xterm controls is unknowable.

poc diff:

Index: util.c
===
RCS file: /cvs/src/bin/ls/util.c,v
retrieving revision 1.16
diff -u -p -r1.16 util.c
--- util.c  21 Nov 2013 15:54:45 -  1.16
+++ util.c  3 Mar 2015 16:56:15 -
@@ -51,7 +51,7 @@ putname(char *name)
int len;

for (len = 0; *name; len++, name++)
-   putchar((!isprint((unsigned char)*name)  f_nonprint) ? '?' : 
*name);
+   putchar((*name == 0x1b  f_nonprint) ? '?' : *name);
return len;
  }




Thank you very much! Colorls still showed me ?? for the majority of 
characters. This patch works as expected.



If the filtering is done for security reasons I just want to humbly add 
that in many circumstances pressing the tab-key instead of enter will 
deliver a unsanitized file listing anyway. (By the shell I would assume.)

Re: uxterm is showing UTF-8 chars with errors?

[Stefan Sperling]

On Tue, Mar 03, 2015 at 11:41:07PM +0100, Thomas Bohl wrote:
 Thank you very much! Colorls still showed me ?? for the majority of
 characters. This patch works as expected.

I'm not sure what kind of behaviour you expect.

colorls showing some ?? indicates that the character set used by
filenames and your locale character set configuration do not match.
See http://www.openbsd.org/faq/faq10.html#locales in case you
didn't set up your locale yet. Set your locale's charset to
the charset used by your filenames. Then try colorls again.
If charsets align, it should just work.

In case you have mixed charsets in filenames which then causes ?? this
is very hard to deal with in any case. In this situation you could
pkg_add convmv and use that tool to straighten out filename charsets.

Perhaps tedu's diff is a good idea, perhaps not.
Making ls(1) aware of character encodings has some advantages (e.g.
multi-column output always aligns properly) and some disadvantages
(need to set up the locale first, can only use one charset at a time).
Just allowing any garbage might sometimes make things appear to
work as if by magic, but could also corrupt your terminal or worse.
Then again, ls probably shouldn't be in the business of compensating
for bugs in terminal emulators. I suspect many other tools aren't,
either (e.g. df(1) doesn't care).