Re: [PLUG] 'tree' output not in US-ASCII character set

[Rich Shepard]

On Sat, 22 Apr 2017, King Beowulf wrote:

> I've poked around and discovered that in Slackware 14.2 we have alpine
> 2.20. There seems to be a problem with alpine not properly supporting
> UTF-8 and so defaults to US-ASCII. Unfortunately, when you then try to
> sent line drawing characters, alpine "guesses" and picks a cyrillic char
> set.

Hi Ed,

   This is on my server/workstation running 14.1 and alpine 2.11. Guess the
problem has been in alpine for a while.

> You may have to use Thunderbird or similar as a work around.

   Gak! For me, using a GUI for text such as e-mails is analagous to reading
commic books ... er, graphic novels ... rather than regular text books.

> Also, you can try sending as an attachment (with different extensions).

   Since I've not before had this issue, and it's not likely that I'll need
to send a graphic directory tree again, it's a very low priority. I
discovered that I can export the non-sending message to a file, then cancel
the oritinal message, start a new one and insert the saved text. Et, woila!
UTF-8.

   Thanks much and I'll let you get back to your more important things. :-)

Best regards,

Rich
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set

[Rich Shepard]

On Thu, 20 Apr 2017, Tom wrote:

> Disclaimer: I have no experience with Alpine or urxvt. In fact, I have
> never hear of either and since I have not clue what they might be
> google is not really helpful either.

   Alpine is the successor to pine, the UDub's MUA. urxvt is the
unicode-enabled version of the rxvt virtual terminal.

> With the above out of the way - your Copy & Paste action is most likely
> routed through your Desktop environment. In KDE/Gnome there is setting
> for Languages and Keyboards in their appropriate setting panels as well
> as in the application defaults and your environment (login shell
> LC_all/defaults..). I suspect that you are probably not a
> mainstream guy, so before you say something like fvwm2, motif, CDE,
> . LXDE - these have language and keyboard settings too, and some
> precedence for application and environment settings also - which could
> mess up you Copy + Paste outcome.

   Well, I've used Xfce since it first came out and have not before
encountered a character encoding change. Then again, I've not before used
the tree command to produce output to be included in the body of an e-mail
message.

> Some email clients, office and editors have "Paste Special" or "Paste
> as Text" function in menu - you could try those if that helps.

   Those are likely GUI applications. Alpine, like mutt and elm are text
based; the editor I use with alpine is joe.

Rich
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set

[Paul Mullen]

On Thu, Apr 20, 2017 at 12:25:10PM -0700, Rich Shepard wrote:
>If I tell tree to use utf8 or us-ascii I get the control sequence
> gibberish. When I paste (or insert) the tree output (without specifying a
> character code) into a composing alpine message something changes the
> character encoding to koi8-r.

As Tomas mentioned in another reply, it seems like something is
munging your cut buffer between selection and pasting.  It could be
XFCE; urxvt also has its own collection of plugin Perl scripts that
can act on the selection.  Try running urxvt with plugins disabled:

  urxvt -xrm 'URxvt*perl-ext-common: ' -xrm 'URxvt*perl-ext: '

>The command to invoke alpine specifies urxvt but that's ignored and the
> alpine window frame shows it's an xterm. Could this change character
> encoding of the enclosed text only with line drawing characters included?

Perhaps.  Have you tried opening another urxvt window and pasting into
that to see if the problem continues?


-- 
Paul Mullen
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set

[Tom]

Disclaimer: I have no experience with Alpine or urxvt. In fact, I have
never hear of either and since I have not clue what they might be
google is not really helpful either.
With the above out of the way - your Copy & Paste action is most likely
routed through your Desktop environment. In KDE/Gnome there is setting
for Languages and Keyboards in their appropriate setting panels as well
as in the application defaults and your environment (login shell
LC_all/defaults..). I suspect that you are probably not a
mainstream guy, so before you say something like fvwm2, motif, CDE,
. LXDE - these have language and keyboard settings too, and some
precedence for application and environment settings also - which could
mess up you Copy + Paste outcome.
Some email clients, office and editors have "Paste Special" or "Paste
as Text" function in menu - you could try those if that helps. Also if
your desktop environment runs on X you could also try to select the
text in terminal and try to paste it by middle click - if that is
wrong, perhaps look at your X settings (they are scattered all over the
place in /etc as well ac in your home and /tmp)
Good luck,
Tomas
On Thu, 2017-04-20 at 12:25 -0700, Rich Shepard wrote:
> On Thu, 20 Apr 2017, Paul Mullen wrote:
> 
> > Just to review...  You run `tree` in your terminal and it looks
> > normal.
> 
> Paul,
> 
>Yes, the urxvt supports unicode, and the file command tells me the
> character encoding is utf8.
> 
> > Then you cut and paste into another terminal window and end up with
> > a lot
> > of control sequence gibberish. When you sent the above-quoted
> > message, did
> > it appear to be koi8-r-encoded?
> 
>If I tell tree to use utf8 or us-ascii I get the control sequence
> gibberish. When I paste (or insert) the tree output (without
> specifying a
> character code) into a composing alpine message something changes the
> character encoding to koi8-r.
> 
>The command to invoke alpine specifies urxvt but that's ignored
> and the
> alpine window frame shows it's an xterm. Could this change character
> encoding of the enclosed text only with line drawing characters
> included?
> 
> Thanks,
> 
> Rich
> ___
> PLUG mailing list
> PLUG@lists.pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set

[John Meissen]

On Thu, Apr 20, 2017 at 05:48:31AM -0700, Rich Shepard wrote:
>Just ran tree on ~/ and copied last few lines of the output to a file:
> 
>  ├── legal
>  ├── palmer
>  │   ├── laxare
>  │   └── suppl-rpt
>  ├── past
>  ├── permitting
>  └── wallace
>  ├── laxare
>  └── response
-8<-
>BUT, when I copy that output to this message the language coding has been
> changed to koi8-r. I'd really like to understand why. Does it have something
> to do with alpine, or urxvt, or something else?

I think you should be looking at your mail client. I see

  Content-Type: text/plain; charset="koi8-r"

among the message headers. That has to get added by whatever you're using to 
send mail (which appears to be Alpine).




___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set

[Paul Mullen]

On Thu, Apr 20, 2017 at 05:48:31AM -0700, Rich Shepard wrote:
>Just ran tree on ~/ and copied last few lines of the output to a file:
> 
>  ├── legal
>  ├── palmer
>  │   ├── laxare
>  │   └── suppl-rpt
>  ├── past
>  ├── permitting
>  └── wallace
>  ├── laxare
>  └── response
-8<-
>BUT, when I copy that output to this message the language coding has been
> changed to koi8-r. I'd really like to understand why. Does it have something
> to do with alpine, or urxvt, or something else?

Just to review...  You run `tree` in your terminal and it looks
normal.  Then you cut and paste into another terminal window and end
up with a lot of control sequence gibberish.  When you sent the
above-quoted message, did it appear to be koi8-r-encoded?


-- 
Paul Mullen
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set

[Rich Shepard]

On Wed, 19 Apr 2017, Tom wrote:

> Have you considered those files/directories to be showing their correct
> UTF-8 names?

Tom,

   I assume that one word directory names follow the character string
definition in locale. So, yes, I assume they show their correct UTF-8 names.

   Just ran tree on ~/ and copied last few lines of the output to a file:

 ├── legal
 ├── palmer
 │   ├── laxare
 │   └── suppl-rpt
 ├── past
 ├── permitting
 └── wallace
 ├── laxare
 └── response
shell-scripts
├── awk
├── perl
├── python
└── sed

   Running the file command on this test file tells me:

$ file test.txt 
test.txt: UTF-8 Unicode text

   BUT, when I copy that output to this message the language coding has been
changed to koi8-r. I'd really like to understand why. Does it have something
to do with alpine, or urxvt, or something else?

> If that is so, and you do not like it, rename them.

   I like it.

> If you cannot easily rename them because of their "funny" names,

   If you mean 'funny' as in the Microsoft norm of spaces, I don't do that.

   Otherwise, I am not following your thoughts.

Thanks,

Rich
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set

[Tom]

Have you considered those files/directories to be showing their correct
UTF-8 names?
If that is so, and you do not like it, rename them.
If you cannot easily rename them because of their "funny" names, I
recommend using find with its plethora of options to pick up those
files individually. Something like this normally saves my skin:
find ./ -name/size/date/type 'searchGlobString/regexp/number/date/...' 
-exec mv {} newNameInASCII \;
Hope it helps, Tomas
On Wed, 2017-04-19 at 05:50 -0700, Rich Shepard wrote:
> On Tue, 18 Apr 2017, Paul Mullen wrote:
> 
> > Do you have any unusual LANG or LC_* environment variable settings?
> 
> Paul,
> 
>Nope:
> 
> $ locale
> LANG=en_US.UTF-8
> LC_CTYPE="en_US.UTF-8"
> LC_NUMERIC="en_US.UTF-8"
> LC_TIME="en_US.UTF-8"
> LC_COLLATE=C
> LC_MONETARY="en_US.UTF-8"
> LC_MESSAGES="en_US.UTF-8"
> LC_PAPER="en_US.UTF-8"
> LC_NAME="en_US.UTF-8"
> LC_ADDRESS="en_US.UTF-8"
> LC_TELEPHONE="en_US.UTF-8"
> LC_MEASUREMENT="en_US.UTF-8"
> LC_IDENTIFICATION="en_US.UTF-8"
> LC_ALL=
> 
> Rich
> ___
> PLUG mailing list
> PLUG@lists.pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set

[Rich Shepard]

On Tue, 18 Apr 2017, Paul Mullen wrote:

> Do you have any unusual LANG or LC_* environment variable settings?

Paul,

   Nope:

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE=C
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Rich
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set

[Paul Mullen]

On Tue, Apr 18, 2017 at 05:09:28PM -0700, Rich Shepard wrote:
>Actually, it does not do the job. The output from --charset='us-ascii' and
> --charset='utf8' have each line enclosed by ASCII ESC codes and unreadable.
> Wonder how I can get the pretty line drawing output in a Latin character set.

Do you have any unusual LANG or LC_* environment variable settings?


-- 
Paul Mullen
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set [RESOLVED]

[Rich Shepard]

On Tue, 18 Apr 2017, Rich Shepard wrote:

> Aha! Re-reading the man page I see the option I missed before: --charset.
> That'll do the job.

   Actually, it does not do the job. The output from --charset='us-ascii' and
--charset='utf8' have each line enclosed by ASCII ESC codes and unreadable.
Wonder how I can get the pretty line drawing output in a Latin character set.

Rich

___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] 'tree' output not in US-ASCII character set [RESOLVED]

[Rich Shepard]

On Tue, 18 Apr 2017, Rich Shepard wrote:

> Can someone suggest where I can learn why tree produces output in this
> character set and whether it's possible to change that to plain ASCII?

   Aha! Re-reading the man page I see the option I missed before: --charset.
That'll do the job.

Rich
___
PLUG mailing list
PLUG@lists.pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug