Bug#891381: vt100 line-drawing is a bad idea here

[Paul Wise]

On Tue, 2018-02-27 at 10:11 -0500, Antoine Beaupré wrote:

> ky, i'll fix the damn thing. :)

Thanks :)

> in the meantime, you can apply this on top of 1.2.0.

Huh, did not know it was possible to insert attachments in the middle
of a mail body...

> not sure i want to roll yet another trivial release just for this, do
> you mind if i wait for other/more bug reports first?

No worries, I don't intend to use it outside the terminal anyway,
was just arguing for correctness :)

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Bug#891381: vt100 line-drawing is a bad idea here

[Antoine Beaupré]

Control: tags -1 +pending
Control: found -1 1.2.0

On 2018-02-27 09:37:03, Paul Wise wrote:
> On Mon, 2018-02-26 at 20:31 -0500, Antoine Beaupré wrote:
>
>> Why can't you use --no-colors here?
>
> I definitely can, but I should not have to. The convention for tools
> that use colour is to gate that on isatty, undertime should do that.

ky, i'll fix the damn thing. :)

in the meantime, you can apply this on top of 1.2.0.

>From 7c69544fc2c6a266ff5a4163242db15f8b3c58ae Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= 
Date: Tue, 27 Feb 2018 10:05:08 -0500
Subject: [PATCH] do not output colors outside of terminals (Closes: #891381)

Finally acquiese to pabs' arguments that a piped program shouldn't
produce escape sequences. My argument was that --no-colors could be
used to fix the behavior, but it's true that the proper behavior by
default would be better, and we can always force colors with --colors
thanks to my shiny NegateAction (I *KNEW* this would be useful
eventually).
---
 undertime | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/undertime b/undertime
index e22fae4..9f1c352 100755
--- a/undertime
+++ b/undertime
@@ -33,6 +33,7 @@ along with this program.  If not, see .
 import argparse
 import datetime
 import logging
+import sys
 
 # also considered colorama and crayons
 # 1. colorama requires to send reset codes. annoying.
@@ -97,7 +98,7 @@ def arg_parser():
 help='end of working day, in hours [default: %(default)s]')
 parser.add_argument('--date', '-d', default=None, metavar='WHEN',
 help='target date for the meeting, supports arbitrary dates like "in two weeks" [default: now]')
-parser.add_argument('--colors', '--no-colors', action=NegateAction,
+parser.add_argument('--colors', '--no-colors', action=NegateAction, default=sys.stdout.isatty(),
 help='show colors [default: %(default)s]')
 parser.add_argument('--default-zone', '--no-default-zone', action=NegateAction,
 help='show current timezone first [default: %(default)s]')
-- 
2.11.0


not sure i want to roll yet another trivial release just for this, do
you mind if i wait for other/more bug reports first?

a.

-- 
My passionate sense of social justice and social responsibility has
always contrasted oddly with my pronounced lack of need for direct
contact with other human beings and communities. I am truly a "lone
traveler" and have never belonged to my country, my home, my friends,
or even my immediate family, with my whole heart; in the face of all
these ties, I have never lost a sense of distance and a need for
solitude.
   - Albert Einstein

Bug#891381: vt100 line-drawing is a bad idea here

[Paul Wise]

On Mon, 2018-02-26 at 20:31 -0500, Antoine Beaupré wrote:

> Why can't you use --no-colors here?

I definitely can, but I should not have to. The convention for tools
that use colour is to gate that on isatty, undertime should do that.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Bug#891381: vt100 line-drawing is a bad idea here

[Antoine Beaupré]

On 2018-02-27 08:20:14, Paul Wise wrote:
> On Mon, 2018-02-26 at 09:22 -0500, Antoine Beaupré wrote:
>
>> I am not sure i see the point of defaulting to "no colors" for !isatty.
>
> If I redirect output to a file and then load that in my text editor
> then I'm going to get icky terminal escape sequences in it.
>
> Personally I don't consider this bug fixed until colours are gated on
> isatty by default. People who want to pipe colours around can use
> --color or pipetty (from colorized-logs).

Why can't you use --no-colors here?

-- 
If it's important for you, you'll find a way.
If it's not, you'll find an excuse.
- Unknown

Bug#891381: vt100 line-drawing is a bad idea here

[Paul Wise]

On Mon, 2018-02-26 at 09:22 -0500, Antoine Beaupré wrote:

> I am not sure i see the point of defaulting to "no colors" for !isatty.

If I redirect output to a file and then load that in my text editor
then I'm going to get icky terminal escape sequences in it.

Personally I don't consider this bug fixed until colours are gated on
isatty by default. People who want to pipe colours around can use
--color or pipetty (from colorized-logs).

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Bug#891381: vt100 line-drawing is a bad idea here

[Antoine Beaupré]

Control: tags -1 +pending

On 2018-02-25 22:17:36, Adam Borowski wrote:
> On Sun, Feb 25, 2018 at 01:06:15PM -0500, Antoine Beaupré wrote:
>> On 2018-02-25 02:29:28, Adam Borowski wrote:
>> >> undertime prints terminal escape sequences to pipes and files.
>> >
>> > There are two types of escape sequences here:
>> >
>> > • SGR.  These make sense, as undertime uses color to convey its output.
>> > A plain text interface would be a reasonable request but is probably a 
>> > feature
>> > (anarcat: you can detect non-terminals by !isatty(1) ).
>> 
>> What's SGR? Do you mean:
>> 
>> https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_(Select_Graphic_Rendition)_parameters
>
> Yeah -- "\e[…m", in your case bold and colors.
>
>> If so, could you expand on what you would expect this would look like?
>> keep in mind that undertime doesn't generate those escape sequences on
>> its own: it passes a list of rows to the terminaltables library and lets
>> it handle that on its own.
>
> I guess that skipping colors would work then.

Skipping colors doesn't fix the problem described by pabs in the
original bug report:

[1014]anarcat@curie:undertime$ ./undertime --no-colors | less
qqqk
  EST  
qqqu
 00:00 
 01:00 
 02:00 
 03:00 
 04:00 
 05:00 
 06:00 
 07:00 
 08:00 
 08:51 
 09:00 
 10:00 
 11:00 
 12:00 
 13:00 
 14:00 
 15:00 
 16:00 
 17:00 
 18:00 
 19:00 
 20:00 
 21:00 
 22:00 
 23:00 
└───┘
[1015]▒┼▒⎼␌▒├@␌┤⎼␋␊:┤┼␍␊⎼├␋└␊$ 

>> undertime is written in python, so:
>> 
>> no Python documentation found for 'isatty'
>> 
>> There is, of course, os.isatty():
>> 
>> https://docs.python.org/3/library/os.html#os.isatty
>
> Yeah, using Python is merely a penance rather than a crime (unlike, say,
> node.js), so usual POSIX interfaces are obvious to find.

Your faith in POSIX is impressive. ;)

>> ..and sys.stdout.isatty(). But isn't this something that should be
>> handled by the underlying terminaltables library?
>
> I think most low- and mid-level libraries assume that the program does such
> logic rather than relying on the library.  In your case, I'd recommend
> defaulting to --colors/--no-colors based on isatty(1) -- this is what the
> vast majority of programs that use colors for highlighting do.
>
> The difference in doing this yourself is, if you decide to convey
> information that you currently give via colors by some other means (such as
> a '*') in plain-text output, you don't need to rely on an interface the
> library might or might not provide.
>
> And, it takes a single line of code to default --colors/--no-colors to
> isatty.  Which is less than what we already spent on this discussion. :)

It's actually more than one line, because of a small bug in
NegateAction, which forcibly overrides the default:

diff --git a/undertime b/undertime
index f28c691..c12b6b8 100755
--- a/undertime
+++ b/undertime
@@ -33,6 +33,7 @@ along with this program.  If not, see 
.
 import argparse
 import datetime
 import logging
+import sys
 
 # also considered colorama and crayons
 # 1. colorama requires to send reset codes. annoying.
@@ -71,9 +72,9 @@ class NegateAction(argparse.Action):
 
 def __init__(self, option_strings, *args, **kwargs):
 '''set default depending on the first argument'''
-default = not option_strings[0].startswith(self.negative)
+kwargs['default'] = kwargs.get('default', not 
option_strings[0].startswith(self.negative))
 super(NegateAction, self).__init__(option_strings, *args,
-   default=default, nargs=0, **kwargs)
+   nargs=0, **kwargs)
 
 def __call__(self, parser, ns, values, option):
 '''set the truth value depending on whether
@@ -92,7 +93,7 @@ def arg_parser():
 help='end of working day, in hours [default: 
%(default)s]')
 parser.add_argument('--date', '-d', default=None,
 help='target date for the meeting, supports arbitrary 
dates like "in two weeks" [default: now]')
-parser.add_argument('--colors', '--no-colors', action=NegateAction,
+parser.add_argument('--colors', '--no-colors', action=NegateAction, 
default=sys.stdout.isatty(),
 help='show colors [default: %(default)s]')
 parser.add_argument('--default-zone', '--no-default-zone', 
action=NegateAction,
 help='show current timezone first [default: 
%(default)s]')

But even then: this doesn't work, as --no-colors still shows the bug.

What *does* fix the issue is using unicode, as you mentioned and as I
suggested in the upstream bug report:

https://github.com/Robpol86/terminaltables/issues/59

What's interesting is that there *is* a style that outputs unicode
already: the "DoubleTable" style:

https://robpol86.github.io/terminaltables/doubletable.html

>> > • VT100 line drawing.  These codes shouldn't be used these days: there's a
>> > debate whether they're allowed within an U

Bug#891381: vt100 line-drawing is a bad idea here

[Adam Borowski]

On Sun, Feb 25, 2018 at 01:06:15PM -0500, Antoine Beaupré wrote:
> On 2018-02-25 02:29:28, Adam Borowski wrote:
> >> undertime prints terminal escape sequences to pipes and files.
> >
> > There are two types of escape sequences here:
> >
> > • SGR.  These make sense, as undertime uses color to convey its output.
> > A plain text interface would be a reasonable request but is probably a 
> > feature
> > (anarcat: you can detect non-terminals by !isatty(1) ).
> 
> What's SGR? Do you mean:
> 
> https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_(Select_Graphic_Rendition)_parameters

Yeah -- "\e[…m", in your case bold and colors.

> If so, could you expand on what you would expect this would look like?
> keep in mind that undertime doesn't generate those escape sequences on
> its own: it passes a list of rows to the terminaltables library and lets
> it handle that on its own.

I guess that skipping colors would work then.

> undertime is written in python, so:
> 
> no Python documentation found for 'isatty'
> 
> There is, of course, os.isatty():
> 
> https://docs.python.org/3/library/os.html#os.isatty

Yeah, using Python is merely a penance rather than a crime (unlike, say,
node.js), so usual POSIX interfaces are obvious to find.

> ..and sys.stdout.isatty(). But isn't this something that should be
> handled by the underlying terminaltables library?

I think most low- and mid-level libraries assume that the program does such
logic rather than relying on the library.  In your case, I'd recommend
defaulting to --colors/--no-colors based on isatty(1) -- this is what the
vast majority of programs that use colors for highlighting do.

The difference in doing this yourself is, if you decide to convey
information that you currently give via colors by some other means (such as
a '*') in plain-text output, you don't need to rely on an interface the
library might or might not provide.

And, it takes a single line of code to default --colors/--no-colors to
isatty.  Which is less than what we already spent on this discussion. :)

> > • VT100 line drawing.  These codes shouldn't be used these days: there's a
> > debate whether they're allowed within an UTF-8 locale, as the relevant
> > standard seems to say no.
> >
> > Also, tools like "less -R", which support colors well, don't handle such
> > line drawing.  This includes my ansi2txt and ansi2html

> So this sounds like a total nightmare that makes timezones and GUI
> toolkits look like an awesome time at the beach. In fact, this makes me
> want to rewrite the whole thing to *not* output on the terminal and
> instead use Kivy or some other gaming engine to output a beautiful
> interface like http://worldchatclock.com/ instead.

That's nowhere near the worst quirk of Unix terminals -- we're talking about
something that has evolved with direct ancestry of 19th century design.

On the other hand, a terminal interface is massively more useful for our
kind of folks than some pretty but fragile web stuff.

> Again: I didn't write those escape sequences, I just use terminaltables
> blindly because it supports colors neatly. I have found other
> alternatives, documented in the source here:

> Let me know when you're done. :p More seriously, underline is
> (deliberately) a small program: 200 lines, and half of that is usage
> instructions, comments and metadata.

> I forwarded this upstream, and, for what it's worth, there are *three*
> pull requests to implement unicode tables in terminaltables
> already. Someone(tm) needs to review those and patch terminaltables in
> Debian to include the right fix.

Yeah, it's reasonable to punt this logic there.  Unlike colors/no-colors,
such implementation details have no place in a program like yours, it's the
library what's supposed to know such things.

> I'm tempted to just reassign this bug and #891378 to the
> python-terminaltables project, because it's not really a bug in
> undertime itself...

Exactly.  We have two issues here, though, so I'd recommend doing the
--no-colors part yourself (resulting in no SGR), and reassign line drawing
(VT100/Unicode).


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄ A master species delegates.

Bug#891381: vt100 line-drawing is a bad idea here

[Antoine Beaupré]

On 2018-02-25 02:29:28, Adam Borowski wrote:
>> undertime prints terminal escape sequences to pipes and files.
>
> There are two types of escape sequences here:
>
> • SGR.  These make sense, as undertime uses color to convey its output.
> A plain text interface would be a reasonable request but is probably a feature
> (anarcat: you can detect non-terminals by !isatty(1) ).

What's SGR? Do you mean:

https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_(Select_Graphic_Rendition)_parameters

If so, could you expand on what you would expect this would look like?
keep in mind that undertime doesn't generate those escape sequences on
its own: it passes a list of rows to the terminaltables library and lets
it handle that on its own.

undertime is written in python, so:

no Python documentation found for 'isatty'

There is, of course, os.isatty():

https://docs.python.org/3/library/os.html#os.isatty

..and sys.stdout.isatty(). But isn't this something that should be
handled by the underlying terminaltables library?

> • VT100 line drawing.  These codes shouldn't be used these days: there's a
> debate whether they're allowed within an UTF-8 locale, as the relevant
> standard seems to say no.  I for one consider this interpretation bogus,
> especially as it forces programs to know about locales even if they don't
> have a reason to care, and _most_ terminal authors think so as well -- but
> "most" doesn't mean all.  Here's some information:
> https://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/utf8-plus-vt100.html
> and putty is one of terminals that does otherwise.
>
> Also, tools like "less -R", which support colors well, don't handle such
> line drawing.  This includes my ansi2txt and ansi2html -- understanding
> these codes would be a reasonable feature, but, like less' author, I haven't
> either thought nor bothered to implement that yet.  Thus, let's assume those
> who claim supporting them with UTF-8 is wrong are right :þ .
>
> This leaves two options for you: either you can detect if the locale is
> UTF-8 and switch codes you output accordingly, or you can consider this a
> waste of your time and just output Unicode unconditionally.  As the latter
> works cleanly with redirection and other non-terminal receivers, this is
> what I'd suggest.  It's overdue to declare ancient locales unsupported.

So this sounds like a total nightmare that makes timezones and GUI
toolkits look like an awesome time at the beach. In fact, this makes me
want to rewrite the whole thing to *not* output on the terminal and
instead use Kivy or some other gaming engine to output a beautiful
interface like http://worldchatclock.com/ instead.

Again: I didn't write those escape sequences, I just use terminaltables
blindly because it supports colors neatly. I have found other
alternatives, documented in the source here:

https://gitlab.com/anarcat/undertime/blob/master/undertime#L46

All of those are in Debian, so it's not a problem to switch, but before
I start messing around with that stuff, I'd love it if someone could do
a little research and see how, exactly, we're supposed output a table,
in Python, on a terminal. When you do, keep in mind that this is only
part of the problem... We also need to:

 * have multi-row cells: https://gitlab.com/anarcat/undertime/issues/2
 * support colors, keeping in mind bugs like #891378
 * support long names (like America/Los_Angeles) without exploding
 * support unicode
 * fallback properly on pipes

Let me know when you're done. :p More seriously, underline is
(deliberately) a small program: 200 lines, and half of that is usage
instructions, comments and metadata. So it should be fairly easy to
hack...

Thanks for the bug report!

A.

-- 
My passionate sense of social justice and social responsibility has
always contrasted oddly with my pronounced lack of need for direct
contact with other human beings and communities. I am truly a "lone
traveler" and have never belonged to my country, my home, my friends,
or even my immediate family, with my whole heart; in the face of all
these ties, I have never lost a sense of distance and a need for
solitude.
   - Albert Einstein

Bug#891381: vt100 line-drawing is a bad idea here

[Adam Borowski]

> undertime prints terminal escape sequences to pipes and files.

There are two types of escape sequences here:

• SGR.  These make sense, as undertime uses color to convey its output.
A plain text interface would be a reasonable request but is probably a feature
(anarcat: you can detect non-terminals by !isatty(1) ).

• VT100 line drawing.  These codes shouldn't be used these days: there's a
debate whether they're allowed within an UTF-8 locale, as the relevant
standard seems to say no.  I for one consider this interpretation bogus,
especially as it forces programs to know about locales even if they don't
have a reason to care, and _most_ terminal authors think so as well -- but
"most" doesn't mean all.  Here's some information:
https://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/utf8-plus-vt100.html
and putty is one of terminals that does otherwise.

Also, tools like "less -R", which support colors well, don't handle such
line drawing.  This includes my ansi2txt and ansi2html -- understanding
these codes would be a reasonable feature, but, like less' author, I haven't
either thought nor bothered to implement that yet.  Thus, let's assume those
who claim supporting them with UTF-8 is wrong are right :þ .

This leaves two options for you: either you can detect if the locale is
UTF-8 and switch codes you output accordingly, or you can consider this a
waste of your time and just output Unicode unconditionally.  As the latter
works cleanly with redirection and other non-terminal receivers, this is
what I'd suggest.  It's overdue to declare ancient locales unsupported.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄ A master species delegates.