bug#22696: ls output changes considered unacceptable

[Assaf Gordon]

Hello,

On 2016-02-17 9:46 a.m., Mike Hodson wrote:

On Tue, Feb 16, 2016 at 6:45 AM, Bernhard Voelker  
wrote:

On 02/16/2016 11:50 AM, Jason A. Donenfeld wrote:
[...] We don't want those single quotes.




We created a summary of common issues and FAQ
regarding the quoting change in ls(1):
  https://www.gnu.org/software/coreutils/quotes.html

If there is an issue that is not addressed there,
please send an email to coreut...@gnu.org .

regards,
 - assaf

bug#22696: ls output changes considered unacceptable

[Mike Hodson]

>On Tue, Feb 16, 2016 at 6:45 AM, Bernhard Voelker  
>wrote:
>> On 02/16/2016 11:50 AM, Jason A. Donenfeld wrote:
>> [...] We don't want those single quotes.
>
>
> Who exactly is "we"?
>
> I can only speak for myself: I'm don't really care too much about
> what output format is choosen.
>
> Have a nice day,
> Berny
>

As part of this community who:
1: personally does not not particularly care about how 'ls' output looks
2: knows how to change it
3: but also knows how many people worldwide do not like such change,
or may not appreciate the subtleties of the change, or know
immediately how to change it if their linux distro hasn't already
'worked around this usability issue',

I decided to take 10 minutes of my time to correlate a list of just
who the "we" is:

http://www.eenyhelp.com/bug-813164-fact-dangerous-help-215955262.html
http://osdir.com/ml/general/2016-02/msg21069.html
https://www.reddit.com/r/linux/comments/457ptk/ls_default_to_quotingshellescape_for_output_to/
http://unix.stackexchange.com/questions/258679/why-is-ls-suddenly-surrounding-items-with-spaces-in-single-quotes
http://linuxmd.net/forum/komandnaya-stroka-terminal/261-gnu-coreutils-8-25-ne-tak-strashen-kak-kazhetsya
https://github.com/thestinger/termite/issues/293
https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1394230.html
'revert'
https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1394231.html
'ls -N not a solution'

Mike

bug#22696: ls output changes considered unacceptable

[isabella parakiss]

On 2/17/16, Mike Frysinger  wrote:
> On 17 Feb 2016 01:41, isabella parakiss wrote:
>> On 2/17/16, Ruediger Meier  wrote:
>> > As already said in practice it is already configurable by different
>> > kind
>> > of distro-specific techniques or shell aliases. For example openSUSE
>> > has envvar LS_OPTIONS. Maybe we could support such LS_OPTIONS variable
>> > officially.
>>
>> Please no.
>> It's painful to write a script that has to fight its environment to use
>> tools with their default behaviour.
>> Fighting against GREP_OPTIONS is already extremely annoying.
>
> it's already done -- export QUOTING_STYLE to control default behavior

I wasn't aware of that, and it's been there since 1998, damn.
Thanks for letting me know.  It's only in info, not even in the man page
(unlike grep) and also pretty hard to find on the web...

> people should generally be avoiding ls in scripting anyways ...
> there is `find` and shell globs to cover most cases.

ls -l filename is still the only portable way to obtain owner, group,
mode etc.  Good thing QUOTING_STYLE doesn't affect that.


---
xoxo iza

bug#22696: ls output changes considered unacceptable

[Pádraig Brady]

On 16/02/16 12:41, Paul Eggert wrote:
> On 02/16/2016 10:48 AM, Ruediger Meier wrote:
> 
>> What about readlink, basename, mktemp ... Why they don't have a 
>> terminal mode too?
> 
> If a command is commonly used at the top level, it probably should do 
> something like 'ls' does, yes. The commands you mention aren't often 
> used for that, though, so it's not high priority to change them.

Also commands like these don't have human interactive formatting
like ls, such as column alignment etc., and thus are more programmatic.
Consequently they can leverage the new %q printf format
that was also added in coreutils 8.25

$ realpath_shell() { realpath -z "$@" | xargs -r0 printf '%q\n'; }

$ touch 'z 1' 'z 2'
$ realpath_shell z*
'/home/padraig/git/coreutils/z 1'
'/home/padraig/git/coreutils/z 2'

cheers,
Pádraig.

bug#22696: ls output changes considered unacceptable

[Mike Frysinger]

On 17 Feb 2016 01:41, isabella parakiss wrote:
> On 2/17/16, Ruediger Meier  wrote:
> > As already said in practice it is already configurable by different kind
> > of distro-specific techniques or shell aliases. For example openSUSE
> > has envvar LS_OPTIONS. Maybe we could support such LS_OPTIONS variable
> > officially.
> 
> Please no.
> It's painful to write a script that has to fight its environment to use
> tools with their default behaviour.
> Fighting against GREP_OPTIONS is already extremely annoying.

it's already done -- export QUOTING_STYLE to control default behavior

people should generally be avoiding ls in scripting anyways ...
there is `find` and shell globs to cover most cases.
-mike


signature.asc
Description: Digital signature

bug#22696: ls output changes considered unacceptable

[isabella parakiss]

On 2/17/16, Ruediger Meier  wrote:
> As already said in practice it is already configurable by different kind
> of distro-specific techniques or shell aliases. For example openSUSE
> has envvar LS_OPTIONS. Maybe we could support such LS_OPTIONS variable
> officially.

Please no.
It's painful to write a script that has to fight its environment to use
tools with their default behaviour.
Fighting against GREP_OPTIONS is already extremely annoying.


---
xoxo iza

bug#22696: ls output changes considered unacceptable

[Eric Blake]

On 02/16/2016 04:09 PM, Ruediger Meier wrote:
> On Tuesday 16 February 2016, Eric Blake wrote:
>> On 02/16/2016 03:13 PM, Ruediger Meier wrote:
>>> Do you really think that this ls output is clear to a newbie?
>>> $ ls
>>> 'a?b'  'a'$'\n''b'  axb  c  'd  e'
>>
>> A newbie isn't going to create a file with a newline in it; and the
>> others seem reasonable to me.
> 
> My mother does this all the time. It's easy with certain gui file-save 
> dialogues or file-browser rename functionalities.

Tabs, single quotes, and spaces, yes. But even in gui's, it's
surprisingly hard to create file names with a literal newline (and
that's a GOOD thing that it's hard, because someday it would be nice to
forbid newlines in filenames at all; the POSIX folks have at least
entertained a discussion of what it would entail to allow [not force,
but not forbid] a system that wants to reject newlines and non-UTF-8
filenames).

> 
> But I am the only one who looks at my mothers files using ls ...

And we're right back to: if you don't like how it looks on your system,
then YOU are in control by setting up an alias.  Just because the
out-of-the-box default doesn't match your preference doesn't mean it is
a bad out-of-the-box default.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#22696: ls output changes considered unacceptable

[Ruediger Meier]

On Tuesday 16 February 2016, Assaf Gordon wrote:
> Hello all,
>
> Regarding the recent change of default-quoting style,
> what do you think about the attached patch,
> enabling to set the default style during './configure' ?
>
> advanced users who prefer the previous behavior (but don't want to
> use a new alias or add QUOTING_STYLE envvar) can build their own
> version with the old style quoting.
>
> (this patch is a proof-of-concept, not well tested or
> syntax-checked).
>
> Comments welcomed,
>assaf

As already said in practice it is already configurable by different kind 
of distro-specific techniques or shell aliases. For example openSUSE 
has envvar LS_OPTIONS. Maybe we could support such LS_OPTIONS variable 
officially.

For example I'm always using "--time-style=long-iso" and I guess many 
people would like to use different flavours.

cu,
Rudi

bug#22696: ls output changes considered unacceptable

[Ruediger Meier]

On Tuesday 16 February 2016, Paul Eggert wrote:
> On 02/16/2016 10:48 AM, Ruediger Meier wrote:
> > If the file name _is_ readable at all, then it was printed in a
> > more readable way.
>
> Sorry, I'm not following. What do you mean by "readable at all"?

I've ment printable should be just printed as is. Please no wrong 
assumptions that any user would want to copy/paste any ls output into 
the next shell prompt.

> > Other tools
> > like less, more, texteditor, webbrowser don't print non-printable
> > chars. Why ls?
>
> If by "non-printable" you mean the character can't be displayed on
> the terminal, then these other tools typically use escape sequences
> to represent non-printable characters unambiguously, which is the
> sort of thing that 'ls' is doing now. For example, the shell command:

>printf '\3' | less
>
> puts a highlighted ^C on my screen. This is the same basic idea that
> 'ls' is now using when it puts ''$'\003' on my screen.In both cases,
> an escape sequence is being used, not merely to prevent the screen
> from being trashed, but also to represent the input unambiguously.

The _old_ ls did it like a text editor, only translating non-printable 
chars into "something". An editor which would quote any ! or ? would be 
insane (Editor developers help shell newbies!?). The new ls output 
assumes that you don't want to read file names but only want to 
copy/paste it to a shell. But if you need to copy/paste that filename 
into a text editor where you want to add this file name to C #define 
then you have to remove the useless shell quotes/escapes again.

Maybe LANG=C should give us C-style char* output!?

cu,
Rudi

bug#22696: ls output changes considered unacceptable

[Ruediger Meier]

On Tuesday 16 February 2016, Eric Blake wrote:
> On 02/16/2016 03:13 PM, Ruediger Meier wrote:
> > Do you really think that this ls output is clear to a newbie?
> > $ ls
> > 'a?b'  'a'$'\n''b'  axb  c  'd  e'
>
> A newbie isn't going to create a file with a newline in it; and the
> others seem reasonable to me.

My mother does this all the time. It's easy with certain gui file-save 
dialogues or file-browser rename functionalities.

But I am the only one who looks at my mothers files using ls ...

cu,
Rudi

bug#22696: ls output changes considered unacceptable

[Eric Blake]

On 02/16/2016 03:13 PM, Ruediger Meier wrote:
> 
> Do you really think that this ls output is clear to a newbie?
> $ ls
> 'a?b'  'a'$'\n''b'  axb  c  'd  e'

A newbie isn't going to create a file with a newline in it; and the
others seem reasonable to me.

Maybe we could make an effort to quote as "a'b" instead of 'a'\''b', if
there are no ", $, or ` within a name, while still using '' quoting for
everything else.

But I _like_ quoting by default.  It's certainly nicer than representing
$'a\nb' as a?b, and having to guess what the glob expands to.

> It doesn't matter if you like the new format or not. This quick change 
> of the default to an even untested new (never released before!) format 
> was clearly over the top. Please revert.

No, I don't think we should revert.  But I have no problem calling
attention to the fact that you can update your environment to select the
previous behavior as _your_ default, regardless of what we select for
the out-of-the-box default for someone that hasn't updated their initial
environment to request an alternate quoting mode.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#22696: ls output changes considered unacceptable

[Paul Eggert]

On 02/16/2016 02:29 PM, Assaf Gordon wrote:

Regarding the recent change of default-quoting style,
what do you think about the attached patch,
enabling to set the default style during './configure' ? 


We should endeavor to avoid behavior forks like that. Besides, such a 
change would violate the GNU Coding Standards, which say, "No '--enable' 
option should ever substitute one useful behavior for another useful 
behavior.  The only proper use for '--enable' is for questions of 
whether to build part of the program

or exclude it."

bug#22696: ls output changes considered unacceptable

[Assaf Gordon]

Hello all,

Regarding the recent change of default-quoting style,
what do you think about the attached patch,
enabling to set the default style during './configure' ?

advanced users who prefer the previous behavior (but don't want to use a new 
alias or add QUOTING_STYLE envvar) can
build their own version with the old style quoting.

(this patch is a proof-of-concept, not well tested or syntax-checked).

Comments welcomed,
  assaf




>From 2a216acd3a52f577880d3a8e02eb5dfc2358adf7 Mon Sep 17 00:00:00 2001
From: Assaf Gordon 
Date: Tue, 16 Feb 2016 17:20:08 -0500
Subject: [PATCH] ls: allow setting default quoting at build time

Allows changing the default quoting style for 'ls' during compilation
as a 'configure' option:

  ./configure --enable-ls-default-quoting-style=[STYLE]

where STYLE is c|literal|shell|shell-always|locale|escape.
Using 'ls --quoting-style' or QUOTING_STYLE env-var can
override the default.

'literal' was the default until version 8.24.
'shell' is the default from 8.25 onwards.

See also:
 commit 109b9220cead6e979d22d16327c4d9f8350431cc
 http://bugs.gnu.org/22696

* configure.ac: new option: --enable-ls-default-quoting-style ,
   will be #define'd in config.h
* src/ls.c: use default style value from config file.
---
 configure.ac | 24 
 src/ls.c |  2 +-
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/configure.ac b/configure.ac
index a484601..efe64ea 100644
--- a/configure.ac
+++ b/configure.ac
@@ -243,6 +243,30 @@ if test "$gl_single_binary" = 'symlinks'; then
 fi
 AM_CONDITIONAL([SINGLE_BINARY], [test "$gl_single_binary" != no])
 
+
+
+AC_ARG_ENABLE([ls-default-quoting-style],
+  [AS_HELP_STRING([--enable-ls-default-quoting-style=[literal|locale|shell|shell-always|c|escape]],
+ ['ls' default quoting style when printing to stdout,
+  can be overriden by 'ls --quoting-style=WORD' and
+  the 'QUOTING_STYLE' environment variable])],
+  [gl_ls_default_quote_style=shell_escape_quoting_style ;
+   case $enableval in
+ literal)  gl_ls_default_quote_style=literal_quoting_style ;;
+ locale)   gl_ls_default_quote_style=locale_quoting_style ;;
+ shell)gl_ls_default_quote_style=shell_escape_quoting_style ;;
+ shell-always) gl_ls_default_quote_style=shell_escape_always_quoting_style ;;
+ c)gl_ls_default_quote_style=c_quoting_style ;;
+ escape)   gl_ls_default_quote_style=escape_quoting_style ;;
+ *)  AC_MSG_ERROR([bad value $enableval for ls-default-quoting-style option.
+   Options are: literal|locale|shell|shell-always|c|escape.]) ;;
+   esac],
+  [gl_ls_default_quote_style=shell_escape_quoting_style]
+)
+AC_DEFINE_UNQUOTED([LS_DEFAULT_QUOTING_STYLE], [$gl_ls_default_quote_style], [Default quoting style for ls])
+
+
+
 AC_FUNC_FORK
 
 optional_bin_progs=
diff --git a/src/ls.c b/src/ls.c
index d976036..eea2da5 100644
--- a/src/ls.c
+++ b/src/ls.c
@@ -1581,7 +1581,7 @@ decode_switches (int argc, char **argv)
   if (isatty (STDOUT_FILENO))
 {
   format = many_per_line;
-  set_quoting_style (NULL, shell_escape_quoting_style);
+  set_quoting_style (NULL, LS_DEFAULT_QUOTING_STYLE);
   /* See description of qmark_funny_chars, above.  */
   qmark_funny_chars = true;
 }
-- 
1.9.1

bug#22696: ls output changes considered unacceptable

[Ruediger Meier]

On Tuesday 16 February 2016, Jon Stanley wrote:
> On Tue, Feb 16, 2016 at 1:48 PM, Ruediger Meier  
wrote:
> > No! IMO Newbies should learn (most painful as possible!) that
> > non-ascii filenames sucks. :) Maybe ls shouldn't show them at all
> > by default ;)
>
> I normally lurk on this list, but this leads me to *vehemently*
> disagree. This is a top FAQ amongst new users, and I think that we
> (as an open source community, I don't personally have a single commit
> in coreutils) have a responsibility to make our ecosystem welcoming
> and accessible to new users. This change, IMO, does exactly that, and
> will save countless hours of supporting new users through the
> intricate mechanisms of shell escapes and why they are necessary.
> Quite obvious to you and I, not so much to new users. Anything that
> we can do to ease their learning curve (ideally, while teaching them
> something) is IMO a positive step.

Do you really think that this ls output is clear to a newbie?
$ ls
'a?b'  'a'$'\n''b'  axb  c  'd  e'

allthough in dolphin shows up like:
12   3 4   5
a?b  a   axb   c   d  e
 b

Do you really think that a newbie will know between which quotes and 
spaces to copy/paste the right file name?

And why do we only care about newbies? Other people have used these 
commands for +20 years. To not bother the "power users" is IMO much 
more important.

I've choosed to use the command line because it's more reliable, stable 
interfaces, appearance and output. That's why I (and many others) 
choosed these command line tools rather than file-browers which look 
completely different between any release or are not even availabe on 
different systems.

It doesn't matter if you like the new format or not. This quick change 
of the default to an even untested new (never released before!) format 
was clearly over the top. Please revert.

cu,
Rudi

bug#22696: ls output changes considered unacceptable

[Jon Stanley]

On Tue, Feb 16, 2016 at 1:48 PM, Ruediger Meier  wrote:
> No! IMO Newbies should learn (most painful as possible!) that non-ascii
> filenames sucks. :) Maybe ls shouldn't show them at all by default ;)

I normally lurk on this list, but this leads me to *vehemently*
disagree. This is a top FAQ amongst new users, and I think that we (as
an open source community, I don't personally have a single commit in
coreutils) have a responsibility to make our ecosystem welcoming and
accessible to new users. This change, IMO, does exactly that, and will
save countless hours of supporting new users through the intricate
mechanisms of shell escapes and why they are necessary. Quite obvious
to you and I, not so much to new users. Anything that we can do to
ease their learning curve (ideally, while teaching them something) is
IMO a positive step.

$0.02

bug#22696: ls output changes considered unacceptable

[Paul Eggert]

On 02/16/2016 10:48 AM, Ruediger Meier wrote:
If the file name _is_ readable at all, then it was printed in a more 
readable way.


Sorry, I'm not following. What do you mean by "readable at all"?



Other tools
like less, more, texteditor, webbrowser don't print non-printable
chars. Why ls?


If by "non-printable" you mean the character can't be displayed on the 
terminal, then these other tools typically use escape sequences to 
represent non-printable characters unambiguously, which is the sort of 
thing that 'ls' is doing now. For example, the shell command:


  printf '\3' | less

puts a highlighted ^C on my screen. This is the same basic idea that 
'ls' is now using when it puts ''$'\003' on my screen. In both cases, an 
escape sequence is being used, not merely to prevent the screen from 
being trashed, but also to represent the input unambiguously.



Who says that ls outout should be copy/pastable (!_into_shell_only_!).


Some sort of escape sequence is needed to represent arbitrary file 
names. Since ls at the top level is mostly used under a shell, the 
shell's escape sequences seem to be the most useful.


why is `ls | grep "files to copy/paste"' not copy/pastable by default 
then?


We can't solve all the problems, but we can solve some of them.

What about readlink, basename, mktemp ... Why they don't have a 
terminal mode too?


If a command is commonly used at the top level, it probably should do 
something like 'ls' does, yes. The commands you mention aren't often 
used for that, though, so it's not high priority to change them.



At least you could have done it human readable like git


Yes, if the shell supports a nicer unambiguous syntax, then ls should 
probably use that instead.



IMO Newbies should learn (most painful as possible!)


I'm afraid we'll have to disagree about this one. I think we should 
encourage new users and make them feel welcome.


I can't control what stupid file names come from other users or 
downloaded from the internet.


This is partly why the new behavior is more important now than it used 
to be. Too many naive users are running 'ls' in directories from 
untrustworthy sources.

bug#22696: ls output changes considered unacceptable

[Ruediger Meier]

On Tuesday 16 February 2016, Paul Eggert wrote:
> On 02/16/2016 08:58 AM, Ruediger Meier wrote:
> > Terminal output should be human readable not machine readable.
>
> Sure, but under the old way of doing things, terminal output *wasn't*
> human-readable. For example:

If the file name _is_ readable at all, then it was printed in a more 
readable way. This is a regression.

> $ ls
> a?b  a?b  axb  c  d  e
> $ rm  a?b  c  d

Of course, non-printable chars are not correctly printed. Other tools 
like less, more, texteditor, webbrowser don't print non-printable 
chars. Why ls?

> rm: cannot remove 'd': No such file or directory

Who says that ls outout should be copy/pastable (!_into_shell_only_!). 
It's a corner case. That's why we have --quoting-style. That's fine as 
long as it's not the default.

And why is `ls | grep "files to copy/paste"' not copy/pastable by 
default then? This makes no sense.

No other command is printing filenames as annyoing as ls. What about 
readlink, basename, mktemp ...
Why they don't have a terminal mode too? I copy paste many other's 
command output very very often every day. Now ls is the only one which 
wants to help me with it ... but I hate it. I also copy/paste file 
names into emails, chat or commit messages ... not only into a _shell_ 
commandline.

At least you could have done it human readable like git, for example:

$ touch $'AAA\nBBB'
$ touch   $'AAA\x05BBB'
$ touch "other's"
$ git add *
$ git st

# Changes to be committed:
#   (use "git rm --cached ..." to unstage)
#
#   new file:   "AAA\005BBB"
#   new file:   "AAA\nBBB"
#   new file:   other's


> The new behavior is much more readable and understandable. Of course
> this is a contrived example (created via:
>
> touch 'a
> b' 'a?b' axb c 'd  e'
>
> )
> , but it's similar to situations that I run into all the time when 
> teaching newbies.New users should be better off with the new
> approach, in ordinary interactive use.

No! IMO Newbies should learn (most painful as possible!) that non-ascii 
filenames sucks. :) Maybe ls shouldn't show them at all by default ;)

If they want to use strange file names than they should use dolphin or 
konqueror. (Deleting files by click, no copy/pasting needed).

> I didn't comment on the original change, because I thought it was a
> no-brainer.

Please revert it.

> Yes, there is a backward-compatibility issue, but users 
> worried about compatibility should be using portable POSIX file names
> anyway, and 'ls' hasn't changed its behavior with portable file
> names.

I can't control what stupid file names come from other users or 
downloaded from the internet.

cu,
Rudi

bug#22696: ls output changes considered unacceptable

[Bernhard Voelker]

tag 22696 notabug invalid
thanks

bummer - accidentally opened this one by CCing bug-coreutils during
reply-all.  Let's keep this on the main list, thus closing here.

Have a nice day,
Berny

bug#22696: ls output changes considered unacceptable

[Bernhard Voelker]

On 02/16/2016 11:50 AM, Jason A. Donenfeld wrote:

And, when such a
change is made in software considered "core", by a single individual
unilaterally without extremely wide consultation of the larger
community, it is clear that a grave an unacceptable thing has
happened.


We already heard a few complaints about this.
However, I don't think your argumentation here is correct and fair:
although there was one committer - well, actually there cannot be
more -, this was publicly in discussion on the mailing list, and
no-one disagreed.
Later on during the release cycle of v8.25, the snapshot versions
were also publicly available for broad testing ... and no-one inter-
vened here either.


[...] We don't want those single quotes.


Who exactly is "we"?

I can only speak for myself: I'm don't really care too much about
what output format is choosen.

Have a nice day,
Berny