bug#24926: ls output has been made ugly

[Assaf Gordon]

Hello,

On 2016-11-12 5:27 a.m., Rüdiger Meier wrote:



On Friday 11 November 2016 21:00:23 Eric Blake wrote:

On 11/11/2016 12:26 PM, Paul Eggert wrote:

Michael Schwager wrote:

Don't you think I can see the spaces in my filenames?




We created a summary of common issues and FAQs
regarding the quoting change in ls(1):
  https://www.gnu.org/software/coreutils/quotes.html

If there is an issue that is not addressed there,
please send an email to coreut...@gnu.org .

regards,
 - assaf

bug#24926: ls output has been made ugly

[Rüdiger Meier]

On Friday 11 November 2016 21:00:23 Eric Blake wrote:
> On 11/11/2016 12:26 PM, Paul Eggert wrote:
> > Michael Schwager wrote:
> >> Don't you think I can see the spaces in my filenames?
> >
> > Not in general, no.  For example:
> >
> > $ ls --quoting-style=literal
> > a  b  c
> > $ ls
> > 'a  b'  c
> >
> > That being said, perhaps 'ls' could quote less aggressively.  If 'ls'
> > always arranges for at least two spaces between file names, for example,
> > 'ls' doesn't need to quote a name merely because it contains a space
> > surrounded by non-whitespace characters.  Come to think of it, 'ls -l'
> > need not quote file names containing spaces at all.
>
> If the idea is that the quoting is there to make copy-and-pasting into a
> shell command line easier, then there is nothing we can do that is less
> aggressive, since failing to quote spaces changes what the shell will
> do.  If the idea is that the quoting should only be added to avoid
> ambiguous situations, then maybe you are right that we can add further
> heuristics to the quoting algorithm to disable quotes on output that is
> unambiguous, even if it can't be pasted back into the shell.  Having two
> different quoting modes, where you can choose between the options, may
> be the way to go - but then you STILL have the problem of what to pick
> as the default of those two modes when neither one was explicitly
> requested.

The old behavior had not a problem at all. There was no need for re-thinking  
ls' purpose of existence, like "What is actually the idea of ls? ... if the 
idea is A, B or C then change X, Y or Z". What about "If most users like it 
as is then no need to change anything."?

Most other ls other implementations still do not have any problems. coreutils 
ls is now an unusual one by default. We have had many nice options to 
_enable_ and select different quoting styles. The only problem is that the 
default behavior changed. However this fact is ignored since months. Maybe 
you should add an FAQ, like:

Q: Why is ls ugly now?
A: It is not just ugly but also better! Imagine if you would like to 
copy/paste file names! It was possible in past using extra complicated 
options only but now even by default. Never thought about this? Just try to 
copy/paste, no need to read the ls output anymore. All your experience that 
ls was good as is since 20 years is wrong. BTW in the next release we will 
copy the ls output automatically into the clipboard without printing anything 
into the terminal (Then nobody can complain about ugly output anymore.)

cu,
Rudi

bug#24926: ls output has been made ugly

[Paul Eggert]

On 11/11/2016 05:20 PM, Pádraig Brady wrote:

BTW \u2019 might not be the best choice, as I tweeted recently
(with corresponding quotes in each example):

   It's awkward for file names to use shell quote
   It’s awkward for word regex to use right quote
   Itʼs best to use apostrophe modifier (\u02BC)


But U+02BC MODIFIER LETTER APOSTROPHE is not the right character for 
“Groovin’” in English, as that word ends in a punctuation apostrophe, 
not a letter. If this happens to be awkward for regular expressions,we 
should extend the regular-expression syntax to make it less awkward.

bug#24926: ls output has been made ugly

[Pádraig Brady]

On 12/11/16 00:24, Paul Eggert wrote:
> Michael Schwager wrote:
>> please illustrate a real life problem
> 
> In general, file names can be chosen by an adversary. This is a real-life 
> problem for me, and for many other people (whether they know it or not).
> 
> Eric Blake's arguments for leaving things alone are cogent ones.
> 
> You can try to track down the design discussion by doing a git blame and 
> looking 
> for something on the mailing list around the time the changes were installed.
> 
> Also, this may not be obvious, but you can avoid the ugliest part of the 
> quoting 
> by using the proper English-language character in your song titles. For 
> example:
> 
> $ ls
> Groovin’  'Higher Ground'  UB40
> 
> This works because ’ (U+2019 RIGHT SINGLE QUOTATION MARK), the ordinary 
> character to use in the English-language title “Groovin’”, is not subject to 
> the 
> off-putting replacement by '\'' (also, this character avoids other problems 
> when 
> cutting and pasting into the shell). This would be better than what you have 
> now. Perhaps a suggestion along these lines should be put into the coreutils 
> manual?

Note v8.26 will simplify the quoting when just traditional single quotes are 
present
by using double quotes like: "Groovin'"

BTW \u2019 might not be the best choice, as I tweeted recently
(with corresponding quotes in each example):

  It's awkward for file names to use shell quote
  It’s awkward for word regex to use right quote
  Itʼs best to use apostrophe modifier (\u02BC)

cheers,
Pádraig.

bug#24926: ls output has been made ugly

[Paul Eggert]

Michael Schwager wrote:

please illustrate a real life problem


In general, file names can be chosen by an adversary. This is a real-life 
problem for me, and for many other people (whether they know it or not).


Eric Blake's arguments for leaving things alone are cogent ones.

You can try to track down the design discussion by doing a git blame and looking 
for something on the mailing list around the time the changes were installed.


Also, this may not be obvious, but you can avoid the ugliest part of the quoting 
by using the proper English-language character in your song titles. For example:


$ ls
Groovin’  'Higher Ground'  UB40

This works because ’ (U+2019 RIGHT SINGLE QUOTATION MARK), the ordinary 
character to use in the English-language title “Groovin’”, is not subject to the 
off-putting replacement by '\'' (also, this character avoids other problems when 
cutting and pasting into the shell). This would be better than what you have 
now. Perhaps a suggestion along these lines should be put into the coreutils manual?

bug#24926: ls output has been made ugly

[Michael Schwager]

Can anyone point me to the mailing list discussion where this originally
was decided?

There are legion of coreutils users out there who are dumbfounded and who
detest this change; I see that Debian has reverted it. However, I (and
Debian) could be wrong. I'd like to see the original discussion which will
hopefully point to the well-founded reasoning for its implementation (eg,
POSIX requires it, a tremendous clamour from the user community, what have
you).

I have searched in the mailing list but not found the original point of the
decision; I don't have the proper expression foo to return it...

-- 
-Mike Schwager

bug#24926: ls output has been made ugly

[Reuti]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

Am 11.11.2016 um 22:23 schrieb Eric Blake:
> 
> Knowing the pitfalls makes it easier to justify why an output that is
> unambiguous

Will it honor alt-space at some point?


> was chosen as the new default over the previous ambiguous
> output; any further changes to the default are now a matter of
> fine-tuning about how much (or how little) decoration we can get away
> with, while still avoiding a regression to the situation of ambiguous
> output.


For copy and paste you don't need quotes around alt-space, but you don't see 
where it ends:

$ ls --quoting-style=literal
a b  a b  a  b  a   b   xx 
$ ls --quoting-style=shell
'a b'  a b  a  b  a   b   xx 
$ ls --quoting-style=shell-always
'a b'  'a b'  'a  b'  'a   b'  ' xx '
$ ls -l
total 0
- -rw-r--r-- 1 reuti staff 0 Nov 11 21:13 'a b'
- -rw-r--r-- 1 reuti staff 0 Nov 11 21:13 a b
- -rw-r--r-- 1 reuti staff 0 Nov 11 21:17 a  b
- -rw-r--r-- 1 reuti staff 0 Nov 11 21:17 a   b
- -rw-r--r-- 1 reuti staff 0 Nov 11 21:45  xx 
$ ls -l --quoting-style=shell-always
total 0
- -rw-r--r-- 1 reuti staff 0 Nov 11 21:13 'a b'
- -rw-r--r-- 1 reuti staff 0 Nov 11 21:13 'a b'
- -rw-r--r-- 1 reuti staff 0 Nov 11 21:17 'a  b'
- -rw-r--r-- 1 reuti staff 0 Nov 11 21:17 'a   b'
- -rw-r--r-- 1 reuti staff 0 Nov 11 21:45 ' xx '

Personally I like only the two options to quote always (as you expect such 
names) or never, but not to quote sometimes.

- -- Reuti
-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iEYEARECAAYFAlgmPLUACgkQo/GbGkBRnRoTsgCfeApMOM3wijQorhmpWX1y1cuJ
YYEAn2n9j0eSzoGWLVbDmg2g8l+A1Evj
=jqwR
-END PGP SIGNATURE-

bug#24926: ls output has been made ugly

[L. A. Walsh]

Eric Blake wrote:

On 11/11/2016 03:25 PM, L. A. Walsh wrote:

Eric Blake wrote:

If the idea is that the quoting is there to make copy-and-pasting into a
shell command line easier, then there is nothing we can do that is less
aggressive, since failing to quote spaces changes what the shell will
do. 


   I assume you are talking about "quoting-style=shell-always"
and not the conditional quoting you get for "quoting-style=shell"?


quoting-style=shell-always uses quotes ALWAYS, even if the quotes are
redundant.  quoting-style=shell uses quotes ONLY if the shell could
otherwise interpret it differently without the quotes.

---
I understand that.  It sounded like you were talking
about changing the default output to use quotes, which would be
a disaster.

	I know of multiple scripts that use 
"\ls -1" to get a single column of filenames for input (assuming

it is known that there are no embedded newlines in your
filenames).  For most people that is true.  If you need more 
quoting, then output strings with a \00 at the end.  It's the

only guarantee of not being used in a filename.

bug#24926: ls output has been made ugly

[Eric Blake]

On 11/11/2016 03:25 PM, L. A. Walsh wrote:
> Eric Blake wrote:
>>
>> If the idea is that the quoting is there to make copy-and-pasting into a
>> shell command line easier, then there is nothing we can do that is less
>> aggressive, since failing to quote spaces changes what the shell will
>> do. 
> 
>I assume you are talking about "quoting-style=shell-always"
> and not the conditional quoting you get for "quoting-style=shell"?

quoting-style=shell-always uses quotes ALWAYS, even if the quotes are
redundant.  quoting-style=shell uses quotes ONLY if the shell could
otherwise interpret it differently without the quotes.  Reusing Paul's
example:

$ \ls --quoting-style=shell
'a  b'  c
$ \ls --quoting-style=shell-always
'a  b'  'c'

Note that 'a  b' needed quoting under both styles, but 'c' was conditional.

>>  If the idea is that the quoting should only be added to avoid
>> ambiguous situations, then maybe you are right that we can add further
>> heuristics to the quoting algorithm to disable quotes on output that is
>> unambiguous, even if it can't be pasted back into the shell.  Having two
>> different quoting modes, where you can choose between the options, may
>> be the way to go - but then you STILL have the problem of what to pick
>> as the default of those two modes when neither one was explicitly
>> requested.
>>   
> 
>Seems like the default is to not put quotes.  That's what
> is used now.  Why would you break it?

Paul was suggesting _yet another mode_ (which someone would have to
write patches for), which avoids quotes in all situations where the
output is unambiguous (a file 'a b' is unambiguous in 'ls' output even
without quotes, since the ambiguity only arises if you have two spaces;
but a file 'a  b' would still need quotes, as would a file that itself
contains literal quotes in the name), but also pointing out that the
definition of "not ambiguous" may be context sensitive (the ambiguity of
plain 'ls' [two spaces, embedded newline, or something that uses the
same mechanism we use for escaping the other ambiguous cases] differs
from the ambiguity of 'ls -l' [embedded newline, or something that uses
the same mechanism we use for escaping the other ambiguous cases],
because two spaces is ambiguous in one but not the other).

Personally, I think that trying hard to avoid quotes makes the addition
of quotes that much more surprising when hitting the corner cases, which
by their nature are already corner cases and therefore somewhat rare, so
I'd rather stick with a style that is very easy to describe ("if the
shell could misintepret it without quotes, add quotes; and this applies
regardless of the app using this quoting rule") over one that is
difficult ("if it matches potential ambiguity A, B, or C, add quotes -
but now you have to learn a different list of A, B, and C for every app
and context that has different ambiguities").

Furthermore, I _like_ quoting-style=shell (and NOT
quoting-style=shell-always) in one regards: when dealing solely with
file names that are declared portable according to POSIX,
quoting-style=shell is indistinguishable from quoting-style=literal.  It
is only on file names that are already inherently non-portable to all
possible file systems where the additional quotes calls my attention to
the potential portability issue, whereas quoting-style=always does not
have that benefit.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#24926: ls output has been made ugly

[L. A. Walsh]

Eric Blake wrote:

He didn't have to. His point was merely that with the old ls, you can
have inherently ambiguous situations.  Think of it as an exercise for
the reader to figure out ways to get into those ambiguous situations.


Good communication isn't supposed to be
a puzzle.  Creating puzzles in order for people to get your
message is, again, a way to "make points" and play games.

Sorry.  I was told to provide the means to reproduce my examples
when reporting a problem.  That wasn't done.  Poor example.
Justifying it as providing a puzzle for readers to solve is
just another game.

bug#24926: ls output has been made ugly

[L. A. Walsh]

Eric Blake wrote:

Knowing the pitfalls makes it easier to justify why an output that is
unambiguous was chosen as the new default over the previous ambiguous
output; any further changes to the default are now a matter of
fine-tuning about how much (or how little) decoration we can get away
with, while still avoiding a regression to the situation of ambiguous
output.

---
	That's bull!  You don't just randomly change 
output in programs and break compatibility.  If the user

wanted unambiguous, there are already options.

bug#24926: ls output has been made ugly

[L. A. Walsh]

Eric Blake wrote:


If the idea is that the quoting is there to make copy-and-pasting into a
shell command line easier, then there is nothing we can do that is less
aggressive, since failing to quote spaces changes what the shell will
do. 


   I assume you are talking about "quoting-style=shell-always"
and not the conditional quoting you get for "quoting-style=shell"?




 If the idea is that the quoting should only be added to avoid
ambiguous situations, then maybe you are right that we can add further
heuristics to the quoting algorithm to disable quotes on output that is
unambiguous, even if it can't be pasted back into the shell.  Having two
different quoting modes, where you can choose between the options, may
be the way to go - but then you STILL have the problem of what to pick
as the default of those two modes when neither one was explicitly requested.
  


   Seems like the default is to not put quotes.  That's what
is used now.  Why would you break it?

bug#24926: ls output has been made ugly

[Eric Blake]

On 11/11/2016 03:08 PM, L. A. Walsh wrote:
> 
> 
> Eric Blake wrote:
  touch 'a b' c
>>
>> That's your problem.  Paul did:
>>
>> $ touch 'a  b' c
> 
> He didn't list his creation command.  How
> would you know?

Because that's what worked for me to reproduce his commands.

> 
> 
>> with two spaces, not one.
> ---
> You are assuming that.  But if he didn't list how
> he created them...

He didn't have to. His point was merely that with the old ls, you can
have inherently ambiguous situations.  Think of it as an exercise for
the reader to figure out ways to get into those ambiguous situations.

Knowing the pitfalls makes it easier to justify why an output that is
unambiguous was chosen as the new default over the previous ambiguous
output; any further changes to the default are now a matter of
fine-tuning about how much (or how little) decoration we can get away
with, while still avoiding a regression to the situation of ambiguous
output.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#24926: ls output has been made ugly

[L. A. Walsh]

Eric Blake wrote:

 touch 'a b' c


That's your problem.  Paul did:

$ touch 'a  b' c


He didn't list his creation command.  How
would you know?



with two spaces, not one.

---
You are assuming that.  But if he didn't list how
he created them...




Where (or under what conditions)
do you see the "a, b and c" being
spaced apart equally?


When you intentionally create the spacing between a and b to match the
spacing between a*b and c.

---
	Why would you intentionally create something to be 
confusing?  ;-?


Oh... to make a point and win an argument.  Silly me.

bug#24926: ls output has been made ugly

[Michael Schwager]

On Fri, Nov 11, 2016 at 2:00 PM, Eric Blake  wrote:

> ...Having two
> different quoting modes, where you can choose between the options, may
> be the way to go - but then you STILL have the problem of what to pick
> as the default of those two modes when neither one was explicitly
> requested.
>

Exactly. So if you want to call ls' old behavior "broken", which I presume
that one would do to justify creating this change, then your choices boil
down to:
- Leave decades of behavior in place, and deal with "broken" output, or
- Create new, different broken output.

It boggles my mind that someone decided we should have a new default. The
output is not better, confusingly, the output is different depending on if
you send to terminal or not, and I have not seen any great desire in the
community for this change.

-- 
-Mike Schwager

bug#24926: ls output has been made ugly

[Eric Blake]

On 11/11/2016 12:26 PM, Paul Eggert wrote:
> Michael Schwager wrote:
>> Don't you think I can see the spaces in my filenames?
> 
> Not in general, no.  For example:
> 
> $ ls --quoting-style=literal
> a  b  c
> $ ls
> 'a  b'  c
> 
> That being said, perhaps 'ls' could quote less aggressively.  If 'ls'
> always arranges for at least two spaces between file names, for example,
> 'ls' doesn't need to quote a name merely because it contains a space
> surrounded by non-whitespace characters.  Come to think of it, 'ls -l'
> need not quote file names containing spaces at all.

If the idea is that the quoting is there to make copy-and-pasting into a
shell command line easier, then there is nothing we can do that is less
aggressive, since failing to quote spaces changes what the shell will
do.  If the idea is that the quoting should only be added to avoid
ambiguous situations, then maybe you are right that we can add further
heuristics to the quoting algorithm to disable quotes on output that is
unambiguous, even if it can't be pasted back into the shell.  Having two
different quoting modes, where you can choose between the options, may
be the way to go - but then you STILL have the problem of what to pick
as the default of those two modes when neither one was explicitly requested.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

bug#24926: ls output has been made ugly

[L. A. Walsh]

Michael Schwager wrote:

 I
don't have single quotes in many of my filenames, although I do in some.
  

---
   How did this get displayed?  In shell,
'foo \' bar' wouldn't be displayed correctly since you can't
use backslash to escape inside a single quoted string.


That is just plain ugly. 


   Yeah, though I might use the term "noisy" as a primary
adjective.  :-)


Paul Eggert wrote:

Michael Schwager wrote:

Don't you think I can see the spaces in my filenames?

Not in general, no.  For example:

$ ls --quoting-style=literal
a  b  c


   You must have a strange version of something -- that's not
what I see:


 touch 'a b' c
 Ishtar:/tmp/tmp> ls

a b  c
Ishtar:/tmp/tmp> ls --quoting-style=literal
a b  c
Ishtar:/tmp/tmp> ls --quoting-style=shell
'a b'  c

Where (or under what conditions)
do you see the "a, b and c" being
spaced apart equally?

bug#24926: ls output has been made ugly

[Paul Eggert]

Michael Schwager wrote:

Don't you think I can see the spaces in my filenames?


Not in general, no.  For example:

$ ls --quoting-style=literal
a  b  c
$ ls
'a  b'  c

That being said, perhaps 'ls' could quote less aggressively.  If 'ls' always 
arranges for at least two spaces between file names, for example, 'ls' doesn't 
need to quote a name merely because it contains a space surrounded by 
non-whitespace characters.  Come to think of it, 'ls -l' need not quote file 
names containing spaces at all.

bug#24926: ls output has been made ugly

[Michael Schwager]

"...we don't mean to dictate, only to improve things" -Pádraig Brady
 Feb 15 at
20:46


Somebody went and dictated ls' new behavior, and now my files with spaces
look really really ugly.

I just upgraded to Fedora 24 which has the new ls. I rsync'ed my music from
my old hard drive to my new one and noticed spurious characters. I was
looking around, trying to find out what I did wrong/what it meant/etc. I
don't have single quotes in many of my filenames, although I do in some.

Look at this directory listing:
AWOLNation  'Joe Jackson'
'Cage the Elephant' 'Joy Division'
'Calabria 2008' 'Joy Division-All Gods Angels Beware.m3u'
'Carly Rae Jepsen'  'Joy Division-Joy Division  The Peel Sessions
10 Dec 79.m3u'
'Cat Power' 'Joy Division-The Peel Sessions Unreleased
Tracks.m3u'
'Culture Club'  'Katy Perry'
'Cyndi Lauper'  'Laura Branigan'
'Daft Punk' Offspring

That is just plain ugly. Don't you think I can see the spaces in my
filenames? What do those quotes do but add cruft? Who thinks they know
better than I do? How was this decision made? (that's a rhetorical
question, I don't want an answer because I've read the arguments and I
think it's a really bad idea.) Who ever thinks that what you get is not
what is actually there is ever a good idea???

Yes, you can opt out. This is akin to what email spammers have been teling
us for years about tracking us and sticking ads in our inboxes. That "Hey,
no problem- you can always opt out..." Even if you can (and there are some
legitimate sites out there that will opt you out), the point is that you
should not stick your choices into people's faces. Offer up the
alternative... don't create an alternative, then say "Now you get it
because I say so. Don't like it? You can always change it back..." That's
insulting.

-- 
-Mike Schwager