Re: shell-expand-line drops quotation marks [FIXED]

2016-11-03 Thread Dabrien 'Dabe' Murphy 

On 11/3/16, 4:21 PM, Chet Ramey wrote:

Quote Removal makes sense during command EXECUTION (since you wouldn't want
your quotes passed in with the arguments) but it doesn't make sense during
(readline) EDITING, IMHO...


OK. So let's talk about a mechanism to provide alternate behavior.  The
conventional way to do that in the context of key bindings is to modify
behavior based on whether or not the user supplies a numeric argument
(e.g., M-1M-C-e).


As if Esc + Control-E wasn't already enough of a contortion!  «grin»

I've actually bound `shell-expand-line` to ^X+Tab --- sort of an über 
expansion, if you will...  I guess one could always add Yet Another 
level of indirection:


prompt% bind '"\C-x\C-i":"\e1\e\C-e"'   # Ow, my brain!



At the very least, I would expect `shell-expand-line` to be more or less
idempotent; expanding the line multiple times shouldn't change the behavior
of the command that actually gets executed:


This is not a reasonable expection.  Think of it as running a command
through `eval' several times.


prompt% echo $'\007'


prompt% eval eval eval eval eval echo $'\007'


(Okay, so I know you were thinking more along the lines of `eval $(eval 
$(eval echo echo echo 'Hi There!') )` but I don't think anybody would 
expect that to be repeatable...)


I will have to think a little bit more, however, about what I mean when 
I do:


prompt% foo="one two"   # two words
prompt% bar='$foo'  # bare reference
prompt% baz='"$foo"'# double-quoted reference
prompt% bad=\'\$foo\'   # single-quoted reference

Currently I don't like *EITHER* scenario:

old% echo $bad   []
old% echo '$foo' []
old% echo $foo   []
old% echo one two

new% echo $bad   []
new% echo '$foo' []
new% echo 'one two'

In my mind, '$foo' would be unexpandable...  Ditto:

prompt% echo \$foo   # Should not change anything...


 prompt% alias ls="ls -F"
 prompt% ls []
 prompt% ls -F []
 prompt% ls -F -F []
 prompt% ls -F -F -F


This is actually the correct behavior.  `ls' has an alias, and that alias
is expanded.


I don't disagree that that's what it's doing, but it seems like it could 
be ensmartened, somehow, to recognize when an alias would expand to 
itself...


I'm also on the fence about whether I would prefer:

prompt% alias zz=yy
prompt% alias yy=xx
prompt% alias xx=ls

prompt% zz []
prompt% yy

or:

prompt% zz []
prompt% ls

THAT'S the kind of thing I'd expect a numeric prefix argument to 
control: whether to recurse or not...  And the logic already exists to 
prevent the following circular definition from getting stuck in an 
infinite loop; I would imagine it could be reused to short-circuit the 
expansion, as well:


prompt% alias xx=yy
prompt% alias yy=zz
prompt% alias zz=xx

prompt% xx
-bash: xx: command not found


Nice catch.  This is pretty close, and gets you most of the way you want
to go. You'd also like a way to inhibit process and command substitution
during shell-expand-line and allow those to be performed as part of
execution to avoid side effects.


Process substitution, yeah, I can't actually imagine it ever being 
useful to actually SEE the "/dev/fd/##" --- any my fix actually inhibits 
it already...  Though after you mentioned it, I was actually kind of 
surprised to see the old behavior even worked at all:


old% cat <(echo Hi There) []
old% cat /dev/fd/63   []
Hi There

COMMAND substitution, however...  I could see that being useful:

prompt% echo server-$(date +%Y%m%d).log  []
prompt% echo server-20161103.log

Or here's a more practical example:

prompt% vi $(grep -ri -l old_text lib/)
prompt% git add !*# WHOOPS! "old_text" doesn't match anymore

prompt% vi $(grep -ri -l old_text lib/) []
prompt% vi lib/Module/foo.c lib/Module/foo.h lib/main.c
prompt% gid add !*# OKAY


As an aside, I did not expect this:

prompt% echo server-$(date +%Y%m%d).log
server-20161103.log

prompt% echo !* []
prompt% echo server-$ ( date +%Y%m%d ) .log
-bash: syntax error near unexpected token `('

But that's not actually related to shell-expand-line, itself:

prompt% echo server-$(date +%Y%m%d).log
server-20161103.log

prompt% echo !*
-bash: syntax error near unexpected token `('

### Quotes, however, seem to make things work again...

prompt% echo "server-$(date +%Y%m%d).log"
server-20161103.log

prompt% echo !*
server-20161103.log

prompt% echo !* []
prompt% echo "server-20161103.log"   # OKAY


The old behavior will remain the default for now -- it's been this way for
about 25 years, after all -- but the inhibiting-quote-removal behavior
(and probably command and process substitution as well) will be available
if the user supplies a numeric argument.


My vote would still be to provide a way to make quote preservation the 
default --- e.g., a "quote-shell-expand-line" readline variable:

diff --git a/bashline.c b/bashline.c
index 759e1c5..f5b59e6 100644
--- a/bashline.c
+++ b/bashline.c
@@ -315,6 +315,9 @@ static int dabbrev_expand_active = 0;
 #define

Re: shell-expand-line drops quotation marks [FIXED]

2016-11-03 Thread Chet Ramey 
On 11/2/16 6:03 PM, Dabrien 'Dabe' Murphy wrote:

> I know this thread
> 
> is a year old, but I do have to say I agree with the OP that
> `shell-expand-line`'s decision to perform Quote Removal seems to violate
> the Principle of Least Astonishment...
> 
> To say `shell-expand-line` "Expand[s] the line as the shell does" seems,
> shall we say, "disingenuous" — if not an outright lie...  The shell
> preserves whitespace:

As does shell-expand-line, but that's not what this example shows.  The
real issue is whether or not shell-expand-line should perform quote
removal.

> Quote Removal makes sense during command EXECUTION (since you wouldn't want
> your quotes passed in with the arguments) but it doesn't make sense during
> (readline) EDITING, IMHO...

OK. So let's talk about a mechanism to provide alternate behavior.  The
conventional way to do that in the context of key bindings is to modify
behavior based on whether or not the user supplies a numeric argument
(e.g., M-1M-C-e).
> 
>  
> Consider the following variable expansion:
> 
> prompt% foo="one two"
> prompt% argDump $foo
> ARG: 'one'
> ARG: 'two'
> 
> prompt% argDump "$foo"
> ARG: 'one two'
> 
> prompt% argDump $foo []
> prompt% argDump one two# So far, so good, actually...
> 
> prompt% argDump "$foo" []
> prompt% argDump one two# FAIL FIXED

Sure, you're just restating your argument several times.

> At the very least, I would expect `shell-expand-line` to be more or less
> idempotent; expanding the line multiple times shouldn't change the behavior
> of the command that actually gets executed:

This is not a reasonable expection.  Think of it as running a command
through `eval' several times.


> I understand it's hard to do the Right Thing sometimes:  [Still unsolved]
> 
> prompt% alias ls="ls -F"
> prompt% ls []
> prompt% ls -F []
> prompt% ls -F -F []
> prompt% ls -F -F -F

This is actually the correct behavior.  `ls' has an alias, and that alias
is expanded.

> 
> So what's the fix, you might ask?
> diff --git a/bashline.c b/bashline.c
> index 238a190..e17a49d 100644
> --- a/bashline.c
> +++ b/bashline.c
> @@ -2689,7 +2689,7 @@ shell_expand_line (count, ignore)
>/* If there is variable expansion to perform, do that as a separate
>  operation to be undone. */
>new_line = savestring (rl_line_buffer);
> -  expanded_string = expand_string (new_line, 0);
> +  expanded_string = expand_string (new_line, 1);
>FREE (new_line);
>if (expanded_string == 0)
> {

Nice catch.  This is pretty close, and gets you most of the way you want
to go. You'd also like a way to inhibit process and command substitution
during shell-expand-line and allow those to be performed as part of
execution to avoid side effects.

> If you're really concerned that people are actually relying on the old
> behavior, I'm sure it would be easy to create some sort of
> "shell-expand-preserve-quotes" readline variable, or some such...  Show me
> where to submit a Pull Request and I'd be happy to whip one up!  :-D

The old behavior will remain the default for now -- it's been this way for
about 25 years, after all -- but the inhibiting-quote-removal behavior
(and probably command and process substitution as well) will be available
if the user supplies a numeric argument.

> PS — Another example where `shell-expand-line` decidedly does NOT "expand
> the line as the shell does" is with globs and tilde prefixes, but I'm aware
> this is a known limitation:
> 
> prompt% echo ~/.bash* []

It's true: globbing is not one of the shell word expansions that this
function performs, or has ever performed.  There's a separate key binding
that does this.

Tilde expansion is a little trickier, since shell-expand-line does not run
the line buffer through the shell parser to break it into words and tilde
expansion only happens at the start of a word.  (If you were to start the
command with an unquoted tilde, you'd find tilde expansion is performed.)

That's one of the reasons there is a separate key binding to perform tilde
expansion on the current word.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

shell-expand-line drops quotation marks [FIXED]

2016-11-02 Thread Dabrien 'Dabe' Murphy 
[NOTE]  Below is a message I started to write listing a whole slew of 
cases where `shell-expand-line` didn't Do What I Mean.


After familiarizing myself with the source code, however, I was 
pleasantly surprised to discover that there's actually a one line one 
character fix for almost every case I came up with(!)


So, without further ado...

    8<    8<    Cut Here    8<    8<  

I know this thread 
 
is a year old, but I do have to say I agree with the OP that 
`shell-expand-line`'s decision to perform Quote Removal seems to violate 
the Principle of Least Astonishment...


To say `shell-expand-line` "Expand[s] the line as the shell does" seems, 
shall we say, "disingenuous" --- if not an outright lie...  The shell 
preserves whitespace:


### Helper Function ###
prompt% argDump() { while [[ $# -gt 0 ]]; do echo "ARG: '$1'"; 
shift; done; }


prompt% argDump "one two three"
ARG: 'one two three'

whereas `shell-expand-line` does not.

prompt% argDump "one two three" []
prompt% argDump one two three   # FAIL FIXED
ARG: 'one'
ARG: 'two'
ARG: 'three'


Quote Removal makes sense during command EXECUTION (since you wouldn't 
want your quotes passed in with the arguments) but it doesn't make sense 
during (readline) EDITING, IMHO...



Consider the following variable expansion:

prompt% foo="one two"
prompt% argDump $foo
ARG: 'one'
ARG: 'two'

prompt% argDump "$foo"
ARG: 'one two'

prompt% argDump $foo []
prompt% argDump one two# So far, so good, actually...

prompt% argDump "$foo" []
prompt% argDump one two# FAIL FIXED

Conversely: [This is one case I have yet to solve; I imagine it involves 
a carefully placed `sh_backslash_quote_for_double_quotes()`, but I 
haven't worked it out, yet...]


prompt% foo='"one two"'# Double quotes inside single quotes
prompt% echo $foo
"one two"

prompt% argDump $foo
ARG: '"one'
ARG: 'two"'# I'm happy with that...
prompt% argDump $foo []
prompt% argDump "one two"
ARG: 'one two' # ARGH!!


Command Injection, anyone?

prompt% bar='; date'
prompt% echo $bar [ ]
Wed Nov  2 15:28:29 EDT 2016# Fool me once...

prompt% echo "$bar" [ ]   # Better Use Protection!
Wed Nov  2 15:28:41 EDT 2016# FAIL FIXED


At the very least, I would expect `shell-expand-line` to be more or less 
idempotent; expanding the line multiple times shouldn't change the 
behavior of the command that actually gets executed:


prompt% echo $'\007'
# Generates ^G (BEL)

prompt% echo $'\007' [  ]
# Becomes...
prompt% echo $\007
prompt% echo $007
prompt% echo -bash07 # WTF??! How'd $0 get in there? FIXED


Ditto history expansion:

prompt% echo $'\007'
prompt% echo !$ []
prompt% echo $\007# FAIL FIXED
$007


I understand it's hard to do the Right Thing sometimes: [Still unsolved]

prompt% alias ls="ls -F"
prompt% ls []
prompt% ls -F []
prompt% ls -F -F []
prompt% ls -F -F -F
...


I thought that maybe prepending "command" or "\" might help, but you'd 
still have:


prompt% alias ls="ls -F"
prompt% alias qq="ls"
prompt% qq /bin/bash
/bin/bash*# with '*', good

prompt% alias qq="\ls"# Note the backslash
prompt% qq /bin/bash
/bin/bash # zero '*', also good

prompt% qq /bin/bash []
prompt% ls /bin/bash
/bin/bash*# WITH '*'... BAD :-( FIXED

(It's worth noting that the shell itself is smart enough to figure out 
when to stop expanding aliases; it seems like that logic would be useful 
here, as well...)



And lastly, even when you ARE doing something "correctly", you *STILL* 
have to watch out for pitfalls:


prompt% alias su='sudo -sEp '\''[sudo] password for %p: '\'''

prompt% su
[sudo] password for dabe: 

prompt% su []
prompt% sudo -sEp [sudo] password for %p:# FAIL FIXED
[sudo]
/bin/bash: password: command not found


So what's the fix, you might ask?
diff --git a/bashline.c b/bashline.c
index 238a190..e17a49d 100644
--- a/bashline.c
+++ b/bashline.c
@@ -2689,7 +2689,7 @@ shell_expand_line (count, ignore)
   /* If there is variable expansion to perform, do that as a separate
 operation to be undone. */
   new_line = savestring (rl_line_buffer);
-  expanded_string = expand_string (new_line, 0);
+  expanded_string = expand_string (new_line, 1);
   FREE (new_line);
   if (expanded_string == 0)
{
If you're really concerned that people are actually relying on the old 
behavior, I'm sure it would be easy to create some sort of 
"shell-expand-preserve-quotes" readline variable, or some such...  Show 
me where to submit a Pull Request and I'd be happy to whip one up!