Re: [Python-Dev] subprocess not escaping "^" on Windows

[Barry Scott]

My feeling is that the number of uses for calling cmd /c is rather limited on 
Windows.

Certainly calling out to use the CMD builtin is not to be encouraged I'd say.
Between shutil and the os module you have most of the file handling commands.
Admin tools might want to run special commands, but they are not builtins.

In all the cases where you have a command line exe to run you can avoid
calling into cmd and the associated quoting problems.

I've found that in all my windows python apps I typically end up using 
CreateProcess
and ShellExecute for the useful stuff. (I use ctypes to call them).

Is it worth changing the quoting at all? I would say not.

Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[eryk sun]

On Mon, Jan 8, 2018 at 9:26 PM, Steve Dower  wrote:
> On 09Jan2018 0744, eryk sun wrote:
>>
>> It's common to discourage using `shell=True` because it's considered
>> insecure. One of the reasons to use CMD in Windows is that it tries
>> ShellExecuteEx if CreateProcess fails. ShellExecuteEx supports "App
>> Paths" commands, file actions (open, edit, print), UAC elevation (via
>> "runas" or if requested by the manifest), protocols (including
>> "shell:"), and opening folders in Explorer. It isn't a scripting
>> language, however, so it doesn't pose the same risk as using CMD.
>> Calling ShellExecuteEx could be integrated in subprocess as a new
>> Popen parameter, such as `winshell` or `shellex`.
>
> This can also be used directly as os.startfile, the only downside being that
> you can't wait for the process to complete (but that's due to the underlying
> API, which may not end up starting a process but rather sending a message to
> an existing long-running one such as explorer.exe). I'd certainly recommend
> it for actions like "open this file with its default editor" or "browse to
> this web page with the default browser".

Yes, I forgot to mention that os.startfile can work sometimes. But
often one needs to pass command-line parameters. Also, os.startfile
also can't set a different working directory, nShow SW_* window state,
or flags such as SEE_MASK_NO_CONSOLE (prevent allocating a new
console). Rather than extend os.startfile, it seems more useful in
general to wrap ShellExecuteEx in _winapi and extend subprocess. Then
os.startfile can be reimplemented in terms of subprocess.Popen, like
os.popen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Steve Dower]

On 09Jan2018 0744, eryk sun wrote:

It's common to discourage using `shell=True` because it's considered
insecure. One of the reasons to use CMD in Windows is that it tries
ShellExecuteEx if CreateProcess fails. ShellExecuteEx supports "App
Paths" commands, file actions (open, edit, print), UAC elevation (via
"runas" or if requested by the manifest), protocols (including
"shell:"), and opening folders in Explorer. It isn't a scripting
language, however, so it doesn't pose the same risk as using CMD.
Calling ShellExecuteEx could be integrated in subprocess as a new
Popen parameter, such as `winshell` or `shellex`.


This can also be used directly as os.startfile, the only downside being 
that you can't wait for the process to complete (but that's due to the 
underlying API, which may not end up starting a process but rather 
sending a message to an existing long-running one such as explorer.exe). 
I'd certainly recommend it for actions like "open this file with its 
default editor" or "browse to this web page with the default browser".


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[eryk sun]

On Sun, Jan 7, 2018 at 6:48 PM, Christian Tismer  wrote:
> That is true.
> list2cmdline escapes partially, but on NT and Windows10, the "^" must
> also be escaped, but is not. The "|" pipe symbol must also be escaped
> by "^", as many others as well.
>
> The effect was that passing a rexexp as parameter to a windows program
> gave me strange effects, and I recognized that "^" was missing.
>
> So I was asking for a coherent solution:
> Escape things completely or omit "shell=True".
>
> Yes, there is a list of chars to escape, and it is Windows version
> dependent. I can provide it if it makes sense.

subprocess.list2cmdline is meant to help support cross-platform code,
since Windows uses a command-line instead of an argv array. The
command-line parsing rules used by VC++ (and CommandLineToArgvW) are
the most common in practice. list2cmdline is intended for this set of
applications. Otherwise pass args as a string instead of a list.

In CMD we can quote part of a command line in double quotes to escape
special characters. The quotes are preserved in the application
command line. This can get complicated when we need to preserve
literal quotes in the command line of an application that uses VC++
backslash escaping. CMD doesn't recognize backslash as an escape
character, which gives rise to a quoting conflict between CMD and the
application. Some applications support translating single quotes to
double quotes in this case (e.g. schtasks.exe). Single quotes
generally aren't used in CMD, except in a `for /f` loop, but this can
be forced to use backquotes instead via `usebackq`.

Quoting doesn't escape the percent character that's used for
environment variables. In batch scripts percent can be escaped by
doubling it, but not in /c commands. Some applications can translate a
substitute character in this case, such as "~" (e.g. setx.exe).
Otherwise, we can usually disrupt matching an existing variable by
adding a "^" character after the first percent character. The "^"
escape character gets consumed later on in parsing -- as long as it's
not quoted (see the previous paragraph for complications).
Nonetheless, "^" is a valid name character, so there's still a
possibility of matching an environment variable (perhaps a malicious
one).  For example:

C:\>python -c "print('"%^"time%')"
%time%

C:\>set "^"time=spam"
C:\>python -c "print('"%^"time%')"
spam

Anyway, we're supposed to pass args as a string when using the shell
in POSIX, so we may as well stay consistent with this in Windows.
Practically no one wants the resulting behavior when passing a shell
command as a list in POSIX. For example:

>>> subprocess.call(['echo \\$0=$0 \\$1=$1', 'spam', 'eggs'], shell=True)
$0=spam $1=eggs

It's common to discourage using `shell=True` because it's considered
insecure. One of the reasons to use CMD in Windows is that it tries
ShellExecuteEx if CreateProcess fails. ShellExecuteEx supports "App
Paths" commands, file actions (open, edit, print), UAC elevation (via
"runas" or if requested by the manifest), protocols (including
"shell:"), and opening folders in Explorer. It isn't a scripting
language, however, so it doesn't pose the same risk as using CMD.
Calling ShellExecuteEx could be integrated in subprocess as a new
Popen parameter, such as `winshell` or `shellex`.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Random832]

On Sun, Jan 7, 2018, at 19:50, Steve Dower wrote:
> Considering there is no cross-platform compatibility here anyway, I 
> don’t think it’s that bad an option to let users do their own escaping, 
> especially since those who are successfully using this feature already 
> do.

I don't really think we should give up on cross-platform compatibility that 
easily. There are a number of constructs supported with the same syntax by both 
cmd and unix shells (pipes and redirection, mainly) that people may want to use.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Steve Dower]

Quoting the /c and /k values to cmd.exe is... complicated, to say the least. I 
struggle to get it right for a single example, let alone generalising it. The 
/s option also has an impact – sometimes it helps you avoid double-escaping 
everything, but not always.

Writing complex shell=True commands to a temporary batch file and executing 
that is more reliable wrt quoting, though now you'd need somewhere writable and 
executable on disk, which is becoming hard to come by.

Considering there is no cross-platform compatibility here anyway, I don’t think 
it’s that bad an option to let users do their own escaping, especially since 
those who are successfully using this feature already do.

Cheers,
Steve

Top-posted from my Windows phone

From: Guido van Rossum
Sent: Monday, January 8, 2018 8:02
To: Gregory P. Smith
Cc: Christian Tismer; Python-Dev
Subject: Re: [Python-Dev] subprocess not escaping "^" on Windows

On Sun, Jan 7, 2018 at 12:30 PM, Gregory P. Smith <g...@krypto.org> wrote:
the best way to improve shell escaping on windows is to send a PR against the 
list2cmdline code that escapes everything you believe it should when running on 
windows. With hyperlinks to the relevant msdn info about what might need 
escaping.


Agreed. FWIW the call to list2cmdline seems to compound the problem, since it 
just takes args and puts double quotes around it, mostly undoing the work of 
list2cmdline. For example if I use (args=['a', 'b c'], shell=True) I think 
list2cmdline turns that to args='a "b c"', and then the format() expression 
constructs the command:

    cmd.exe /c "a "b c""

I really have no idea what that means on Windows (and no quick access to a 
Windows box to try it) but on Windows that would create *two* arguments, the 
first one being 'a b' and the second one 'c'.

At this point I can understand that Christian recommends against shell=True -- 
it's totally messed up! But the fix should really be to fix this, not inventing 
a new feature.

-- 
--Guido van Rossum (python.org/~guido)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Christian Tismer]

Ok, then I'm happy to improve the escaping!

I was confused because I could not understand that nobody than me should
have run into this problem before.

There are many special cases. I'll try my very best :)

Cheers -- Chris

On 07.01.18 21:59, Guido van Rossum wrote:
> On Sun, Jan 7, 2018 at 12:30 PM, Gregory P. Smith  > wrote:
> 
> the best way to improve shell escaping on windows is to send a PR
> against the list2cmdline code that escapes everything you believe it
> should when running on windows. With hyperlinks to the relevant msdn
> info about what might need escaping.
> 
> 
> Agreed. FWIW the call to list2cmdline seems to compound the problem,
> since it just takes args and puts double quotes around it, mostly
> undoing the work of list2cmdline. For example if I use (args=['a', 'b
> c'], shell=True) I think list2cmdline turns that to args='a "b c"', and
> then the format() expression constructs the command:
> 
>     cmd.exe /c "a "b c""
> 
> I really have no idea what that means on Windows (and no quick access to
> a Windows box to try it) but on Windows that would create *two*
> arguments, the first one being 'a b' and the second one 'c'.
> 
> At this point I can understand that Christian recommends against
> shell=True -- it's totally messed up! But the fix should really be to
> fix this, not inventing a new feature.
> 
> -- 
> --Guido van Rossum (python.org/~guido )
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com
> 


-- 
Christian Tismer :^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Guido van Rossum]

On Sun, Jan 7, 2018 at 12:30 PM, Gregory P. Smith  wrote:

> the best way to improve shell escaping on windows is to send a PR against
> the list2cmdline code that escapes everything you believe it should when
> running on windows. With hyperlinks to the relevant msdn info about what
> might need escaping.
>

Agreed. FWIW the call to list2cmdline seems to compound the problem, since
it just takes args and puts double quotes around it, mostly undoing the
work of list2cmdline. For example if I use (args=['a', 'b c'], shell=True)
I think list2cmdline turns that to args='a "b c"', and then the format()
expression constructs the command:

cmd.exe /c "a "b c""

I really have no idea what that means on Windows (and no quick access to a
Windows box to try it) but on Windows that would create *two* arguments,
the first one being 'a b' and the second one 'c'.

At this point I can understand that Christian recommends against shell=True
-- it's totally messed up! But the fix should really be to fix this, not
inventing a new feature.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Gregory P. Smith]

the best way to improve shell escaping on windows is to send a PR against
the list2cmdline code that escapes everything you believe it should when
running on windows. With hyperlinks to the relevant msdn info about what
might need escaping.

On Sun, Jan 7, 2018 at 11:38 AM Christian Tismer 
wrote:

> Ok, I thought only about Windows where people often use shell=True.
> I did not see that as a Linux problem, too.
>
> Not meant as a proposal, just loud thinking... :-)
>
> But as said, the incomplete escaping is a complete mess.
>
> Ciao -- Chris
>
> On 07.01.18 19:54, Christian Tismer wrote:
> > By "normal user expectations" I meant the behavior when the builtin
> commands
> > were normal programs.
> >
> > Using "shell=True" is everywhere recommended to avoid, and I believe
> > we could avoid it by giving them replacements for build-ins.
> >
> > But I don't care if the shell escaping is correct. And that is not
> > trivial, either.
> >
> > On 07.01.18 18:22, Guido van Rossum wrote:
> >> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer  >> > wrote:
> >>
> >> As a side note: In most cases where shell=True is found, people
> >> seem to need evaluation of the PATH variable. To my understanding,
> >>
> >> >>> from subprocess import call
> >> >>> call(("ls",))
> >>
> >> works in Linux, but (with dir) not in Windows. But that is
> misleading
> >> because "dir" is a builtin command but "ls" is not. The same holds
> for
> >> "del" (Windows) and "rm" (Linux).
> >>
> >> So I thought that using shell=True was a good Thing on windows,
> >> but actually it is the start of all evil.
> >> Using regular commands like "git" works fine on Windows and Linux
> >> without the shell=True parameter.
> >>
> >> Perhaps it would be a good thing to emulate the builtin programs
> >> in python by some shell=True replacement (emulate_shell=True?)
> >> to match the normal user expectations without using the shell?
> >>
> >>
> >> That feels like a terrible idea to me. How do you define "normal user
> >> expectations" here? If people want shell builtins they should just use
> >> shell=True. (Also note IIUC there are several quite different shells
> >> commonly used on Windows, e.g. PowerShell.)
> >>
> >> --
> >> --Guido van Rossum (python.org/~guido )
> >
> >
>
>
> --
> Christian Tismer-Sperling:^)   tis...@stackless.com
> Software Consulting  : http://www.stackless.com/
> Karl-Liebknecht-Str. 121 : https://github.com/PySide
> 14482 Potsdam: GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776 <+49%20173%202418776>  fax +49 (30) 700143-0023
> <+49%2030%207001430023>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Christian Tismer]

Ok, I thought only about Windows where people often use shell=True.
I did not see that as a Linux problem, too.

Not meant as a proposal, just loud thinking... :-)

But as said, the incomplete escaping is a complete mess.

Ciao -- Chris

On 07.01.18 19:54, Christian Tismer wrote:
> By "normal user expectations" I meant the behavior when the builtin commands
> were normal programs.
> 
> Using "shell=True" is everywhere recommended to avoid, and I believe
> we could avoid it by giving them replacements for build-ins.
> 
> But I don't care if the shell escaping is correct. And that is not
> trivial, either.
> 
> On 07.01.18 18:22, Guido van Rossum wrote:
>> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer > > wrote:
>>
>> As a side note: In most cases where shell=True is found, people
>> seem to need evaluation of the PATH variable. To my understanding,
>>
>> >>> from subprocess import call
>> >>> call(("ls",))
>>
>> works in Linux, but (with dir) not in Windows. But that is misleading
>> because "dir" is a builtin command but "ls" is not. The same holds for
>> "del" (Windows) and "rm" (Linux).
>>
>> So I thought that using shell=True was a good Thing on windows,
>> but actually it is the start of all evil.
>> Using regular commands like "git" works fine on Windows and Linux
>> without the shell=True parameter.
>>
>> Perhaps it would be a good thing to emulate the builtin programs
>> in python by some shell=True replacement (emulate_shell=True?)
>> to match the normal user expectations without using the shell?
>>
>>
>> That feels like a terrible idea to me. How do you define "normal user
>> expectations" here? If people want shell builtins they should just use
>> shell=True. (Also note IIUC there are several quite different shells
>> commonly used on Windows, e.g. PowerShell.)
>>
>> -- 
>> --Guido van Rossum (python.org/~guido )
> 
> 


-- 
Christian Tismer-Sperling:^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Christian Tismer]

By "normal user expectations" I meant the behavior when the builtin commands
were normal programs.

Using "shell=True" is everywhere recommended to avoid, and I believe
we could avoid it by giving them replacements for build-ins.

But I don't care if the shell escaping is correct. And that is not
trivial, either.

On 07.01.18 18:22, Guido van Rossum wrote:
> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer  > wrote:
> 
> As a side note: In most cases where shell=True is found, people
> seem to need evaluation of the PATH variable. To my understanding,
> 
> >>> from subprocess import call
> >>> call(("ls",))
> 
> works in Linux, but (with dir) not in Windows. But that is misleading
> because "dir" is a builtin command but "ls" is not. The same holds for
> "del" (Windows) and "rm" (Linux).
> 
> So I thought that using shell=True was a good Thing on windows,
> but actually it is the start of all evil.
> Using regular commands like "git" works fine on Windows and Linux
> without the shell=True parameter.
> 
> Perhaps it would be a good thing to emulate the builtin programs
> in python by some shell=True replacement (emulate_shell=True?)
> to match the normal user expectations without using the shell?
> 
> 
> That feels like a terrible idea to me. How do you define "normal user
> expectations" here? If people want shell builtins they should just use
> shell=True. (Also note IIUC there are several quite different shells
> commonly used on Windows, e.g. PowerShell.)
> 
> -- 
> --Guido van Rossum (python.org/~guido )


-- 
Christian Tismer-Sperling:^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Christian Tismer]

That is true.
list2cmdline escapes partially, but on NT and Windows10, the "^" must
also be escaped, but is not. The "|" pipe symbol must also be escaped
by "^", as many others as well.

The effect was that passing a rexexp as parameter to a windows program
gave me strange effects, and I recognized that "^" was missing.

So I was asking for a coherent solution:
Escape things completely or omit "shell=True".

Yes, there is a list of chars to escape, and it is Windows version
dependent. I can provide it if it makes sense.

Cheers -- Chris


On 07.01.18 18:20, Guido van Rossum wrote:
> I assume you're talking about list2cmdline()? That seems to be used to
> construct a string that can be passed to `cmd /c "{}"` -- it gets
> substituted instead of the {}, i.e. surrounded by ". I honestly can't
> say I follow that code completely, but I see that it escapes double
> quotes. Why is there a need to escape other characters? Is there a
> definitive list of special characters somewhere?
> 
> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer  > wrote:
> 
> Hi Guys,
> 
> yes I know there was a lengthy thread on python-dev in 2014
> called "subprocess shell=True on Windows doesn't escape ^ character".
> 
> But in the end, I still don't understand why subprocess does
> escape the double quote when shell=True but not other special
> characters like "^"?
> 
> Yes I know that certain characters are escaped under certain
> Windows versions and others are not. And it is not trivial to make
> that work correctly in all cases. But I think if we support
> some escaping at all, then we should also support all special
> cases. Or what sense should an escape make if it works sometimes
> and sometimes not?
> 
> The user would have to know which cases work and which not. But
> I thought we want to remove exactly that burden from him?
> 
> -
> 
> As a side note: In most cases where shell=True is found, people
> seem to need evaluation of the PATH variable. To my understanding,
> 
> >>> from subprocess import call
> >>> call(("ls",))
> 
> works in Linux, but (with dir) not in Windows. But that is misleading
> because "dir" is a builtin command but "ls" is not. The same holds for
> "del" (Windows) and "rm" (Linux).
> 
> So I thought that using shell=True was a good Thing on windows,
> but actually it is the start of all evil.
> Using regular commands like "git" works fine on Windows and Linux
> without the shell=True parameter.
> 
> Perhaps it would be a good thing to emulate the builtin programs
> in python by some shell=True replacement (emulate_shell=True?)
> to match the normal user expectations without using the shell?
> 
> Cheers - Chris
> 
> --
> Christian Tismer-Sperling    :^)   tis...@stackless.com
> 
> Software Consulting          :     http://www.stackless.com/
> Karl-Liebknecht-Str .
> 121     :     https://github.com/PySide
> 14482 Potsdam                :     GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776   fax +49
> (30) 700143-0023 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org 
> https://mail.python.org/mailman/listinfo/python-dev
> 
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org 
> 
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido )


-- 
Christian Tismer-Sperling:^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Guido van Rossum]

On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer 
wrote:

> As a side note: In most cases where shell=True is found, people
> seem to need evaluation of the PATH variable. To my understanding,
>
> >>> from subprocess import call
> >>> call(("ls",))
>
> works in Linux, but (with dir) not in Windows. But that is misleading
> because "dir" is a builtin command but "ls" is not. The same holds for
> "del" (Windows) and "rm" (Linux).
>
> So I thought that using shell=True was a good Thing on windows,
> but actually it is the start of all evil.
> Using regular commands like "git" works fine on Windows and Linux
> without the shell=True parameter.
>
> Perhaps it would be a good thing to emulate the builtin programs
> in python by some shell=True replacement (emulate_shell=True?)
> to match the normal user expectations without using the shell?
>

That feels like a terrible idea to me. How do you define "normal user
expectations" here? If people want shell builtins they should just use
shell=True. (Also note IIUC there are several quite different shells
commonly used on Windows, e.g. PowerShell.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Guido van Rossum]

I assume you're talking about list2cmdline()? That seems to be used to
construct a string that can be passed to `cmd /c "{}"` -- it gets
substituted instead of the {}, i.e. surrounded by ". I honestly can't say I
follow that code completely, but I see that it escapes double quotes. Why
is there a need to escape other characters? Is there a definitive list of
special characters somewhere?

On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer 
wrote:

> Hi Guys,
>
> yes I know there was a lengthy thread on python-dev in 2014
> called "subprocess shell=True on Windows doesn't escape ^ character".
>
> But in the end, I still don't understand why subprocess does
> escape the double quote when shell=True but not other special
> characters like "^"?
>
> Yes I know that certain characters are escaped under certain
> Windows versions and others are not. And it is not trivial to make
> that work correctly in all cases. But I think if we support
> some escaping at all, then we should also support all special
> cases. Or what sense should an escape make if it works sometimes
> and sometimes not?
>
> The user would have to know which cases work and which not. But
> I thought we want to remove exactly that burden from him?
>
> -
>
> As a side note: In most cases where shell=True is found, people
> seem to need evaluation of the PATH variable. To my understanding,
>
> >>> from subprocess import call
> >>> call(("ls",))
>
> works in Linux, but (with dir) not in Windows. But that is misleading
> because "dir" is a builtin command but "ls" is not. The same holds for
> "del" (Windows) and "rm" (Linux).
>
> So I thought that using shell=True was a good Thing on windows,
> but actually it is the start of all evil.
> Using regular commands like "git" works fine on Windows and Linux
> without the shell=True parameter.
>
> Perhaps it would be a good thing to emulate the builtin programs
> in python by some shell=True replacement (emulate_shell=True?)
> to match the normal user expectations without using the shell?
>
> Cheers - Chris
>
> --
> Christian Tismer-Sperling:^)   tis...@stackless.com
> Software Consulting  : http://www.stackless.com/
> Karl-Liebknecht-Str. 121 : https://github.com/PySide
> 14482 Potsdam: GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776  fax +49 (30) 700143-0023
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] subprocess not escaping "^" on Windows

[Christian Tismer]

Hi Guys,

yes I know there was a lengthy thread on python-dev in 2014
called "subprocess shell=True on Windows doesn't escape ^ character".

But in the end, I still don't understand why subprocess does
escape the double quote when shell=True but not other special
characters like "^"?

Yes I know that certain characters are escaped under certain
Windows versions and others are not. And it is not trivial to make
that work correctly in all cases. But I think if we support
some escaping at all, then we should also support all special
cases. Or what sense should an escape make if it works sometimes
and sometimes not?

The user would have to know which cases work and which not. But
I thought we want to remove exactly that burden from him?

-

As a side note: In most cases where shell=True is found, people
seem to need evaluation of the PATH variable. To my understanding,

>>> from subprocess import call
>>> call(("ls",))

works in Linux, but (with dir) not in Windows. But that is misleading
because "dir" is a builtin command but "ls" is not. The same holds for
"del" (Windows) and "rm" (Linux).

So I thought that using shell=True was a good Thing on windows,
but actually it is the start of all evil.
Using regular commands like "git" works fine on Windows and Linux
without the shell=True parameter.

Perhaps it would be a good thing to emulate the builtin programs
in python by some shell=True replacement (emulate_shell=True?)
to match the normal user expectations without using the shell?

Cheers - Chris

-- 
Christian Tismer-Sperling:^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com