[Python-Dev] subprocess not escaping "^" on Windows

[Christian Tismer]

Hi Guys,

yes I know there was a lengthy thread on python-dev in 2014
called "subprocess shell=True on Windows doesn't escape ^ character".

But in the end, I still don't understand why subprocess does
escape the double quote when shell=True but not other special
characters like "^"?

Yes I know that certain characters are escaped under certain
Windows versions and others are not. And it is not trivial to make
that work correctly in all cases. But I think if we support
some escaping at all, then we should also support all special
cases. Or what sense should an escape make if it works sometimes
and sometimes not?

The user would have to know which cases work and which not. But
I thought we want to remove exactly that burden from him?

-

As a side note: In most cases where shell=True is found, people
seem to need evaluation of the PATH variable. To my understanding,

>>> from subprocess import call
>>> call(("ls",))

works in Linux, but (with dir) not in Windows. But that is misleading
because "dir" is a builtin command but "ls" is not. The same holds for
"del" (Windows) and "rm" (Linux).

So I thought that using shell=True was a good Thing on windows,
but actually it is the start of all evil.
Using regular commands like "git" works fine on Windows and Linux
without the shell=True parameter.

Perhaps it would be a good thing to emulate the builtin programs
in python by some shell=True replacement (emulate_shell=True?)
to match the normal user expectations without using the shell?

Cheers - Chris

-- 
Christian Tismer-Sperling:^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Eric V. Smith]

On 1/3/2018 1:17 PM, Eric V. Smith wrote:

I’ll open an issue after I have time to read this thread and comment on it.


https://bugs.python.org/issue32513
I need to think though how __eq__ and __ne__ work, as well as the 
ordering operators.


My specific concern with __ne__ is that there's one flag to control 
their generation, but python will use "not __eq__" if you don't provide 
__ne__. I need to think through what happens if the user only provides 
__eq__: does dataclasses do nothing, does it add __ne__, and how does 
this interact with a base class that does provide __ne__.


Eric.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Guido van Rossum]

I assume you're talking about list2cmdline()? That seems to be used to
construct a string that can be passed to `cmd /c "{}"` -- it gets
substituted instead of the {}, i.e. surrounded by ". I honestly can't say I
follow that code completely, but I see that it escapes double quotes. Why
is there a need to escape other characters? Is there a definitive list of
special characters somewhere?

On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer 
wrote:

> Hi Guys,
>
> yes I know there was a lengthy thread on python-dev in 2014
> called "subprocess shell=True on Windows doesn't escape ^ character".
>
> But in the end, I still don't understand why subprocess does
> escape the double quote when shell=True but not other special
> characters like "^"?
>
> Yes I know that certain characters are escaped under certain
> Windows versions and others are not. And it is not trivial to make
> that work correctly in all cases. But I think if we support
> some escaping at all, then we should also support all special
> cases. Or what sense should an escape make if it works sometimes
> and sometimes not?
>
> The user would have to know which cases work and which not. But
> I thought we want to remove exactly that burden from him?
>
> -
>
> As a side note: In most cases where shell=True is found, people
> seem to need evaluation of the PATH variable. To my understanding,
>
> >>> from subprocess import call
> >>> call(("ls",))
>
> works in Linux, but (with dir) not in Windows. But that is misleading
> because "dir" is a builtin command but "ls" is not. The same holds for
> "del" (Windows) and "rm" (Linux).
>
> So I thought that using shell=True was a good Thing on windows,
> but actually it is the start of all evil.
> Using regular commands like "git" works fine on Windows and Linux
> without the shell=True parameter.
>
> Perhaps it would be a good thing to emulate the builtin programs
> in python by some shell=True replacement (emulate_shell=True?)
> to match the normal user expectations without using the shell?
>
> Cheers - Chris
>
> --
> Christian Tismer-Sperling:^)   tis...@stackless.com
> Software Consulting  : http://www.stackless.com/
> Karl-Liebknecht-Str. 121 : https://github.com/PySide
> 14482 Potsdam: GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776  fax +49 (30) 700143-0023
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Guido van Rossum]

On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer 
wrote:

> As a side note: In most cases where shell=True is found, people
> seem to need evaluation of the PATH variable. To my understanding,
>
> >>> from subprocess import call
> >>> call(("ls",))
>
> works in Linux, but (with dir) not in Windows. But that is misleading
> because "dir" is a builtin command but "ls" is not. The same holds for
> "del" (Windows) and "rm" (Linux).
>
> So I thought that using shell=True was a good Thing on windows,
> but actually it is the start of all evil.
> Using regular commands like "git" works fine on Windows and Linux
> without the shell=True parameter.
>
> Perhaps it would be a good thing to emulate the builtin programs
> in python by some shell=True replacement (emulate_shell=True?)
> to match the normal user expectations without using the shell?
>

That feels like a terrible idea to me. How do you define "normal user
expectations" here? If people want shell builtins they should just use
shell=True. (Also note IIUC there are several quite different shells
commonly used on Windows, e.g. PowerShell.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Guido van Rossum]

On Sun, Jan 7, 2018 at 9:09 AM, Eric V. Smith  wrote:

> On 1/3/2018 1:17 PM, Eric V. Smith wrote:
>
>> I’ll open an issue after I have time to read this thread and comment on
>> it.
>>
>
> https://bugs.python.org/issue32513
> I need to think though how __eq__ and __ne__ work, as well as the ordering
> operators.
>
> My specific concern with __ne__ is that there's one flag to control their
> generation, but python will use "not __eq__" if you don't provide __ne__. I
> need to think through what happens if the user only provides __eq__: does
> dataclasses do nothing, does it add __ne__, and how does this interact with
> a base class that does provide __ne__.


Maybe dataclasses should only ever provide __eq__ and always assume
Python's default for __ne__ kicks in? If that's not acceptable (maybe there
are cases where a user did write an explicit __ne__ that needs to be
overridden) I would recommend the following rule:

- If there's an __eq__, don't do anything (regardless of whether there's an
__ne__)
- If there no __eq__ but there is an __ne__, generate __eq__ but don't
generate __ne__
- If neither exists, generate both

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Christian Tismer]

That is true.
list2cmdline escapes partially, but on NT and Windows10, the "^" must
also be escaped, but is not. The "|" pipe symbol must also be escaped
by "^", as many others as well.

The effect was that passing a rexexp as parameter to a windows program
gave me strange effects, and I recognized that "^" was missing.

So I was asking for a coherent solution:
Escape things completely or omit "shell=True".

Yes, there is a list of chars to escape, and it is Windows version
dependent. I can provide it if it makes sense.

Cheers -- Chris


On 07.01.18 18:20, Guido van Rossum wrote:
> I assume you're talking about list2cmdline()? That seems to be used to
> construct a string that can be passed to `cmd /c "{}"` -- it gets
> substituted instead of the {}, i.e. surrounded by ". I honestly can't
> say I follow that code completely, but I see that it escapes double
> quotes. Why is there a need to escape other characters? Is there a
> definitive list of special characters somewhere?
> 
> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer  > wrote:
> 
> Hi Guys,
> 
> yes I know there was a lengthy thread on python-dev in 2014
> called "subprocess shell=True on Windows doesn't escape ^ character".
> 
> But in the end, I still don't understand why subprocess does
> escape the double quote when shell=True but not other special
> characters like "^"?
> 
> Yes I know that certain characters are escaped under certain
> Windows versions and others are not. And it is not trivial to make
> that work correctly in all cases. But I think if we support
> some escaping at all, then we should also support all special
> cases. Or what sense should an escape make if it works sometimes
> and sometimes not?
> 
> The user would have to know which cases work and which not. But
> I thought we want to remove exactly that burden from him?
> 
> -
> 
> As a side note: In most cases where shell=True is found, people
> seem to need evaluation of the PATH variable. To my understanding,
> 
> >>> from subprocess import call
> >>> call(("ls",))
> 
> works in Linux, but (with dir) not in Windows. But that is misleading
> because "dir" is a builtin command but "ls" is not. The same holds for
> "del" (Windows) and "rm" (Linux).
> 
> So I thought that using shell=True was a good Thing on windows,
> but actually it is the start of all evil.
> Using regular commands like "git" works fine on Windows and Linux
> without the shell=True parameter.
> 
> Perhaps it would be a good thing to emulate the builtin programs
> in python by some shell=True replacement (emulate_shell=True?)
> to match the normal user expectations without using the shell?
> 
> Cheers - Chris
> 
> --
> Christian Tismer-Sperling    :^)   tis...@stackless.com
> 
> Software Consulting          :     http://www.stackless.com/
> Karl-Liebknecht-Str .
> 121     :     https://github.com/PySide
> 14482 Potsdam                :     GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776   fax +49
> (30) 700143-0023 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org 
> https://mail.python.org/mailman/listinfo/python-dev
> 
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org 
> 
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido )


-- 
Christian Tismer-Sperling:^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Christian Tismer]

By "normal user expectations" I meant the behavior when the builtin commands
were normal programs.

Using "shell=True" is everywhere recommended to avoid, and I believe
we could avoid it by giving them replacements for build-ins.

But I don't care if the shell escaping is correct. And that is not
trivial, either.

On 07.01.18 18:22, Guido van Rossum wrote:
> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer  > wrote:
> 
> As a side note: In most cases where shell=True is found, people
> seem to need evaluation of the PATH variable. To my understanding,
> 
> >>> from subprocess import call
> >>> call(("ls",))
> 
> works in Linux, but (with dir) not in Windows. But that is misleading
> because "dir" is a builtin command but "ls" is not. The same holds for
> "del" (Windows) and "rm" (Linux).
> 
> So I thought that using shell=True was a good Thing on windows,
> but actually it is the start of all evil.
> Using regular commands like "git" works fine on Windows and Linux
> without the shell=True parameter.
> 
> Perhaps it would be a good thing to emulate the builtin programs
> in python by some shell=True replacement (emulate_shell=True?)
> to match the normal user expectations without using the shell?
> 
> 
> That feels like a terrible idea to me. How do you define "normal user
> expectations" here? If people want shell builtins they should just use
> shell=True. (Also note IIUC there are several quite different shells
> commonly used on Windows, e.g. PowerShell.)
> 
> -- 
> --Guido van Rossum (python.org/~guido )


-- 
Christian Tismer-Sperling:^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Christian Tismer]

Ok, I thought only about Windows where people often use shell=True.
I did not see that as a Linux problem, too.

Not meant as a proposal, just loud thinking... :-)

But as said, the incomplete escaping is a complete mess.

Ciao -- Chris

On 07.01.18 19:54, Christian Tismer wrote:
> By "normal user expectations" I meant the behavior when the builtin commands
> were normal programs.
> 
> Using "shell=True" is everywhere recommended to avoid, and I believe
> we could avoid it by giving them replacements for build-ins.
> 
> But I don't care if the shell escaping is correct. And that is not
> trivial, either.
> 
> On 07.01.18 18:22, Guido van Rossum wrote:
>> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer > > wrote:
>>
>> As a side note: In most cases where shell=True is found, people
>> seem to need evaluation of the PATH variable. To my understanding,
>>
>> >>> from subprocess import call
>> >>> call(("ls",))
>>
>> works in Linux, but (with dir) not in Windows. But that is misleading
>> because "dir" is a builtin command but "ls" is not. The same holds for
>> "del" (Windows) and "rm" (Linux).
>>
>> So I thought that using shell=True was a good Thing on windows,
>> but actually it is the start of all evil.
>> Using regular commands like "git" works fine on Windows and Linux
>> without the shell=True parameter.
>>
>> Perhaps it would be a good thing to emulate the builtin programs
>> in python by some shell=True replacement (emulate_shell=True?)
>> to match the normal user expectations without using the shell?
>>
>>
>> That feels like a terrible idea to me. How do you define "normal user
>> expectations" here? If people want shell builtins they should just use
>> shell=True. (Also note IIUC there are several quite different shells
>> commonly used on Windows, e.g. PowerShell.)
>>
>> -- 
>> --Guido van Rossum (python.org/~guido )
> 
> 


-- 
Christian Tismer-Sperling:^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Gregory P. Smith]

the best way to improve shell escaping on windows is to send a PR against
the list2cmdline code that escapes everything you believe it should when
running on windows. With hyperlinks to the relevant msdn info about what
might need escaping.

On Sun, Jan 7, 2018 at 11:38 AM Christian Tismer 
wrote:

> Ok, I thought only about Windows where people often use shell=True.
> I did not see that as a Linux problem, too.
>
> Not meant as a proposal, just loud thinking... :-)
>
> But as said, the incomplete escaping is a complete mess.
>
> Ciao -- Chris
>
> On 07.01.18 19:54, Christian Tismer wrote:
> > By "normal user expectations" I meant the behavior when the builtin
> commands
> > were normal programs.
> >
> > Using "shell=True" is everywhere recommended to avoid, and I believe
> > we could avoid it by giving them replacements for build-ins.
> >
> > But I don't care if the shell escaping is correct. And that is not
> > trivial, either.
> >
> > On 07.01.18 18:22, Guido van Rossum wrote:
> >> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer  >> > wrote:
> >>
> >> As a side note: In most cases where shell=True is found, people
> >> seem to need evaluation of the PATH variable. To my understanding,
> >>
> >> >>> from subprocess import call
> >> >>> call(("ls",))
> >>
> >> works in Linux, but (with dir) not in Windows. But that is
> misleading
> >> because "dir" is a builtin command but "ls" is not. The same holds
> for
> >> "del" (Windows) and "rm" (Linux).
> >>
> >> So I thought that using shell=True was a good Thing on windows,
> >> but actually it is the start of all evil.
> >> Using regular commands like "git" works fine on Windows and Linux
> >> without the shell=True parameter.
> >>
> >> Perhaps it would be a good thing to emulate the builtin programs
> >> in python by some shell=True replacement (emulate_shell=True?)
> >> to match the normal user expectations without using the shell?
> >>
> >>
> >> That feels like a terrible idea to me. How do you define "normal user
> >> expectations" here? If people want shell builtins they should just use
> >> shell=True. (Also note IIUC there are several quite different shells
> >> commonly used on Windows, e.g. PowerShell.)
> >>
> >> --
> >> --Guido van Rossum (python.org/~guido )
> >
> >
>
>
> --
> Christian Tismer-Sperling:^)   tis...@stackless.com
> Software Consulting  : http://www.stackless.com/
> Karl-Liebknecht-Str. 121 : https://github.com/PySide
> 14482 Potsdam: GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776 <+49%20173%202418776>  fax +49 (30) 700143-0023
> <+49%2030%207001430023>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Guido van Rossum]

On Sun, Jan 7, 2018 at 12:30 PM, Gregory P. Smith  wrote:

> the best way to improve shell escaping on windows is to send a PR against
> the list2cmdline code that escapes everything you believe it should when
> running on windows. With hyperlinks to the relevant msdn info about what
> might need escaping.
>

Agreed. FWIW the call to list2cmdline seems to compound the problem, since
it just takes args and puts double quotes around it, mostly undoing the
work of list2cmdline. For example if I use (args=['a', 'b c'], shell=True)
I think list2cmdline turns that to args='a "b c"', and then the format()
expression constructs the command:

cmd.exe /c "a "b c""

I really have no idea what that means on Windows (and no quick access to a
Windows box to try it) but on Windows that would create *two* arguments,
the first one being 'a b' and the second one 'c'.

At this point I can understand that Christian recommends against shell=True
-- it's totally messed up! But the fix should really be to fix this, not
inventing a new feature.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Christian Tismer]

Ok, then I'm happy to improve the escaping!

I was confused because I could not understand that nobody than me should
have run into this problem before.

There are many special cases. I'll try my very best :)

Cheers -- Chris

On 07.01.18 21:59, Guido van Rossum wrote:
> On Sun, Jan 7, 2018 at 12:30 PM, Gregory P. Smith  > wrote:
> 
> the best way to improve shell escaping on windows is to send a PR
> against the list2cmdline code that escapes everything you believe it
> should when running on windows. With hyperlinks to the relevant msdn
> info about what might need escaping.
> 
> 
> Agreed. FWIW the call to list2cmdline seems to compound the problem,
> since it just takes args and puts double quotes around it, mostly
> undoing the work of list2cmdline. For example if I use (args=['a', 'b
> c'], shell=True) I think list2cmdline turns that to args='a "b c"', and
> then the format() expression constructs the command:
> 
>     cmd.exe /c "a "b c""
> 
> I really have no idea what that means on Windows (and no quick access to
> a Windows box to try it) but on Windows that would create *two*
> arguments, the first one being 'a b' and the second one 'c'.
> 
> At this point I can understand that Christian recommends against
> shell=True -- it's totally messed up! But the fix should really be to
> fix this, not inventing a new feature.
> 
> -- 
> --Guido van Rossum (python.org/~guido )
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com
> 


-- 
Christian Tismer :^)   tis...@stackless.com
Software Consulting  : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Steve Dower]

Quoting the /c and /k values to cmd.exe is... complicated, to say the least. I 
struggle to get it right for a single example, let alone generalising it. The 
/s option also has an impact – sometimes it helps you avoid double-escaping 
everything, but not always.

Writing complex shell=True commands to a temporary batch file and executing 
that is more reliable wrt quoting, though now you'd need somewhere writable and 
executable on disk, which is becoming hard to come by.

Considering there is no cross-platform compatibility here anyway, I don’t think 
it’s that bad an option to let users do their own escaping, especially since 
those who are successfully using this feature already do.

Cheers,
Steve

Top-posted from my Windows phone

From: Guido van Rossum
Sent: Monday, January 8, 2018 8:02
To: Gregory P. Smith
Cc: Christian Tismer; Python-Dev
Subject: Re: [Python-Dev] subprocess not escaping "^" on Windows

On Sun, Jan 7, 2018 at 12:30 PM, Gregory P. Smith  wrote:
the best way to improve shell escaping on windows is to send a PR against the 
list2cmdline code that escapes everything you believe it should when running on 
windows. With hyperlinks to the relevant msdn info about what might need 
escaping.


Agreed. FWIW the call to list2cmdline seems to compound the problem, since it 
just takes args and puts double quotes around it, mostly undoing the work of 
list2cmdline. For example if I use (args=['a', 'b c'], shell=True) I think 
list2cmdline turns that to args='a "b c"', and then the format() expression 
constructs the command:

    cmd.exe /c "a "b c""

I really have no idea what that means on Windows (and no quick access to a 
Windows box to try it) but on Windows that would create *two* arguments, the 
first one being 'a b' and the second one 'c'.

At this point I can understand that Christian recommends against shell=True -- 
it's totally messed up! But the fix should really be to fix this, not inventing 
a new feature.

-- 
--Guido van Rossum (python.org/~guido)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Random832]

On Sun, Jan 7, 2018, at 19:50, Steve Dower wrote:
> Considering there is no cross-platform compatibility here anyway, I 
> don’t think it’s that bad an option to let users do their own escaping, 
> especially since those who are successfully using this feature already 
> do.

I don't really think we should give up on cross-platform compatibility that 
easily. There are a number of constructs supported with the same syntax by both 
cmd and unix shells (pipes and redirection, mainly) that people may want to use.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 568: how we could extend PEP 567 to handle generator contexts

[Nathaniel Smith]

See below for a first pass version of PEP 568, which builds on PEP 567
to add context isolation for generators. Basically, as far as features
go:

   PEP 567 + PEP 568 = PEP 550

The actual API details are all different though.

To reiterate, *this is not currently a proposal for adding to Python*.
It probably will become one for 3.8, because Yury and I still think
this is a good idea. But for *right now*, it's just a hypothetical
"what if?" to help guide the PEP 567 discussion, because if we want to
avoid accidentally ruling out the option of making an extension like
this later, it helps to know how such an extension would work.

There's a rendered version at:

   https://www.python.org/dev/peps/pep-0568

(very slightly out of date compared to the version pasted below)

-n

--

PEP: 568
Title: Generator-sensitivity for Context Variables
Author: Nathaniel J. Smith 
Status: Deferred
Type: Standards Track
Content-Type: text/x-rst
Created: 04-Jan-2018
Python-Version: 3.8
Post-History: 2018-01-07


Abstract


Context variables provide a generic mechanism for tracking dynamic,
context-local state, similar to thread-local storage but generalized
to cope work with other kinds of thread-like contexts, such as asyncio
Tasks. PEP 550 proposed a mechanism for context-local state that was
also sensitive to generator context, but this was pretty complicated,
so the BDFL requested it be simplified. The result was PEP 567, which
is targeted for inclusion in 3.7. This PEP then extends PEP 567's
machinery to add generator context sensitivity.

This PEP is starting out in the "deferred" status, because there isn't
enough time to give it proper consideration before the 3.7 feature
freeze. The only goal *right now* is to understand what would be
required to add generator context sensitivity in 3.8, so that we can
avoid shipping something in 3.7 that would rule it out by accident.
(Ruling it out on purpose can wait until 3.8 ;-).)


Rationale
=

[Currently the point of this PEP is just to understand *how* this
would work, with discussion of *whether* it's a good idea deferred
until after the 3.7 feature freeze. So rationale is TBD.]


High-level summary
==

Instead of holding a single ``Context``, the threadstate now holds a
``ChainMap`` of ``Context``\s. ``ContextVar.get`` and
``ContextVar.set`` are backed by the ``ChainMap``. Generators and
async generators each have an associated ``Context`` that they push
onto the ``ChainMap`` while they're running to isolate their
context-local changes from their callers, though this can be
overridden in cases like ``@contextlib.contextmanager`` where
"leaking" context changes from the generator into its caller is
desireable.


Specification
=

Review of PEP 567
-

Let's start by reviewing how PEP 567 works, and then in the next
section we'll describe the differences.

In PEP 567, a ``Context`` is a ``Mapping`` from ``ContextVar`` objects
to arbitrary values. In our pseudo-code here we'll pretend that it
uses a ``dict`` for backing storage. (The real implementation uses a
HAMT, which is semantically equivalent to a ``dict`` but with
different performance trade-offs.)::

   class Context(collections.abc.Mapping):
   def __init__(self):
   self._data = {}
   self._in_use = False

   def __getitem__(self, key):
   return self._data[key]

   def __iter__(self):
   return iter(self._data)

   def __len__(self):
   return len(self._data)

At any given moment, the threadstate holds a current ``Context``
(initialized to an empty ``Context`` when the threadstate is created);
we can use ``Context.run`` to temporarily switch the current
``Context``::

   # Context.run
   def run(self, fn, *args, **kwargs):
   if self._in_use:
   raise RuntimeError("Context already in use")
   tstate = get_thread_state()
   old_context = tstate.current_context
   tstate.current_context = self
   self._in_use = True
   try:
   return fn(*args, **kwargs)
   finally:
   state.current_context = old_context
   self._in_use = False

We can fetch a shallow copy of the current ``Context`` by calling
``copy_context``; this is commonly used when spawning a new task, so
that the child task can inherit context from its parent::

   def copy_context():
   tstate = get_thread_state()
   new_context = Context()
   new_context._data = dict(tstate.current_context)
   return new_context

In practice, what end users generally work with is ``ContextVar``
objects, which also provide the only way to mutate a ``Context``. They
work with a utility class ``Token``, which can be used to restore a
``ContextVar`` to its previous value::

   class Token:
   MISSING = sentinel_value()

   # Note: constructor is private
   def __init__(self, context, var, old_value):
   self._context = context
   self.var = var
   self.