Re: [Python-Dev] Help preventing SIGPIPE/SIG_DFL anti-pattern.
On Sat, Jun 30, 2018 at 9:02 PM Alfred Perlstein wrote: > > > On 6/30/18 4:20 PM, Greg Ewing wrote: > > Alfred Perlstein wrote: > >> I am asking if there's a way we can discourage the use of > >> "signal(SIGPIPE, SIG_DFL)" unless the user really understands what > >> they are doing. > > > > Maybe there's some way that SIGPIPEs on stdout could be handled > > differently by default, so that they exit silently instead of > > producing an ugly message. That would remove the source of pain > > that's leading people to do this. > > > Thank you Greg, I can poke around into this, it would be a bit of a > challenge as the descriptor which causes BrokenPipeError does not appear > to be stored within the exception so differentiating it from other > exceptions might be a bit tricky. > > I will look into this in the coming weeks. Any tips on accomplishing > this? I was thinking of encoding the fd responsible for causing the > error into the exception somehow and then checking to see if it was > stdout, then not reporting on it. > There's PyErr_SetFromErrnoWithFilenameObject(), which generates an OSError with a filename. We could have a similar call PyErr_SetFromErrnoWithFileDescriptor(), which generates an OSError (or a subclass like BrokenPipeError) with a file descriptor. Just pick a new attribute name to store the fd on, e.g. file_descriptor (I guess it would default to None). Then of course you will have to write the code that calls this instead of plain PyErr_SetFromErrno() for those syscalls where a file descriptor is present. And when Python exits with a BrokenPipeError it could suppress printing the stack trace when the file_descriptor field equals 1. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
[Nick Coghlan]
>>> "NAME := EXPR" exists on a different level of complexity, since it
>>> adds name binding in arbitrary expressions for the sake of minor
>>> performance improvement in code written by developers that are
>>> exceptionally averse to the use of vertical screen real estate,
> >>> ...
[Tim]
>> Note that PEP 572 doesn't contain a single word about "performance"
(neither
> >> that specific word nor any synonym), and I gave only one thought to it
when
> >> writing Appendix A: "is this going to slow anything down
significantly?".
> >> The answer was never "yes", which I thought was self-evident, so I never
> >> mentioned it. Neither did Chris or Guido.
> >>
> >> Best I can recall, nobody has argued for it on the grounds of
"performance".
> >> except in the indirect sense that sometimes it allows a more compact
way of
> >> reusing an expensive subexpression by giving it a name. Which they
already
> >> do by giving it a name in a separate statement, so the possible
improvement
> >> would be in brevity rather than performance.
[Nick]
> > The PEP specifically cites this example as motivation:
The PEP gives many examples. Your original was a strawman
mischaracterization of the PEP's _motivations_ (note the plural: you only
mentioned "minor performance improvement", and snipped my listing of the
major motivations).
>
> group = re.match(data).group(1) if re.match(data) else None
> >
> > That code's already perfectly straightforward to read and write as a
> > single line,
I disagree. In any case of textual repetition, it's a visual
pattern-matching puzzle to identify the common substrings (I have to
visually scan that line about 3 times to be sure), and then a potentially
difficult conceptual puzzle to figure out whether side effects may result
in textually identical substrings evaluating to different objects. That's
why "refererential transparency" is so highly valued in functional
languages ("if subexpressions are spelled the same, they evaluate to the
same result, period" - which isn't generally true in Python - to get that
enormously helpful (to reasoning) guarantee in Python you have to ensure
the subexpression is evaluated exactly once).
And as you of all people should be complaining about, textual repetition is
also prone to "oops - forgot one!" and "oops! made a typo when changing the
second one!" when code is later modified.
> so the only reason to quibble about it
I gave you three better reasons to quibble about it just above ;-)
> is because it's slower than the arguably less clear two-line alternative:
> >
> > _m = re.match(data)
> > group = _m.group(1) if _m else None
>
I find that much clearer than the one-liner above: the visual pattern
matching is easier because the repeated substring is shorter and of much
simpler syntactic structure; it guarantees _by construction_ that the two
instances of `_m` evaluate to the same object, so there's no possible
concern about that (it doesn't even matter if you bound `re` to some
"non-standard" object that has nothing to do with Python's `re` module);
and any later changes to the single instance of `re.match(data)` don't have
to be repeated verbatim elsewhere. It's possible that it runs twice as
fast too, but that's the least of my concerns.
All of those advantages are retained in the one-liner too if an assignment
expression can be used in it.
> Thus the PEP's argument is that it wants to allow the faster version
> > to remain a one-liner that preserves the overall structure of the
> > version that repeats the subexpression:
> >
> > group = _m.group(1) if _m := re.match(data) else None
> >
> > That's a performance argument, not a readability one (as if you don't
> > care about performance, you can just repeat the subexpression).
>
How does that differ from the part of what I said that you did retain above?
>> sometimes it allows a more compact way of reusing an expensive
>> subexpression by giving it a name. Which they already do by giving
>> it a name in a separate statement, so the possible improvement would
>> be in brevity rather than performance.
You already realized the performance gain could be achieved by using two
statements. The _additional_ performance gain by using assignment
expressions is at best trivial (it may save a LOAD_FAST opcode to fetch the
object bound to `_m` for the `if` test).
So, no, gaining performance is _not_ the motivation here. You already had
a way to make it "run fast'. The motivation is the _brevity_ assignment
expressions allow while _retaining_ all of the two-statement form's
advantages in easier readability, easier reasoning, reduced redundancy, and
performance.
As Guido said, in the PEP, of the example you gave here:
Guido found several examples where a programmer repeated
a subexpression, slowing down the program, in order to save
one line of code
It couldn't possibly be clearer that Guido thought the programmer's
motivation was brevity ("in
Re: [Python-Dev] Help preventing SIGPIPE/SIG_DFL anti-pattern.
On 6/30/18 4:20 PM, Greg Ewing wrote: Alfred Perlstein wrote: I am asking if there's a way we can discourage the use of "signal(SIGPIPE, SIG_DFL)" unless the user really understands what they are doing. Maybe there's some way that SIGPIPEs on stdout could be handled differently by default, so that they exit silently instead of producing an ugly message. That would remove the source of pain that's leading people to do this. Thank you Greg, I can poke around into this, it would be a bit of a challenge as the descriptor which causes BrokenPipeError does not appear to be stored within the exception so differentiating it from other exceptions might be a bit tricky. I will look into this in the coming weeks. Any tips on accomplishing this? I was thinking of encoding the fd responsible for causing the error into the exception somehow and then checking to see if it was stdout, then not reporting on it. -Alfred ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
On 1 July 2018 at 02:37, Tim Peters wrote: > [Nick Coghlan] > >> ... > >> "NAME := EXPR" exists on a different level of complexity, since it > >> adds name binding in arbitrary expressions for the sake of minor > >> performance improvement in code written by developers that are > >> exceptionally averse to the use of vertical screen real estate, >> ... > > Note that PEP 572 doesn't contain a single word about "performance" (neither > that specific word nor any synonym), and I gave only one thought to it when > writing Appendix A: "is this going to slow anything down significantly?". > The answer was never "yes", which I thought was self-evident, so I never > mentioned it. Neither did Chris or Guido. > > Best I can recall, nobody has argued for it on the grounds of "performance". > except in the indirect sense that sometimes it allows a more compact way of > reusing an expensive subexpression by giving it a name. Which they already > do by giving it a name in a separate statement, so the possible improvement > would be in brevity rather than performance. The PEP specifically cites this example as motivation: group = re.match(data).group(1) if re.match(data) else None That code's already perfectly straightforward to read and write as a single line, so the only reason to quibble about it is because it's slower than the arguably less clear two-line alternative: _m = re.match(data) group = _m.group(1) if _m else None Thus the PEP's argument is that it wants to allow the faster version to remain a one-liner that preserves the overall structure of the version that repeats the subexpression: group = _m.group(1) if _m := re.match(data) else None That's a performance argument, not a readability one (as if you don't care about performance, you can just repeat the subexpression). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Help preventing SIGPIPE/SIG_DFL anti-pattern.
Alfred Perlstein wrote: I am asking if there's a way we can discourage the use of "signal(SIGPIPE, SIG_DFL)" unless the user really understands what they are doing. Maybe there's some way that SIGPIPEs on stdout could be handled differently by default, so that they exit silently instead of producing an ugly message. That would remove the source of pain that's leading people to do this. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 544 (Protocols): adding a protocol to a class post-hoc
Hi, PEP 544 specifies this address as "Discussions-To" so I hope I'm at the right address. I think protocols as defined in the PEP are a very interesting idea and I'm thinking of ways of applying them. The first use case is in the context of attrs. attrs has a number of functions that work only on attrs classes; asdict for example (turns an attrs class into a dictionary recursively). We can't really type hint this properly using nominal subtyping since attrs classes don't have an exclusive ancestor. But it sounds like we could use structural subtyping! An attrs class has a special class-level field, __attrs_attrs__, which holds the attribute definitions. So maybe we can define a protocol: class AttrsClass(Protocol): __attrs_attrs__: ClassVar[Tuple[Attribute, ...]] then we could define asdict as (simplified): def asdict(inst: AttrsClass) -> Dict[str, Any]: ... and it should work out. My question is how to actually add this protocol to attrs classes. Now, we currently have an attrs plugin in mypy so maybe some magic in there could make it happen in this particular case. My second use case is a small library I've developed for work, which basically wraps attrs and generates and sticks methods on a class for serialization/deserialization. Consider the following short program, which does not typecheck on the current mypy. class Serializable(Protocol): def __serialize__(self) -> int: ... def make_serializable(cl: Type) -> Type: cl = attr.s(cl) cl.__serialize__ = lambda self: 1 return cl @make_serializable class A: a: int = attr.ib() def serialize(inst: Serializable) -> int: return inst.__serialize__() serialize(A(1)) error: Argument 1 to "serialize" has incompatible type "A"; expected "Serializable" error: Too many arguments for "A" I have no desire to write a mypy plugin for this library. So I guess what is needed is a way to annotate the class decorator, telling mypy it's adding a protocol to a class. It seems to have trouble getting to this conclusion by itself. (The second error implies the attrs plugin doesn't handle wrapping attr.s, which is unfortunate but a different issue.) I have found this pattern of decorating classes and enriching them with additional methods at run-time really powerful, especially when used with run-time parsing of type information (that often gets a bad rep on this list, I've noticed :) The structural typing subsystem seems like a good fit for use cases like this, and I think it'd be a shame if we couldn't make it work somehow. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
On 6/30/2018 5:35 AM, Steven D'Aprano wrote:
I've given reasons why I believe that people will expect assignments in
comprehensions to occur in the local scope. Aside from the special case
of loop variables, people don't think of comprehensions as a separate
scope.
I think this is because comprehensions other than generator expressions
were originally *defined* in terms of equivalent code in the *same*
local scope, are still easily thought of in those terms, and, as I
explained in my response to Guido, could, at least in simple cases,
still be implemented in the local scope, so that assignment expressions
would be executed and affect the expected local scope without 'nonlocal'.
Generator expressions, on the other hand, have always been defined in
terms of equivalent code in a *nested* scope, and must be implemented
that way, so some re-definition and re-implementation is needed for
assignment expressions to affect the local scope in which the g.e is
defined and for that effect to be comprehensible. It might be enough to
add something like "any names that are targets of assignment expressions
are added to global or nonlocal declarations within the implementation
generator function."
If the equality [generator-expression] == list(generator-expression] is
preserved, then it could serve as the definition of the list
comprehension. The same could be true for set and dict comprehensions,
with the understanding that the equality is modified to {a:b for ...} ==
dict((a,b) for ...). It should also be mentioned that the defining
equality is not necessarily the implementation.
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
On Sat, Jun 30, 2018 at 9:43 AM Tim Peters wrote: > The attractions are instead in the areas of reducing redundancy, improving > clarity, allowing to remove semantically pointless indentation levels in > some cases, indeed trading away some horizontal whitespace in otherwise > nearly empty lines for freeing up a bit of vertical screen space, and in > the case of comprehensions/genexps adding straightforward ways to > accomplish some conceptually trivial things that at best require trickery > now (like emulating a cell object by hand). > The examples you provided (some were new in this thread, I think) are compelling. While my initial reaction to the proposal was mild horror, I'm not troubled by the scoping questions. Issues still bothering me: 1. Initial reactions from students was confusion over := vs = 2. This seems inconsistent with the push for type hints To be fair, I felt a similar gut reaction to f-strings, and now I can't live without them. Have I become a cranky old man, resistant to change? Your examples have put me into the "on the fence, slightly worried" category instead of "clearly a bad idea". On scoping, beginners seem more confused by UnboundLocalError than by variables bleeding between what they perceive as separate scopes. The concept of a scope can be tricky to communicate. Heck, I still make the mistake of looking up class attributes in instance methods as if they were globals. Same-scope is natural. Natural language is happy with ambiguity. Separate-scope is something programmers dreamed up. Only experienced C, Java, etc. programmers get surprised when they make assumptions about what syntax in Python creates separate scopes, and I'm not so worried about those folks. I remind them that the oldest versions of C didn't have block scopes (1975?) and they quiet down. The PEP lists many exclusions of where the new := operator is invalid [0]. I unfortunately didn't have a chance to read the initial discussion over the operator. I'm sure it was thorough :-). What I can observe is that each syntactical exclusion was caused by a different confusion, probably teased out by that discussion. Many exclusions means many confusions. My intuition is that the awkwardness stems from avoiding the replacement of = with :=. Languages that use := seem to avoid the Yoda-style comparison recommendation that is common to languages that use = for assignment expressions. I understand the reluctance for such a major change to the appearance of Python code, but it would avoid the laundry list of exclusions. There's some value in parsimony. Anyway, we've got some time for testing the idea on live subjects. Have a good weekend, everyone. -- Michael PS. Pepe just tied it up for Portugal vs Uruguay. Woo! ... and now Cavani scored again :-( [0] https://www.python.org/dev/peps/pep-0572/#exceptional-cases ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
[Nick Coghlan] > > ... > > "NAME := EXPR" exists on a different level of complexity, since it > > adds name binding in arbitrary expressions for the sake of minor > > performance improvement in code written by developers that are > > exceptionally averse to the use of vertical screen real estate, > ... Note that PEP 572 doesn't contain a single word about "performance" (neither that specific word nor any synonym), and I gave only one thought to it when writing Appendix A: "is this going to slow anything down significantly?". The answer was never "yes", which I thought was self-evident, so I never mentioned it. Neither did Chris or Guido. Best I can recall, nobody has argued for it on the grounds of "performance". except in the indirect sense that sometimes it allows a more compact way of reusing an expensive subexpression by giving it a name. Which they already do by giving it a name in a separate statement, so the possible improvement would be in brevity rather than performance. The attractions are instead in the areas of reducing redundancy, improving clarity, allowing to remove semantically pointless indentation levels in some cases, indeed trading away some horizontal whitespace in otherwise nearly empty lines for freeing up a bit of vertical screen space, and in the case of comprehensions/genexps adding straightforward ways to accomplish some conceptually trivial things that at best require trickery now (like emulating a cell object by hand). Calling all that "for the sake of minor performance improvements" - which isn't even in the list - is so far off base it should have left me speechless - but it didn't ;-) But now that you mention it, ya, there will be a trivial performance improvement in some cases. I couldn't care less about that, and can confidently channel that Guido doesn't either. It would remain fine by me if assignment expressions ran trivially slower. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Help preventing SIGPIPE/SIG_DFL anti-pattern.
Hello,
I'm looking for someone in the python community to help with a problem
of anti-patterns showing up dealing with SIGPIPE.
Specifically I've noticed an anti-pattern developing where folks will
try to suppress broken pipe errors written to stdout by setting
SIGPIPE's disposition to SIG_DFL. This is actually very common, and
also rather broken due to the fact that for all but the most simple text
filters this opens up a problem where the process can exit unexpectedly
due to SIGPIPE being generated from a remote connection the program makes.
I have attached a test program which shows the problem.
to use this program it takes several args.
# 1. Illustrate the 'ugly output to stderr' that folks want to avoid:
% python3 t0.py nocatch | head -1
# 2. Illustrate the anti-pattern, the program exits on about line 47
which most folks to not understand
% python3 t0.py dfl | head -1
# 3. Show a better solution where we catch the pipe error and cleanup to
avoid the message:
% python3 t0.py | head -1
I did a recent audit of a few code bases and saw this pattern pop often
often enough that I am asking if there's a way we can discourage the use
of "signal(SIGPIPE, SIG_DFL)" unless the user really understands what
they are doing.
I do have a pull req here: https://github.com/python/cpython/pull/6773
where I am trying to document this on the signal page, but I can't sort
out how to land this doc change.
thank you,
-Alfred
#
# Program showing the dangers of setting the SIG_PIPE handler to the default handler (SIG_DFL).
#
# To illustrate the problem run with:
# ./foo.py dfl
#
# The program will exit in do_network_stuff() even though there is a an "except" clause.
# The do_network_stuff() simulates a remote connection that closes before it can be written to
# which happens often enough to be a hazard in practice.
#
#
#
import signal
import sys
import socket
import os
def sigpipe_handler(sig, frame):
sys.stderr.write("Got sigpipe \n\n\n")
sys.stderr.flush()
def get_server_connection():
# simulate making a connection to a remote service that closes the connection
# before we can write to it. (In practice a host rebooting, or otherwise exiting while we are
# trying to interact with it will be the true source of such behavior.)
s1, s2 = socket.socketpair()
s2.close()
return s1
def do_network_stuff():
# simulate interacting with a remote service that closes its connection
# before we can write to it. Example: connecting to an http service and
# issuing a GET request, but the remote server is shutting down between
# when our connection finishes the 3-way handshake and when we are able
# to write our "GET /" request to it.
# In theory this function should be resilient to this, however if SIGPIPE is set
# to SIGDFL then this code will cause termination of the program.
if 'dfl' in sys.argv[1:]:
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
for x in range(5):
server_conn = get_server_connection()
sys.stderr.write("about to write to server socket...\n")
try:
server_conn.send(b"GET /")
except BrokenPipeError as bpe:
sys.stderr.write("caught broken pipe on talking to server, retrying...")
def work():
do_network_stuff()
for x in range(1):
print("y")
sys.stdout.flush()
def main():
if 'nocatch' in sys.argv[1:]:
work()
else:
try:
work()
except BrokenPipeError as bpe:
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
os.kill(os.getpid(), signal.SIGPIPE)
if __name__ == '__main__':
main()
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Help preventing SIGPIPE/SIG_DFL anti-pattern.
(sorry for the double post, looks like maybe attachments are dropped,
inlined the attachment this time.)
Hello,
I'm looking for someone in the python community to help with a problem
of anti-patterns showing up dealing with SIGPIPE.
Specifically I've noticed an anti-pattern developing where folks will
try to suppress broken pipe errors written to stdout by setting
SIGPIPE's disposition to SIG_DFL. This is actually very common, and
also rather broken due to the fact that for all but the most simple text
filters this opens up a problem where the process can exit unexpectedly
due to SIGPIPE being generated from a remote connection the program makes.
I have attached a test program which shows the problem.
to use this program it takes several args.
# 1. Illustrate the 'ugly output to stderr' that folks want to avoid:
% python3 t0.py nocatch | head -1
# 2. Illustrate the anti-pattern, the program exits on about line 47
which most folks to not understand
% python3 t0.py dfl | head -1
# 3. Show a better solution where we catch the pipe error and cleanup to
avoid the message:
% python3 t0.py | head -1
I did a recent audit of a few code bases and saw this pattern pop often
often enough that I am asking if there's a way we can discourage the use
of "signal(SIGPIPE, SIG_DFL)" unless the user really understands what
they are doing.
I do have a pull req here: https://github.com/python/cpython/pull/6773
where I am trying to document this on the signal page, but I can't sort
out how to land this doc change.
thank you,
-Alfred
=== CUT HERE ===
#
# Program showing the dangers of setting the SIG_PIPE handler to the
default handler (SIG_DFL).
#
# To illustrate the problem run with:
# ./foo.py dfl
#
# The program will exit in do_network_stuff() even though there is a an
"except" clause.
# The do_network_stuff() simulates a remote connection that closes
before it can be written to
# which happens often enough to be a hazard in practice.
#
#
#
import signal
import sys
import socket
import os
def sigpipe_handler(sig, frame):
sys.stderr.write("Got sigpipe \n\n\n")
sys.stderr.flush()
def get_server_connection():
# simulate making a connection to a remote service that closes the
connection
# before we can write to it. (In practice a host rebooting, or
otherwise exiting while we are
# trying to interact with it will be the true source of such behavior.)
s1, s2 = socket.socketpair()
s2.close()
return s1
def do_network_stuff():
# simulate interacting with a remote service that closes its connection
# before we can write to it. Example: connecting to an http
service and
# issuing a GET request, but the remote server is shutting down between
# when our connection finishes the 3-way handshake and when we are able
# to write our "GET /" request to it.
# In theory this function should be resilient to this, however if
SIGPIPE is set
# to SIGDFL then this code will cause termination of the program.
if 'dfl' in sys.argv[1:]:
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
for x in range(5):
server_conn = get_server_connection()
sys.stderr.write("about to write to server socket...\n")
try:
server_conn.send(b"GET /")
except BrokenPipeError as bpe:
sys.stderr.write("caught broken pipe on talking to server,
retrying...")
def work():
do_network_stuff()
for x in range(1):
print("y")
sys.stdout.flush()
def main():
if 'nocatch' in sys.argv[1:]:
work()
else:
try:
work()
except BrokenPipeError as bpe:
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
os.kill(os.getpid(), signal.SIGPIPE)
if __name__ == '__main__':
main()
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
On 30 June 2018 at 19:35, Steven D'Aprano wrote: > So I think Q1 is the critical one. And I think the answer is, no, > they're conceptually bloody simple. They evaluate the expression on the > right, assign it to the name on the left, and return that value. And the proposed parent local scoping in PEP 572 has the virtue of making all of the following do *exactly the same thing*, just as they would in the version without the assignment expression: ref = object() container = [item := ref] container = [x for x in [item := ref]] container = [x for x in [item := ref] for i in range(1)] container = list(x for x in [item := ref]) container = list(x for x in [item := ref] for i in range(1)) # All variants pass this assertion, keeping the implicit sublocal scope almost entirely hidden assert container == [ref] and item is ref and item is container[0] My own objections were primarily based on the code-generation-centric concept of wanting Python's statement level scoping semantics to continue to be a superset of its expression level semantics, and Guido's offer to define "__parentlocal" in the PEP as a conventional shorthand for describing comprehension style assignment expression scoping when writing out their statement level equivalents as pseudo-code addresses that. (To put it another way: it turned out it wasn't really the semantics themselves that bothered me, since they have a lot of very attractive properties as shown above, it was the lack of a clear way of referring to those semantics other than "the way assignment expressions behave in implicit comprehension and generator expression scopes"). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
On Thu, Jun 28, 2018 at 03:42:49PM -0700, Chris Barker via Python-Dev wrote:
> If we think hardly anyone is ever going to do that -- then I guess it
> doesn't matter how it's handled.
That's how you get a language with surprising special cases, gotchas and
landmines in its behaviour. (Cough PHP cough.)
It is one thing when gotchas occur because nobody thought of them, or
because there is nothing you can do about them. But I do not think it is
a good idea to *intentionally* leave gotchas lying around because "oh, I
didn't think anyone would ever do that...".
*wink*
[...]
> but here the keyword "nonlocal" is used -- you are clearly declaring that
> you are messing with a nonlocal name here -- that is a lot more obvious
> than simply using a :=
But from the point of view of somebody reading the code, there is no
need for a nonlocal declaration, since the assignment is just a local
assignment.
Forget the loop variable -- it is a special case where "practicality
beats purity" and it makes sense to have it run in a sublocal scope.
Everything else is just a local, regardless of whether it is in a
comprehension or not.
> And "nonlocal" is not used that often, and when it is it's for careful
> closure trickery -- I'm guessing := will be far more common. And, of
> course, when a newbie encounters it, they can google it and see what it
> means -- far different that seeing a := in a comprehension and
> understanding (by osmosis??) that it might make changes in the local scope.
I've given reasons why I believe that people will expect assignments in
comprehensions to occur in the local scope. Aside from the special case
of loop variables, people don't think of comprehensions as a separate
scope.
There's no Comprehension Sublocal-Local-Enclosing Local-Global-Builtin
scoping rule. (Do you want there to be?) Even *class scope* comes as an
unfamiliar surprise to people.
I do not believe that people will "intuitively" expect assignments in a
comprehension to disappear when the comprehension finishes -- I expect
that most of the time they won't even think about it, but when they do,
they'll expect it to hang around like *every other use* of assignment
expressions.
> And I don't think you can even do that with generator expressions now --
> as they can only contain expressions.
It makes me cry to think of the hours I spent -- and the brownie points
I lost with my wife -- showing how you can already simulate this with
locals() or globals(). Did nobody read it? :-(
https://mail.python.org/pipermail/python-dev/2018-June/154114.html
Yes, you can do this *right now*. We just don't because playing around
with locals() is a dodgy thing to do.
> Maybe it's only comprehensions, and maybe it'll be rare to have a confusing
> version of those, so it'll be no big deal, but this thread started talking
> about educators' take on this -- and as an educator, I think this really
> does complicate the language.
See my recent post here:
https://mail.python.org/pipermail/python-dev/2018-June/154184.html
I strongly believe that the "comprehensions are local, like everything
else" scenario is simpler and less surprising and easier to explain than
hiding assignments inside a sublocal comprehension scope that hardly
anyone even knows exists.
Especially if we end up doing it inconsistently and let variables
sometimes leak.
> Python got much of it's "fame" by being "executable pseudo code" -- its
> been moving farther and farther away from those roots. That's generally a
> good thing, as we've gain expressiveness in exchangel, but we shouldn't
> pretend it isn't happening, or that this proposal doesn't contribute to
> that trend.
I think there are two distinct forms of "complication" here.
1. Are assignment expressions in isolation complicated?
2. Given assignment expressions, can people write obfuscated,
complex code?
Of course the answer to Q2 is yes, the opportunity will be there.
Despite my general cynicism about my fellow programmers, I actually do
believe that the Python community does a brilliant job of self-policing
to prevent the worst excesses of obfuscatory one-liners. I don't think
that will change.
So I think Q1 is the critical one. And I think the answer is, no,
they're conceptually bloody simple. They evaluate the expression on the
right, assign it to the name on the left, and return that value.
Here is a question and answer:
Question: after ``result = (x := 2) + 3``, what is the value of x?
Answer: 2.
Question: what if we put the assignment inside a function call?
``f((x:=2), x+3)``
Answer: still 2.
Question: how about inside a list display? ``[1, x:=2, 3]``
Answer: still 2.
Question: what about a dict display? ``{key: x:=2}`` A tuple? A set?
Answer: still 2 to all of them.
Question: how about a list comprehension?
Answer: ah, now, that's complicated, it depends on which bit of the
comprehension you put it in,
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
On Sat, Jun 30, 2018 at 06:30:56PM +1000, Nick Coghlan wrote:
> The significant semantic differences between "{x : 1}" and "{x := 1}"
> are also rather surprising :)
*Significant* and obvious differences are good. It's the subtle
differences that you don't notice immediately that really hurt:
{x+1} versus {x-1}
x > y versus x < y
x/y versus x//y
alist = [a, b]
alist = (a, b)
Sometimes small differences in punctuation or spelling make a big
difference to semantics.
Punctuation Saves Lives!
"Let's eat, grandma!"
"Let's eat grandma!"
Unless you propose to ban all operators and insist on a minimum string
distance between all identifiers:
https://docs.python.org/3/library/os.html#os.spawnl
picking out little differences in functionality caused by little
differences in code is a game we could play all day.
At least we won't have the "=" versus "==" bug magnet from C, or the
"==" versus "===" confusion from Javascript. Compared to that, the
in-your-face obvious consequences of {x: 1} versus {x := 1} are pretty
harmless.
--
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
On 30 June 2018 at 09:49, Chris Barker - NOAA Federal via Python-Dev
wrote:
>> On Jun 28, 2018, at 8:21 PM, Tim Peters wrote:
>
> Seems it’s all been said, and Tim’s latest response made an excellent
> case for consistency.
>
> But:
>
>> Regardless of how assignment expressions work in listcomps and genexps, this
>> example (which uses neither) _will_ rebind the containing block's `x`:
>>
>> [x := 1]
>
> This reinforces my point that it’s not just about comprehensions, but
> rather that the local namespace can be altered anywhere an expression
> is used — which is everywhere.
>
> That trivial example is unsurprising, but as soon as your line of code
> gets a bit longer, it could be far more hidden.
The significant semantic differences between "{x : 1}" and "{x := 1}"
are also rather surprising :)
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
On 29 June 2018 at 08:42, Chris Barker via Python-Dev wrote: > On Thu, Jun 28, 2018 at 9:28 AM, Tim Peters wrote: >> Did adding ternary `if` (truepart if expression else falsepart) complicate >> the language significantly? > > > I don't think so -- no. For two reasons: > > 1) the final chosen form is kind of verbose, but therefor more like > "executable pseudo code" :-) As apposed to the C version, for instance. > > 2) it added one new construct, that if, when someone sees it for the first > (or twenty fifth) time and doesn't understand it, they can look it up, and > find out. and it only effects that line of code. > > So adding ANYTHING does complicate the language, by simply making it a bit > larger, but some things are far more complicating than others. It's worth noting that without the bug prone "C and A or B" construct (which gives the wrong result when "not A" is True), we'd likely never have gotten "A if C else B" (which gives the right result regardless of the truth value of A). In the case of PEP 308, the new construction roughly matched the existing idiom in expressive power, it just handled it correctly by being able to exactly match the developer's intent. "NAME := EXPR" exists on a different level of complexity, since it adds name binding in arbitrary expressions for the sake of minor performance improvement in code written by developers that are exceptionally averse to the use of vertical screen real estate, and making a couple of moderately common coding patterns (loop-and-a-half, if-elif-chains with target binding) more regular, and hence easier to spot. I think the current incarnation of PEP 572 does an excellent job of making the case that says "If we add assignment expressions, we should add them this particular way" - there are a lot of syntactic and semantic complexities to navigate, and it manages to make its way through them and still come out the other side with a coherent and self-consistent proposal that copes with some thoroughly quirky existing scoping behaviour. That only leaves the question of "Does the gain in expressive power match the increase in the cognitive burden imposed on newcomers to the language?", and my personal answer to that is still "No, I don't think it does". It isn't my opinion on that that matters, though: I think that's now going to be a conversation between Guido and folks that are actively teaching Python to new developers, and are willing to get their students involved in some experiments. Cheers, Nick. P.S. It does make me wonder if it would be possible to interest the folks behind https://quorumlanguage.com/evidence.html in designing and conducting fully controlled experiments comparing the comprehensibility of pre-PEP-572 code with post-PEP-572 code *before* the syntax gets added to the language :) -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
On Wed, Jun 27, 2018 at 09:52:43PM -0700, Chris Barker wrote:
> It seems everyone agrees that scoping rules should be the same for
> generator expressions and comprehensions,
Yes. I dislike saying "comprehensions and generator expressions" over
and over again, so I just say "comprehensions".
Principle One:
- we consider generator expressions to be a lazy comprehension;
- or perhaps comprehensions are eager generator expressions;
- either way, they behave the same in regard to scoping rules.
Principle Two:
- the scope of the loop variable stays hidden inside the
sub-local ("comprehension" or "implicit hidden function")
scope;
- i.e. it does not "leak", even if you want it to.
Principle Three:
- glossing over the builtin name look-up, calling list(genexpr)
will remain equivalent to using a list comprehension;
- similarly for set and dict comprehensions.
Principle Four:
- comprehensions (and genexprs) already behave "funny" inside
class scope; any proposal to fix class scope is beyond the,
er, scope of this PEP and can wait for another day.
So far, there should be (I hope!) no disagreement with those first four
principles. With those four principles in place, teaching and using
comprehensions (genexprs) in the absense of assignment expressions does
not need to change one iota.
Normal cases stay normal; weird cases mucking about with locals() inside
the comprehension are already weird and won't change.
> So what about:
>
> l = [x:=i for i in range(3)]
>
> vs
>
> g = (x:=i for i in range(3))
>
> Is there any way to keep these consistent if the "x" is in the regular
> local scope?
Yes. That is what closures already do.
We already have such nonlocal effects in Python 3. Move the loop inside
an inner (nested) function, and then either call it immediately to
simulate the effect of a list comprehension, or delay calling it to
behave more like a generator expression.
Of course the *runtime* effects depend on whether or not the generator
expression is actually evaluated. But that's no mystery, and is
precisely analogous to this case:
def demo():
x = 1
def inner():
nonlocal x
x = 99
inner() # call the inner function
print(x)
This prints 99. But if you comment out the call to the inner function,
it prints 1. I trust that doesn't come as a surprise.
Nor should this come as a surprise:
def demo():
x = 1
# assuming assignment scope is local rather than sublocal
g = (x:= i for i in (99,))
L = list(g)
print(x)
The value of x printed will depend on whether or not you comment out
the call to list(g).
> Note that this thread is titled "Informal educator feedback on PEP 572".
>
> As an educator -- this is looking harder an harder to explain to newbies...
>
> Though easier if any assignments made in a "comprehension" don't "leak out".
Let me introduce two more principles.
Principle Five:
- all expressions are executed in the local scope.
Principle Six:
- the scope of an assignment expression variable inside a
comprehension (genexpr) should not depend on where inside
the comprehension it sits.
Five is, I think, so intuitive that we forget about it in the same way
that we forget about the air we breathe. It would be surprising, even
shocking, if two expressions in the same context were executed in
different scopes:
result = [x + 1, x - 2]
If the first x were local and the second was global, that would be
disturbing. The same rule ought to apply if we include assignment
expressions:
result = [(x := expr) + 1, x := x - 2]
It would be disturbing if the first assignment (x := expr) executed in
the local scope, and the second (x := x - 2) failed with NameError
because it was executed in the global scope.
Or worse, *didn't* fail with NameError, but instead returned something
totally unexpected.
Now bring in a comprehension:
result = [(x := expr) + 1] + [x := x - 2 for a in (None,)]
Do you still want the x inside the comprehension to be a different x to
the one outside the comprehension? How are you going to explain that
UnboundLocalError to your students?
That's not actually a rhetorical question. I recognise that while
Principle Five seems self-evidently desirable to me, you might consider
it less important than the idea that "assignments inside comprehensions
shouldn't leak".
I believe that these two expressions should give the same results even
to the side-effects:
[(x := expr) + 1, x := x - 2]
[(x := expr) + 1] + [x := x - 2 for a in (None,)]
I think that is the simplest and most intuitive behaviour, the one
which will be the least surprising, cause the fewest unexpected
NameErrors, and be the simplest to explain.
If you still prefer the "assignments shouldn't leak" idea, consider
this: under the current implementation of comprehensions as an implicit
hidden function, the scope of a variable depends on *where* it is,
violating Principle Six.
(That was the
Re: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)
On 28 June 2018 at 08:31, Guido van Rossum wrote: > So IIUC you are okay with the behavior described by the PEP but you want an > explicit language feature to specify it? > > I don't particularly like adding a `parentlocal` statement to the language, > because I don't think it'll be generally useful. (We don't have `goto` in > the language even though it could be used in the formal specification of > `if`, for example. :-) > > But as a descriptive mechanism to make the PEP's spec clearer I'm fine with > it. Let's call it `__parentlocal` for now. It would work a bit like > `nonlocal` but also different, since in the normal case (when there's no > matching `nonlocal` in the parent scope) it would make the target a local in > that scope rather than trying to look for a definition of the target name in > surrounding (non-class, non-global) scopes. Also if there's a matching > `global` in the parent scope, `__parentlocal` itself changes its meaning to > `global`. If you want to push a target through several level of target > scopes you can do that by having a `__parentlocal` in each scope that it > should push through (this is needed for nested comprehensions, see below). > > Given that definition of `__parentlocal`, in first approximation the scoping > rule proposed by PEP 572 would then be: In comprehensions (which in my use > in the PEP 572 discussion includes generator expressions) the targets of > inline assignments are automatically endowed with a `__parentlocal` > declaration, except inside the "outermost iterable" (since that already runs > in the parent scope). > > There would have to be additional words when comprehensions themselves are > nested (e.g. `[[a for a in range(i)] for i in range(10)]`) since the PEP's > intention is that inline assignments anywhere there end up targeting the > scope containing the outermost comprehension. But this can all be expressed > by adding `__parentlocal` for various variables in various places (including > in the "outermost iterable" of inner comprehensions). > > I'd also like to keep the rule prohibiting use of the same name as a > comprehension loop control variable and as an inline assignment target; this > rule would also prohibit shenanigans with nested comprehensions (for any set > of nested comprehensions, any name that's a loop control variable in any of > them cannot be an inline assignment target in any of them). This would also > apply to the "outermost iterable". > > Does this help at all, or did I miss something? Yep, it does, and I don't think you missed anything. Using "__parentlocal" to indicate "parent local scoping semantics apply here" still gives the concept a name and descriptive shorthand for use in pseudo-code expansions of assignment expressions in comprehensions, without needing to give it an actually usable statement level syntax, similar to the way we use "_expr_result" and "_outermost_iter" to indicate references that in reality are entries in an interpreter's stack or register set, or else a pseudo-variable that doesn't have a normal attribute identifier. And if anyone does want to make the case for the syntax being generally available, they don't need to specify how it should work - they just need to provide evidence of cases where it would clarify code unrelated to the PEP 572 use case. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Intent to accept PEP 561 -- Distributing and Packaging Type Information
On 28 June 2018 at 09:11, Guido van Rossum wrote: > Well, with that, I am hereby accepting PEP 561. > > Ethan has done a tremendous job writing this PEP and implementing it, and I > am sure that package and stub authors will be very glad to hear that there > are now officially supported ways other than typeshed to distribute type > annotations. Very cool! Congratulations Ethan! Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
