Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
Guido van Rossum writes: And my mind boggles when considering a generator expression containing yield that is returned from a function. I tried this and cannot say I expected the outcome: def f(): return ((yield i) for i in range(3)) print(list(f())) In both Python 2 and Python 3 this prints [0, None, 1, None, 2, None] Even if there's a totally logical explanation for that, I still don't like it, and I think yield in a comprehension should be banned. From this it follows that we should also simply ban yield from comprehensions. Serhiy Storchaka writes: This behavior doesn't look correct to me and Ivan. The behavior is surprising, but it seems quite consistent with how generator expressions are defined in the language. A generator expression is defined by the language reference as "compact generator notation in parentheses", which yields (sic!) a "new generator object". I take that to mean that a generator expression is equivalent to defining and calling a generator function. f() can be transformed to: def f(): def _gen(): for i in range(3): ret = yield i yield ret return _gen() The transformed version shows that there are *two* yields per iteration (one explicitly written and one inserted by the transformation), which is the reason why 6 values are produced. The None values come from list constructor calling __next__() on the generator, which (as per documentation) sends None into the generator. This None value is yielded after the "i" is yielded, which is why Nones follow the numbers. Hrvoje ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python initialization and embedded Python
I proposed a PR to explicitly list functions safe to be called before Py_Initialize(): https://bugs.python.org/issue32124 https://github.com/python/cpython/pull/4540 I found more than 11 functions.. I also found variables ;-) Victor 2017-11-24 5:01 GMT+01:00 Nick Coghlan : > On 24 November 2017 at 12:21, Glenn Linderman wrote: >> >> On 11/23/2017 5:31 PM, Nick Coghlan wrote: >> >> - a consolidated list of *all* the APIs that can safely be called before >> Py_Initialize >> >> So it is interesting to know that list, of course, but the ones that are >> to be supported and documented might be a smaller list. Or might not. > > > Ah, sorry - "safely" was a bit ambiguous there. By "safely" I meant "CPython > has a regression test that ensures that particular API will keep working > before Py_Initialize(), regardless of any changes we may make to the way we > handle interpreter initialization". > > We've long had a lot of other APIs that happen to work well enough for > CPython itself to get away with using them during the startup process, but > the official position on those is "Don't count on these APIs working prior > to Py_Initialize() in the general case - we only get away with it because we > can adjust the exact order in which we do things in order to account for any > other changes that break it". > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2017-11-17 - 2017-11-24) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open6267 ( +6) closed 37610 (+57) total 43877 (+63) Open issues with patches: 2416 Issues opened (45) == #32067: Deprecate accepting unrecognized braces in regular expressions https://bugs.python.org/issue32067 opened by serhiy.storchaka #32068: textpad from curses package isn't handling backspace key https://bugs.python.org/issue32068 opened by bacher09 #32070: Clarify the behavior of the staticmethod builtin https://bugs.python.org/issue32070 opened by haikuginger #32071: Add py.test-like "-k" test selection to unittest https://bugs.python.org/issue32071 opened by jonash #32072: Issues with binary plists https://bugs.python.org/issue32072 opened by serhiy.storchaka #32073: Add copy_directory_metadata parameter to shutil.copytree https://bugs.python.org/issue32073 opened by desbma #32075: Expose ZipImporter Type Object in the include header files. https://bugs.python.org/issue32075 opened by Decorater #32076: Expose LockFile on Windows https://bugs.python.org/issue32076 opened by pitrou #32077: Documentation: Some Unicode object functions don't indicate wh https://bugs.python.org/issue32077 opened by Mathew M. #32079: version install 3.6.3 hangs in test_socket https://bugs.python.org/issue32079 opened by Raúl Alvarez #32080: Error Installing Python 3.6.3 on ubuntu 16.04 https://bugs.python.org/issue32080 opened by sachin #32081: ipaddress should support fast IP lookups https://bugs.python.org/issue32081 opened by Attila Nagy #32082: atexit module: allow getting/setting list of handlers directly https://bugs.python.org/issue32082 opened by erik.bray #32083: sqlite3 Cursor.description can't return column types https://bugs.python.org/issue32083 opened by kyoshidajp #32084: [Security] http.server can be abused to redirect to (almost) a https://bugs.python.org/issue32084 opened by vstinner #32085: [Security] A New Era of SSRF - Exploiting URL Parser in Trendi https://bugs.python.org/issue32085 opened by vstinner #32087: deprecated-removed directive generates overlapping msgids in . https://bugs.python.org/issue32087 opened by cocoatomo #32089: In developer mode (-X dev), ResourceWarning is only emited onc https://bugs.python.org/issue32089 opened by vstinner #32090: test_put() of test_multiprocessing queue tests has a race cond https://bugs.python.org/issue32090 opened by vstinner #32091: test_s_option() of test_site.HelperFunctionsTests failed on x8 https://bugs.python.org/issue32091 opened by vstinner #32092: mock.patch with autospec does not consume self / cls argument https://bugs.python.org/issue32092 opened by cbelu #32093: macOS: implement time.thread_time() using thread_info() https://bugs.python.org/issue32093 opened by vstinner #32096: Py_DecodeLocale() fails if used before the runtime is initiali https://bugs.python.org/issue32096 opened by eric.snow #32097: doctest does not consider \r\n a https://bugs.python.org/issue32097 opened by X-Istence #32098: Hardcoded value in Lib/test/test_os.py:L1324:URandomTests.get_ https://bugs.python.org/issue32098 opened by hackan #32101: Add PYTHONDEVMODE=1 to enable the developer mode https://bugs.python.org/issue32101 opened by vstinner #32102: Add "capture_output=True" option to subprocess.run https://bugs.python.org/issue32102 opened by ncoghlan #32104: add method throw() to asyncio.Task https://bugs.python.org/issue32104 opened by Oleg K2 #32107: test_uuid uses the incorrect bitmask https://bugs.python.org/issue32107 opened by barry #32108: configparser bug: section is emptied if you assign a section t https://bugs.python.org/issue32108 opened by simonltwick #32110: Make codecs.StreamReader.read() more compatible with read() of https://bugs.python.org/issue32110 opened by serhiy.storchaka #32112: Should uuid.UUID() accept another UUID() instance? https://bugs.python.org/issue32112 opened by mjpieters #32113: Strange behavior with await in a generator expression https://bugs.python.org/issue32113 opened by levkivskyi #32114: The get_event_loop change in bpo28613 did not update the docum https://bugs.python.org/issue32114 opened by r.david.murray #32115: Ignored SIGCHLD causes asyncio.Process.wait to hang forever https://bugs.python.org/issue32115 opened by rogpeppe #32116: CSV import and export simplified https://bugs.python.org/issue32116 opened by paullongnet #32117: Tuple unpacking in return and yield statements https://bugs.python.org/issue32117 opened by dacut #32118: Docs: add note about sequence comparisons containing non-order https://bugs.python.org/issue32118 opened by Dubslow #32119: test_notify_all() of test_multiprocessing_forkserver failed on https://bugs.python.org/issue32119 opened by vstinner #32121: tracemalloc.Tra
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
OK, so my 24 hours are over :-) On 24 November 2017 at 01:50, Nick Coghlan wrote: > On 23 November 2017 at 23:04, Ivan Levkivskyi > wrote: > >> I don't see why this particular case qualifies for such a radical measure >> as an exception to syntactic rules, >> instead of just fixing it (sorry Nick :-) >> > > I've posted in more detail about this to the issue tracker, but the > argument here is: because making it behave differently from the way it does > now while still hiding the loop iteration variable potentially requires > even more radical revisions to the lexical scoping rules :) > If somebody can come up with a clever trick to allow yield inside a > comprehension to jump levels in a relatively intuitive way, that would > actually be genuinely cool, but the lexical scoping rules mean it's > trickier than it sounds. > "potentially" is the key word here. The plan is to avoid "more radical revisions". > Now that I frame the question that way, though, I'm also remembering that > we didn't have "yield from" yet when I wrote the current comprehension > implementation, and given that, it may be as simple as having an explicit > yield expression in a comprehension imply delegation to a subgenerator. > My understanding is that this is exactly how async comprehensions are currently implemented and why they work as one would naively expect, i.e. `await` is "bound" to the surrounding async def, not to the implicit scope async def. So that an async comprehension is just equivalent to a for-loop. However, although "implicit yield from" solution is simpler than the one proposed by Serhiy, I still more like the latter one. The former has its strange cases, for example I mentioned before in this thread: >>> async def f(): ... for i in range(3): ... yield i ... >>> async def g(): ... return [(yield i) async for i in f()] ... >>> g().send(None) Traceback (most recent call last): File "", line 1, in File "", line 2, in g TypeError: object async_generator can't be used in 'await' expression My understanding is that the strange error is exactly because of the implicit `yield from`. With Serhiy's approach this would work. Another minor bonus of Serhiy's idea is a performance gain: we will not push an execution frame. -- Ivan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
The more I hear about this topic, the more I think that `await`, `yield` and `yield from` should all be banned from occurring in all comprehensions and generator expressions. That's not much different from disallowing `return` or `break`. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 25 November 2017 at 01:22, Guido van Rossum wrote: > The more I hear about this topic, the more I think that `await`, `yield` > and `yield from` should all be banned from occurring in all comprehensions > and generator expressions. That's not much different from disallowing > `return` or `break`. > > IIUC this would essentially mean rejecting PEP 530. What do you think about banning `await`, `yield` and `yield from` only from generator expressions? Comprehensions can be fixed (just make them equivalent to for-loops without leaks as discussed). -- Ivan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum wrote: > The more I hear about this topic, the more I think that `await`, `yield` and > `yield from` should all be banned from occurring in all comprehensions and > generator expressions. That's not much different from disallowing `return` > or `break`. I would say that banning `yield` and `yield from` is like banning `return` and `break`, but banning `await` is like banning function calls. There's no reason for most users to even know that `await` is related to generators, so a rule disallowing it inside comprehensions is just confusing. AFAICT 99% of the confusion around async/await is because people think of them as being related to generators, when from the user point of view it's not true at all and `await` is just a funny function-call syntax. Also, at the language level, there's a key difference between these cases. A comprehension has implicit `yield`s in it, and then mixing in explicit `yield`s as well obviously leads to confusion. But when you use an `await` in a comprehension, that turns it into an async generator expression (thanks to PEP 530), and in an async generator, `yield` and `await` use two separate, unrelated channels. So there's no confusion or problem with having `await` inside a comprehension. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Fri, Nov 24, 2017 at 7:22 PM, Guido van Rossum wrote: > The more I hear about this topic, the more I think that `await`, `yield` and > `yield from` should all be banned from occurring in all comprehensions and > generator expressions. That's not much different from disallowing `return` > or `break`. IMO disallowing using await in comprehensions would be a huge mistake. I've personally seen a lot of code that uses the syntax. Moreover, I haven't seen any complaints about await expressions in comprehensions in this thread or *anywhere else*. In this thread I only see complaints about 'yield' expression, and that's easy to understand -- yield in comprehensions is just unusable. 'await' is fundamentally different, because it's essentially a calling convention. While "provisional" status for a PEP ultimately gives us power to remove every trace of it in the next release, I think that it decisions like that have to be based on some commonly reported problems and complaints. My 2c. Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 11/24/2017 04:22 PM, Guido van Rossum wrote:
The more I hear about this topic, the more I think that `await`, `yield` and
`yield from` should all be banned from
occurring in all comprehensions and generator expressions. That's not much
different from disallowing `return` or `break`.
For me, the deciding factor would be the the affect upon:
1) the containing function (if any); and
2) the result of the genexp/comprehension
I think of generator expressions / comprehensions as self-contained units of code that will have no (side-)effects upon
the containing function and/or surrounding code (and a containing function is not necessary), and that each will return
the type expressed by the syntax.
In other words:
[ l for l in ...] -> list
{s for s in ...] -> set
{k:v for k, v in ...] -> dict
(g for g in ...] -> genexp
Since 'yield' and 'yield from' invalidate those, I'm in favor of declaring them
syntax errors.
If 'await' does not invalidate the above constraints then I'm fine with
allowing them.
--
~Ethan~
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Fri, Nov 24, 2017 at 5:06 PM, Nathaniel Smith wrote: > On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum wrote: >> The more I hear about this topic, the more I think that `await`, `yield` and >> `yield from` should all be banned from occurring in all comprehensions and >> generator expressions. That's not much different from disallowing `return` >> or `break`. > > I would say that banning `yield` and `yield from` is like banning > `return` and `break`, but banning `await` is like banning function > calls. I agree. I was going to make the point earlier in the thread that using "await" can mostly just be thought of as a delayed function call, but it didn't seem at risk of getting banned until Guido's comment so I didn't say anything (and there were too many comments anyways). I think it's in a different category for that reason. It's much easier to reason about than, say, "yield" and "yield from". --Chris ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 25 November 2017 at 11:04, Ivan Levkivskyi wrote:
> On 25 November 2017 at 01:22, Guido van Rossum wrote:
>
>> The more I hear about this topic, the more I think that `await`, `yield`
>> and `yield from` should all be banned from occurring in all comprehensions
>> and generator expressions. That's not much different from disallowing
>> `return` or `break`.
>>
>>
> IIUC this would essentially mean rejecting PEP 530.
> What do you think about banning `await`, `yield` and `yield from` only
> from generator expressions?
> Comprehensions can be fixed (just make them equivalent to for-loops
> without leaks as discussed).
>
I'm leaning towards this as well - the immediate evaluation on
comprehensions means that injecting an automatic "yield from" when will do
the right thing due to this component of PEP 380:
https://www.python.org/dev/peps/pep-0380/#the-refactoring-principle
Restating how folks intuitively think Py3 comprehensions containing "yield"
and "await" should work (''__var" indicates hidden variables only
accessible to the interpreter):
result = [(yield x) for x in iterable]
# Readers expect this to work much like
__listcomp1 = []
for __x1 in iterable:
__listcomp1.append(yield __x1)
result = __listcomp1
PEP 380's refactoring principle then means the above is required to be
equivalent to:
def __listcomp1(__outermost_iterable):
__result = []
for x in __outermost_iterable:
__result.append(yield x)
return __result
result = yield from __listcomp1(iterable)
Which means that only missing piece in the current implementation is
inserting the implicit "yield from" into the function containing the
comprehension. No need for new lexical scoping conventions, so need for
syntactical special cases, just teaching this part of the compiler to
delegate to a subgenerator correctly. The compiler should just need a small
local adjustment to check whether the implicit nested scope is a generator
scope or not (and any Python source compiler, including CPython's, already
needs to track that information in order to ensure it emits a generator
function instead of a regular synchronous function).
>From a language evolution perspective, such a change is relatively
straightforward to explain on the basis of "Comprehensions became
implicitly nested scopes in 3.0, but we didn't implement PEP 380's 'yield
from' construct until Python 3.3, and it then took us until 3.7 to realise
we could combine them to make yield expressions inside comprehensions
behave more intuitively".
For "await", I'm pretty sure the current semantics are actually OK, but I
confess I haven't been in a position to have to explain them to anyone
either.
I honestly don't see a good way to salvage yield or yield from inside
generator expressions though. If we dropped the implicit "yield" when an
explicit one is present, that would break the symmetry between "[(yield x)
for x in iterable]" and "list((yield x) for x in iterable)", while
inserting an implicit "yield from" (as I'm now suggesting we do for
comprehensions) wouldn't work at all (since a genexp returns a generator
*fuction*, not a generator-iterator object). I'm also pretty sure it will
be difficult for the compiler to tell the difference between explicit and
implicit yield expressions in the subfunction (as there's nothing else
anywhere in the language that needs to make that distinction).
Cheers,
Nick.
--
Nick Coghlan | [email protected] | Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 2017-11-25 02:21, Chris Jerdonek wrote: On Fri, Nov 24, 2017 at 5:06 PM, Nathaniel Smith wrote: On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum wrote: The more I hear about this topic, the more I think that `await`, `yield` and `yield from` should all be banned from occurring in all comprehensions and generator expressions. That's not much different from disallowing `return` or `break`. I would say that banning `yield` and `yield from` is like banning `return` and `break`, but banning `await` is like banning function calls. I agree. I was going to make the point earlier in the thread that using "await" can mostly just be thought of as a delayed function call, but it didn't seem at risk of getting banned until Guido's comment so I didn't say anything (and there were too many comments anyways). I think it's in a different category for that reason. It's much easier to reason about than, say, "yield" and "yield from". +1 Seeing "await" there didn't/doesn't confuse me; seeing "yield" or "yield from" does. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum wrote: > The more I hear about this topic, the more I think that `await`, `yield` > and `yield from` should all be banned from occurring in all comprehensions > and generator expressions. That's not much different from disallowing > `return` or `break`. > >From the responses it seems that I tried to simplify things too far. Let's say that `await` in comprehensions is fine, as long as that comprehension is contained in an `async def`. While we *could* save `yield [from]` in comprehensions, I still see it as mostly a source of confusion, and the fact that the presence of `yield [from]` *implicitly* makes the surrounding `def` a generator makes things worse. It just requires too many mental contortions to figure out what it does. I still propose to rule out all of the above from generator expressions, because those can escape from the surrounding scope. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 25 November 2017 at 13:30, Guido van Rossum wrote:
>
> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum wrote:
>>
>> The more I hear about this topic, the more I think that `await`, `yield` and
>> `yield from` should all be banned from occurring in all comprehensions and
>> generator expressions. That's not much different from disallowing `return`
>> or `break`.
>
>
> From the responses it seems that I tried to simplify things too far. Let's
> say that `await` in comprehensions is fine, as long as that comprehension is
> contained in an `async def`. While we *could* save `yield [from]` in
> comprehensions, I still see it as mostly a source of confusion, and the fact
> that the presence of `yield [from]` *implicitly* makes the surrounding `def`
> a generator makes things worse. It just requires too many mental contortions
> to figure out what it does.
While I'm OK with ruling the interaction of all of these
subexpressions with generator expression too confusingly weird to be
supportable, in https://bugs.python.org/issue10544?#msg306940 I came
up with an example for the existing comprehension behaviour that I'm
hesitant to break without a clear replacement: code that wraps the
*current* comprehension behaviour in an explicit "yield from"
expression.
That is, code like:
def example():
comp1 = yield from [(yield x) for x in ('1st', '2nd')]
comp2 = yield from [(yield x) for x in ('3rd', '4th')]
return comp1, comp2
Creates a generator that returns a 2-tuple containing two 2-item
lists, but the exact contents of those lists depend on the values you
send into the generator while it is running.
The basic principle behind the construct is that a comprehension
containing a yield expression essentially defines an inline
subgenerator, so you then have to yield from that subgenerator in
order to produce the desired list.
If the implicit "yield from" idea seems too magical, then the other
direction we could go is to make the immediate "yield from" mandatory,
such that leaving it out produces a SyntaxWarning in 3.7, and then a
SyntaxError in 3.8+. (Something like "'yield from' clause missing
before subgenerator comprehension")
While we don't generally like bolting two pieces of syntax together
like that, I think it's defensible in this case based on the fact that
allowing "[expr for var in iterable]" to ever return anything other
than a list (and similarly for the other comprehension types) is
genuinely confusing.
The nicest aspect of the "an explicit and immediate 'yield from' is
required when using subgenerator comprehensions" approach is that the
above syntax already works today, and has actually worked since 3.3 -
we'd just never put the pieces together properly to establish this as
a potential pattern for comprehension use in coroutines.
Cheers,
Nick.
--
Nick Coghlan | [email protected] | Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Fri, Nov 24, 2017 at 9:04 PM, Nick Coghlan wrote:
> def example():
> comp1 = yield from [(yield x) for x in ('1st', '2nd')]
> comp2 = yield from [(yield x) for x in ('3rd', '4th')]
> return comp1, comp2
Isn't this a really confusing way of writing
def example():
return [(yield '1st'), (yield '2nd')], [(yield '3rd'), (yield '4th')]
?
-n
--
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 565: show DeprecationWarning in __main__ (round 2)
This is a new version of the proposal to show DeprecationWarning in __main__.
The proposal itself hasn't changed (it's still recommending a new
entry in the default filter list), but there have been several updates
to the PEP text based on further development work and comments in the
initial thread:
- there's now a linked issue and reference implementation
- it turns out we don't currently support the definition of module
based filters at startup time, so I've explicitly noted the relevant
enhancement that turned out to be necessary (allowing
plain-string-or-compiled-regex in stored filter definitions where we
currently only allow compiled regexes)
- I've noted the intended changes to the warnings-related documentation
- I've noted a couple of other relevant changes that Victor already
implemented for 3.7
- I've noted that the motivation for the change in 2.7 & 3.1 covered
all Python applications, not just developer tools (developer tools
just provide a particularly compelling example of why "revert to the
Python 2.6 behaviour" isn't a good answer)
Cheers,
Nick.
=
PEP: 565
Title: Show DeprecationWarning in __main__
Author: Nick Coghlan
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Nov-2017
Python-Version: 3.7
Post-History: 12-Nov-2017, 25-Nov-2017
Abstract
In Python 2.7 and Python 3.2, the default warning filters were updated to hide
DeprecationWarning by default, such that deprecation warnings in development
tools that were themselves written in Python (e.g. linters, static analysers,
test runners, code generators), as well as any other applications that merely
happened to be written in Python, wouldn't be visible to their users unless
those users explicitly opted in to seeing them.
However, this change has had the unfortunate side effect of making
DeprecationWarning markedly less effective at its primary intended purpose:
providing advance notice of breaking changes in APIs (whether in CPython, the
standard library, or in third party libraries) to users of those APIs.
To improve this situation, this PEP proposes a single adjustment to the
default warnings filter: displaying deprecation warnings attributed to the main
module by default.
This change will mean that code entered at the interactive prompt and code in
single file scripts will revert to reporting these warnings by default, while
they will continue to be silenced by default for packaged code distributed as
part of an importable module.
The PEP also proposes a number of small adjustments to the reference
interpreter and standard library documentation to help make the warnings
subsystem more approachable for new Python developers.
Specification
=
The current set of default warnings filters consists of::
ignore::DeprecationWarning
ignore::PendingDeprecationWarning
ignore::ImportWarning
ignore::BytesWarning
ignore::ResourceWarning
The default ``unittest`` test runner then uses ``warnings.catch_warnings()``
``warnings.simplefilter('default')`` to override the default filters while
running test cases.
The change proposed in this PEP is to update the default warning filter list
to be::
default::DeprecationWarning:__main__
ignore::DeprecationWarning
ignore::PendingDeprecationWarning
ignore::ImportWarning
ignore::BytesWarning
ignore::ResourceWarning
This means that in cases where the nominal location of the warning (as
determined by the ``stacklevel`` parameter to ``warnings.warn``) is in the
``__main__`` module, the first occurrence of each DeprecationWarning will once
again be reported.
This change will lead to DeprecationWarning being displayed by default for:
* code executed directly at the interactive prompt
* code executed directly as part of a single-file script
While continuing to be hidden by default for:
* code imported from another module in a ``zipapp`` archive's ``__main__.py``
file
* code imported from another module in an executable package's ``__main__``
submodule
* code imported from an executable script wrapper generated at installation time
based on a ``console_scripts`` or ``gui_scripts`` entry point definition
As a result, API deprecation warnings encountered by development tools written
in Python should continue to be hidden by default for users of those tools
While not its originally intended purpose, the standard library documentation
will also be updated to explicitly recommend the use of
``FutureWarning`` (rather
than ``DeprecationWarning``) for backwards compatibility warnings that are
intended to be seen by *users* of an application.
This will give the following three distinct categories of backwards
compatibility warning, with three different intended audiences:
* ``PendingDeprecationWarning``: reported by default only in test runners that
override the default set of warning filters. The intended audience is Python
developers that take an active interest in ensuring the future compa
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 25 November 2017 at 15:27, Nathaniel Smith wrote:
> On Fri, Nov 24, 2017 at 9:04 PM, Nick Coghlan wrote:
>> def example():
>> comp1 = yield from [(yield x) for x in ('1st', '2nd')]
>> comp2 = yield from [(yield x) for x in ('3rd', '4th')]
>> return comp1, comp2
>
> Isn't this a really confusing way of writing
>
> def example():
> return [(yield '1st'), (yield '2nd')], [(yield '3rd'), (yield '4th')]
A real use case wouldn't be iterating over hardcoded tuples in the
comprehensions, it would be something more like:
def example(iterable1, iterable2):
comp1 = yield from [(yield x) for x in iterable1]
comp2 = yield from [(yield x) for x in iterable2]
return comp1, comp2
Defining an interesting for loop isn't the point of the example though
- it's just to show that if you're inside a generator, you can already
make a subgenerator comprehension do something sensible by sticking
"yield from" in front of it (and have actually been able to do so
since 3.3, when "yield from" was first introduced).
Cheers,
Nick.
--
Nick Coghlan | [email protected] | Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Fri, Nov 24, 2017 at 9:39 PM, Nick Coghlan wrote:
> On 25 November 2017 at 15:27, Nathaniel Smith wrote:
>> On Fri, Nov 24, 2017 at 9:04 PM, Nick Coghlan wrote:
>>> def example():
>>> comp1 = yield from [(yield x) for x in ('1st', '2nd')]
>>> comp2 = yield from [(yield x) for x in ('3rd', '4th')]
>>> return comp1, comp2
>>
>> Isn't this a really confusing way of writing
>>
>> def example():
>> return [(yield '1st'), (yield '2nd')], [(yield '3rd'), (yield '4th')]
>
> A real use case
Do you have a real use case? This seems incredibly niche...
> wouldn't be iterating over hardcoded tuples in the
> comprehensions, it would be something more like:
>
> def example(iterable1, iterable2):
> comp1 = yield from [(yield x) for x in iterable1]
> comp2 = yield from [(yield x) for x in iterable2]
> return comp1, comp2
I submit that this would still be easier to understand if written out like:
def map_iterable_to_yield_values(iterable):
"Yield the values in iterable, then return a list of the values sent back."
result = []
for obj in iterable:
result.append(yield obj)
return result
def example(iterable1, iterable2):
values1 = yield from map_iterable_to_yield_values(iterable1)
values2 = yield from map_iterable_to_yield_values(iterable2)
return values1, values2
-n
--
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 25 November 2017 at 16:18, Nathaniel Smith wrote:
> On Fri, Nov 24, 2017 at 9:39 PM, Nick Coghlan wrote:
>> On 25 November 2017 at 15:27, Nathaniel Smith wrote:
>>> On Fri, Nov 24, 2017 at 9:04 PM, Nick Coghlan wrote:
def example():
comp1 = yield from [(yield x) for x in ('1st', '2nd')]
comp2 = yield from [(yield x) for x in ('3rd', '4th')]
return comp1, comp2
>>>
>>> Isn't this a really confusing way of writing
>>>
>>> def example():
>>> return [(yield '1st'), (yield '2nd')], [(yield '3rd'), (yield '4th')]
>>
>> A real use case
>
> Do you have a real use case? This seems incredibly niche...
That's not how backwards compatibility works - we were suggesting
getting rid of this syntax, because there was no current way to make
it do anything sensible.
It turns out there is a way to make it behave reasonably - you just
need to stick "yield from" in front of it, and it goes back to being
equivalent to the corresponding for loop (the same as the synchronous
version).
>> wouldn't be iterating over hardcoded tuples in the
>> comprehensions, it would be something more like:
>>
>> def example(iterable1, iterable2):
>> comp1 = yield from [(yield x) for x in iterable1]
>> comp2 = yield from [(yield x) for x in iterable2]
>> return comp1, comp2
>
> I submit that this would still be easier to understand if written out like:
>
> def map_iterable_to_yield_values(iterable):
> "Yield the values in iterable, then return a list of the values sent
> back."
> result = []
> for obj in iterable:
> result.append(yield obj)
> return result
>
> def example(iterable1, iterable2):
> values1 = yield from map_iterable_to_yield_values(iterable1)
> values2 = yield from map_iterable_to_yield_values(iterable2)
> return values1, values2
The same can be said for comprehensions in general. Composing them
with coroutines certainly doesn't make either easier to understand,
but I don't think replacing the comprehension with its full imperative
form is particularly helpful in aiding that understanding.
Cheers,
Nick.
--
Nick Coghlan | [email protected] | Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sat, 25 Nov 2017 03:03:02 + MRAB wrote: > On 2017-11-25 02:21, Chris Jerdonek wrote: > > On Fri, Nov 24, 2017 at 5:06 PM, Nathaniel Smith wrote: > >> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum > >> wrote: > >>> The more I hear about this topic, the more I think that `await`, `yield` > >>> and > >>> `yield from` should all be banned from occurring in all comprehensions and > >>> generator expressions. That's not much different from disallowing `return` > >>> or `break`. > >> > >> I would say that banning `yield` and `yield from` is like banning > >> `return` and `break`, but banning `await` is like banning function > >> calls. > > > > I agree. I was going to make the point earlier in the thread that > > using "await" can mostly just be thought of as a delayed function > > call, but it didn't seem at risk of getting banned until Guido's > > comment so I didn't say anything (and there were too many comments > > anyways). > > > > I think it's in a different category for that reason. It's much easier > > to reason about than, say, "yield" and "yield from". > > > +1 > > Seeing "await" there didn't/doesn't confuse me; seeing "yield" or "yield > from" does. +1 as well. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sat, 25 Nov 2017 17:10:12 +1000
Nick Coghlan wrote:
> On 25 November 2017 at 16:18, Nathaniel Smith wrote:
> > On Fri, Nov 24, 2017 at 9:39 PM, Nick Coghlan wrote:
> >> On 25 November 2017 at 15:27, Nathaniel Smith wrote:
> >>> On Fri, Nov 24, 2017 at 9:04 PM, Nick Coghlan wrote:
> >>>
> def example():
> comp1 = yield from [(yield x) for x in ('1st', '2nd')]
> comp2 = yield from [(yield x) for x in ('3rd', '4th')]
> return comp1, comp2
> >>>
> >>> Isn't this a really confusing way of writing
> >>>
> >>> def example():
> >>> return [(yield '1st'), (yield '2nd')], [(yield '3rd'), (yield '4th')]
> >>>
> >>
> >> A real use case
> >
> > Do you have a real use case? This seems incredibly niche...
>
> That's not how backwards compatibility works - we were suggesting
> getting rid of this syntax, because there was no current way to make
> it do anything sensible.
>
> It turns out there is a way to make it behave reasonably - you just
> need to stick "yield from" in front of it, and it goes back to being
> equivalent to the corresponding for loop (the same as the synchronous
> version).
I don't know if it behaves reasonably, but it's entirely unreadable to
me. I think there's a case for discouraging unreadable constructs.
Regards
Antoine.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
