[Python-Dev] Re: Tagged pointer experiment: need help to optimize

[Ivan Levkivskyi]

On Wed, 23 Sep 2020 at 11:42, Mark Shannon  wrote:

>
>
> On 23/09/2020 9:47 am, Ivan Levkivskyi wrote:
> >
> >
> > On Wed, 23 Sep 2020 at 01:19, Guido van Rossum  > <mailto:gu...@python.org>> wrote:
> >
> > On Tue, Sep 22, 2020 at 4:58 PM Greg Ewing
> > mailto:greg.ew...@canterbury.ac.nz>>
> > wrote:
> >
> > What are you trying to achieve by using tagged pointers?
> >
> > It seems to me that in a dynamic environment like Python, tagged
> > pointer tricks are only ever going to reduce memory usage, not
> > make anything faster, and in fact can only make things slower
> > if applied everywhere.
> >
> >
> > Hm... mypyc (an experimental Python-to-C compiler bundled with mypy)
> > uses tagged pointers to encode integers up to 63 bits. I think it's
> > done for speed, and it's probably faster in part because it avoids
> > slow memory accesses. But (a) I don't think there's overflow
> > checking, and (b) mypyc is very careful that tagged integers are
> > never passed to the CPython runtime (since mypyc apps link with an
> > unmodified CPython runtime for data types, compatibility with
> > extensions and pure Python code). Nevertheless I think it puts your
> > blanket claim in some perspective.
> >
> >
> > FWIW mypyc does overflow checking, see  e.g.
> > https://github.com/python/mypy/blob/master/mypyc/lib-rt/CPy.h#L168,
> also
> > tagged pointers did bring some speed wins, not big however. My
> > expectation is that speed wins may be even more modest without static
> > information that mypyc has.
> >
> > // offtopic below, sorry
> >
> > In general, I think any significant perf wins will require some static
> > info given to the Python compiler. I was thinking a long while ago about
> > defining some kind of a standard protocol so that static type checkers
> > can optionally provide some info to the Python compiler (e.g.
> > pre-annotated ASTs and pre-annotated symbol tables), and having a lot of
> > specialized bytecodes. For example, if we know x is a list and y is an
> > int, we can emit a special byte code for x[y] that will call
> > PyList_GetItem, and will fall back to PyObject_GetItem in rare cases
> > when type-checker didn't infer right (or there were some Any involved).
> > Another example is having special byte codes for direct pointer access
> > to instance attributes, etc. The main downside of such ideas is it will
> > take a lot of work to implement.
>
> Performance improvements do not need static annotations, otherwise PyPy,
> V8 and luajit wouldn't exist.
> Even HotSpot was originally based on a VM for SmallTalk.
>

Sure, but JIT optimizations assume there are some "hot spots" in the code
where e.g. a function is called in a loop, so that type information can be
gathered and re-used.
The problem is that in my experience there are many applications where this
is not the case: there are no major hot spots. For such applications JITs
will not be efficient,
while static annotations will work.

Another thing is that making CPython itself JITted may be even harder than
adding some (opt-in) static based optimizations, but
I am clearly biased here.

Actually what would be really cool is having both: i.e. have a JIT that
would use static annotations to speed-up the warmup significantly.
I don't know if anyone tried something like this, but it doesn't sound
impossible.

--
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JKF6LKGZL6S46B43HH3SADXKK26XGSKU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Tagged pointer experiment: need help to optimize

[Ivan Levkivskyi]

On Wed, 23 Sep 2020 at 01:19, Guido van Rossum  wrote:

> On Tue, Sep 22, 2020 at 4:58 PM Greg Ewing 
> wrote:
>
>> What are you trying to achieve by using tagged pointers?
>>
>> It seems to me that in a dynamic environment like Python, tagged
>> pointer tricks are only ever going to reduce memory usage, not
>> make anything faster, and in fact can only make things slower
>> if applied everywhere.
>>
>
> Hm... mypyc (an experimental Python-to-C compiler bundled with mypy) uses
> tagged pointers to encode integers up to 63 bits. I think it's done for
> speed, and it's probably faster in part because it avoids slow memory
> accesses. But (a) I don't think there's overflow checking, and (b) mypyc is
> very careful that tagged integers are never passed to the CPython runtime
> (since mypyc apps link with an unmodified CPython runtime for data types,
> compatibility with extensions and pure Python code). Nevertheless I think
> it puts your blanket claim in some perspective.
>

FWIW mypyc does overflow checking, see  e.g.
https://github.com/python/mypy/blob/master/mypyc/lib-rt/CPy.h#L168, also
tagged pointers did bring some speed wins, not big however. My expectation
is that speed wins may be even more modest without static information that
mypyc has.

// offtopic below, sorry

In general, I think any significant perf wins will require some static info
given to the Python compiler. I was thinking a long while ago about
defining some kind of a standard protocol so that static type checkers can
optionally provide some info to the Python compiler (e.g. pre-annotated
ASTs and pre-annotated symbol tables), and having a lot of specialized
bytecodes. For example, if we know x is a list and y is an int, we can emit
a special byte code for x[y] that will call PyList_GetItem, and will fall
back to PyObject_GetItem in rare cases when type-checker didn't infer right
(or there were some Any involved). Another example is having special byte
codes for direct pointer access to instance attributes, etc. The main
downside of such ideas is it will take a lot of work to implement.

--
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AUVHLLTEARU6HGMEFPPA5Q744O33WHOH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Ivan Levkivskyi]

These are good news. I think the new parser is indeed both simpler and more
flexible - great!

--
Ivan



On Thu, 2 Apr 2020 at 19:19, Guido van Rossum  wrote:

> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros
> Nikolaou and myself have been working on a new parser for CPython. We are
> now far enough along that we present a PEP we've written:
>
> https://www.python.org/dev/peps/pep-0617/
>
> Hopefully the PEP speaks for itself. We are hoping for a speedy resolution
> so we can land the code we've written before 3.9 beta 1.
>
> If people insist I can post a copy of the entire PEP here on the list, but
> since a lot of it is just background information on the old LL(1) and the
> new PEG parsing algorithms, I figure I'd spare everyone the need of reading
> through that. Below is a copy of the most relevant section from the PEP.
> I'd also like to point out the section on performance (which you can find
> through the above link) -- basically performance is on a par with that of
> the old parser.
>
> ==
> Migration plan
> ==
>
> This section describes the migration plan when porting to the new
> PEG-based parser
> if this PEP is accepted. The migration will be executed in a series of
> steps that allow
> initially to fallback to the previous parser if needed:
>
> 1.  Before Python 3.9 beta 1, include the new PEG-based parser machinery
> in CPython
> with a command-line flag and environment variable that allows
> switching between
> the new and the old parsers together with explicit APIs that allow
> invoking the
> new and the old parsers independently. At this step, all Python APIs
> like ``ast.parse``
> and ``compile`` will use the parser set by the flags or the
> environment variable and
> the default parser will be the current parser.
>
> 2.  After Python 3.9 Beta 1 the default parser will be the new parser.
>
> 3.  Between Python 3.9 and Python 3.10, the old parser and related code
> (like the
> "parser" module) will be kept until a new Python release happens
> (Python 3.10). In
> the meanwhile and until the old parser is removed, **no new Python
> Grammar
> addition will be added that requires the peg parser**. This means that
> the grammar
> will be kept LL(1) until the old parser is removed.
>
> 4.  In Python 3.10, remove the old parser, the command-line flag, the
> environment
> variable and the "parser" module and related code.
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/HOZ2RI3FXUEMAT4XAX4UHFN4PKG5J5GR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/772SIMSA6B7P43ICIHVWPGSB3OYP5SSP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request to postpone some Python 3.9 incompatible changes to Python 3.10

[Ivan Levkivskyi]

On Fri, 24 Jan 2020 at 10:05, Victor Stinner  wrote:

>
> The proposal is to give one year to project maintainers to drop Python
> 2.7 support, since Python 2.7 end of support just happened a few weeks
> ago (2020-01-01).
>

IMO creating this kind of "gray areas" in support and deprecation issues is
bad.
What this will create is just more sources for arguing/debates. Once
deprecation or EoL schedule is set,
it is best to align to it. Discussions about the schedules should happen
when setting them, not when
the deadline is coming.

Also I am not sure it is really worth it. For example, importing ABCs
directly from collections was deprecated 8 years ago,
what would 1 extra year change?

--
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YIBWOAXT5LGP3WM4T4DR5WJIVU7VVF7Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should set objects maintain insertion order too?

[Ivan Levkivskyi]

On Tue, 17 Dec 2019 at 11:48, Paul Moore  wrote:

> On Tue, 17 Dec 2019 at 11:13, Larry Hastings  wrote:
> > I lack this strongly mathematical view of sets others have espoused;
> instead I view them more like "dicts without values".  I'm therefore
> disgruntled by this inconsistency between what are I see as closely related
> data structures, and it makes sense to me that they'd maintain their
> insertion order the same way that dictionaries now do.
>
> I'll admit to a mathematical background, which probably influences my
> views. But I view sets as "collections" of values - values are "in"
> the set or not. I don't see them operationally, in the sense that you
> add and remove things from the set. Adding and removing are mechanisms
> for changing the set, but they aren't fundamentally what sets are
> about (which for me is membership). So insertion order on sets is
> largely meaningless for me (it doesn't matter *how* it got there, what
> matters is whether it's there now).
>
> Having said that, I also see dictionaries as mappings from key to
> value, so insertion order is mostly not something I care about for
> dictionaries either. I can see the use cases for ordered dictionaries,
> and now we have insertion order, it's useful occasionally, but it's
> not something that was immediately obvious to me based on my
> intuition. Similarly, I don't see sets as *needing* insertion order,
> although I concede that there probably are use cases, similar to
> dictionaries. The question for me, as with dictionaries, is whether
> the cost (in terms of clear semantics, constraints on future
> implementation choices, and actual performance now) is justified by
> the additional capabilities (remembering that personally, I'd consider
> insertion order preservation as very much a niche requirement).
>

As a random data point, I often see the pattern where one needs to remove
duplicates
from the list while preserving the order of first appearance.
This is for example needed to get stability in various type-checking
situations (like union items, type variables in base classes, type queries
etc.)

One can write a four line helper to achieve this, but I can see that having
order preserving set could be useful.
So again, it is "nice to have but not really critical".

--
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/STQVILVCC7ECBRF53ZQCIWQS45IYOIKJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Exposing Tools/parser/unparse.py in the stdlib?

[Ivan Levkivskyi]

Just wanted to add my +1 to this idea. Moving it to ast module will add
little maintenance costs, but will make it easier to use.

--
Ivan



On Tue, 19 Nov 2019 at 00:46, Pablo Galindo Salgado 
wrote:

> Hi,
>
> What do people feel about exposing Tools/parser/unparse.py in the standard
> library? Here is my initial rationale:
>
> * The tool already needs to be maintained and updated as is tested as part
> of the test suite.
> * I have used the tool almost all the time I needed to deal with AST
> transformations.
> * The public interface will have a very low surface API, keeping
> maintaining it (the public interface) a very small burden IMHO.
>
> We could add the public interface to the ast.py module or a new one if
> people feel strongly about it.
>
> Does anyone feel strongly against this or have any objection that I am not
> contemplating?
>
> Regards from rainy London,
> Pablo Galindo Salgado
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/JAQDBMC23HW2PQ27HQNJ7G244T423IDD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GKFY5NBBD5VL5EODKMH7D2TFNB7RWZEJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Implementation of PEP-0604

[Ivan Levkivskyi]

To add some more input about forward references, here are couple of
thoughts:

1. I think ideally we should support forward references to the same extent
they are currently supported:
i.e. Union[None, "ForwardRef"], Union["OtherForwardRef", None], and
Union["ForwardRef", "OtherForwardRef"]
should all be supported. The last one looks the hardest, in the sense that
it looks like we will need
to add `str.__or__()` to support these forms.

2. Forward references to type variables were never supported and we
shouldn't start supporting this
(even if it accidentally worked in some situations). Type variables should
always be defined before use.
So I think we should not worry about point 3 in Richard's e-mail.

Philippe, I think it makes sense to reflect these points to the PEP draft.

--
Ivan



On Thu, 14 Nov 2019 at 02:23, Richard Eames  wrote:

> Thanks for the encouragement!
>
> I've been working some more on it since I had some more free cycles in the
> last few days and I think I've got to the limit of my capabilities. I think
> I've got it to a point where it needs more eyes because my experience level
> writing code in the python interpreter is pretty low, so I know that I have
> holes in there, especially around using INCREF/DECREF.
> I'm currently stuck on 3 things:
>
> 1. repr( pickle.loads( pickle.dumps(int | str) ) ) causes an infinite
> recursion that I can't figure out. I might end up re-specializing `repr()`
> again since it isn't defined in the PEP.
>
> 2.
> - `None | "forwardref"` and `None | None`
> - `"forwardref" | None` and `"forwardRefA" | "forwardRefB"`
> I've had a go at adding a `type.__ror__` as suggested by GvR, but I think
> I'm missing something.
>
> 3. type parameters
> - I'm not sure how to handle this:
> ```
> MaybeT = None | "T" # creates shadow union
> T = TypeVar('T')
> maybe_int = MaybeT[int] # should this stay in shadow land, or go back to
> typing.py
> isinstance(1, maybe_int)
> isinstance(1, MaybeT[int])
> ```
> I have a couple options for this:
> - implement the type substitution in c; which can be done, but feels like
> overkill?
> - store the type parameters on the shadow object and vivify when needed
> (which will end up being in the isinstance call)
> - implement the __getitem__ as a call into the typing.py library and
> promote as it's requested. This is how I've currently implemented it.
>
> I'm happy to keep going on this if/when needed,
>
> Richard
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RYH3TYK5RXII5ESUYMTJAROBUAE3SY7H/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7P5VPD4AODJXDCBU6Y7H66TWVZEPXLI2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Implementation of PEP-0604

[Ivan Levkivskyi]

On Thu, 31 Oct 2019 at 23:20, Nick Coghlan  wrote:

> [...snip...]
>
>>
>> It's not enough to move the typing classes, I must move
>> functools.lru_cache() and dependencies, collections.abs.Mapping and
>> dependencies, and track the frame level.
>>
>> *It's too big for me.*
>>
>>
> It's certainly not an easy project to tackle.
>
> For some of the specific points you raise though, you wouldn't translate
> the existing Python code directly to C.
>

I agree with Nick, this is indeed big, but not impossible. If you are not
sure yet whether you will work on implementation, you can focus on
polishing the PEP text, and then if it is accepted and you will decide to
give implementation to someone else, we will find a volunteer.

--
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AKXBKSBYESEJVKAAPCXHG4Z5EWOS54MM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migrate typing in builtin

[Ivan Levkivskyi]

You will need to rewrite most of things in C.

--
Ivan


On Tue 8 Oct 2019, 08:53 Philippe Prados,  wrote:

> Ok,
>
> But _GenericAlias and dependencies are written with Python
> (Lib/typing.py), not with C.
> So, I must rewrite the _GenericAlias in C or it's possible to merge the C
> and Python in builtin and add a direct reference to _GenericAlias with C,
> and add the reference in builtin module ?
>
> Philippe
>
>
> Le lun. 7 oct. 2019 à 22:58, Random832  a écrit :
>
>> On Mon, Oct 7, 2019, at 12:02, Philippe Prados wrote:
>> > Because this PEP propose to accept, for all classes
>> > assert isinstance("", int | str)
>> > assert issubclass(int, int | str)
>> > and add an operator __or__() for type type.
>> > def f(list: List[int | str], param: int | None) -> float | str:
>> > pass
>>
>> Oh, sorry, I didn't realize that this also included the | operator, I
>> thought this was just for isinstance("", Union[int, str]).
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GD7WXPD26VUPMZT6WAATCJJBB42DDYYQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5UT7NKKB3KM6K67YL2ERMLX6YKLMAPAB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migrate typing in builtin

[Ivan Levkivskyi]

On Mon, 7 Oct 2019 at 17:16, Guido van Rossum  wrote:

> Hi Philippe,
>
> Maybe the implementation of `_GenericAlias` should live in
> Objects/typobject.c, and be a part of the public API. Then the "|" overload
> on PyType_Type (also there) can reference it directly, it can be exported
> from Python/bltinsmodule.c, and typing can reference it as a builtin.
>

That file (typeobject) is already quite big, so I think we might consider
also creating a new file Objects/genericalias.c, but it is just a minor
wish. I think both ways are fine.

--
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6N2KUJOQRTXTDTDXJL57GECGA5YETA7P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: typing: how to use names in result-tuples?

[Ivan Levkivskyi]

On Mon, 12 Aug 2019 at 11:24, Christian Tismer  wrote:

> On 12.08.19 10:52, Ivan Levkivskyi wrote:
> > On Thu, 8 Aug 2019 at 17:17, Christian Tismer  > <mailto:tis...@stackless.com>> wrote:
> >
> > Yes, that's what I mean.
> > Probably retval or whatever people prefer would be adequate,
> > with a special rule if that name is taken.
> >
> > I think btw. that using StructSequence(somename=sometype, ..., ) that
> > does a dict lookup is quite appealing. It returns a class like
> > stat_result, but the function call shows its arguments (see answer to
> > Ronald).
> >
> > Ciao -- Chris
> >
> >
> > Just a little comment: there is a (vague) plan to add a feature (key
> > types) to the type system that will allow user-defined constructs
> > similar to NamedTuple (for example StructSequence you mention).
> > However, taking into account current schedule it is unlikely it will be
> > added before mid-2020, so your best bet is indeed using ad-hoc named
> tuples.
>
> Interesting! Can I read something more about this? I'm curious ;-)
>

Some preliminary ideas were discussed at one of recent typing summits, see
https://paper.dropbox.com/doc/Type-system-improvements--AfxnxPOd_hhYvtamiI9nOdZ9Ag-HHOkniMG9WcCgS0LzXZAe

--
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4TSPDB7U7T5EJ6IWGD3UDI3F2WM3ZZDO/

[Python-Dev] Re: typing: how to use names in result-tuples?

[Ivan Levkivskyi]

On Thu, 8 Aug 2019 at 17:17, Christian Tismer  wrote:

> Yes, that's what I mean.
> Probably retval or whatever people prefer would be adequate,
> with a special rule if that name is taken.
>
> I think btw. that using StructSequence(somename=sometype, ..., ) that
> does a dict lookup is quite appealing. It returns a class like
> stat_result, but the function call shows its arguments (see answer to
> Ronald).
>
> Ciao -- Chris
>

Just a little comment: there is a (vague) plan to add a feature (key types)
to the type system that will allow user-defined constructs similar to
NamedTuple (for example StructSequence you mention).
However, taking into account current schedule it is unlikely it will be
added before mid-2020, so your best bet is indeed using ad-hoc named tuples.

--
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O2ZFIAUHR2Q4TUXQUOOOW4KE4T7BPEWM/

[Python-Dev] Re: Enum and annotations

[Ivan Levkivskyi]

On Mon, 10 Jun 2019 at 22:37, Ethan Furman  wrote:

> Greetings!
>
> I saw my first annotation mix-up with regards to Enum today:
>
>https://stackoverflow.com/q/56532591/208880
>
>  #enum import:
>  from enum import Enum
>
>  # enum definition:
>  class Status(Enum):
>  on: 1
>  off: 2
>
> My question for the group:
>
> Is this worth "fixing" or should I leave it alone?  On the one hand it
> seems like an obvious error, but on the other I hate getting in the way of
> needs I haven't thought of.
>

>From the point of view of static type checkers this is an error as well
(e.g. mypy gives an error at the definition site because 1 and 2 are not
valid types).
I don't have any opinion about whether this should be a runtime error as
well.

--
Ivan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/67F4LWTCYBMWJKESYXB6V65S7ZECWADM/

Re: [Python-Dev] Definition of equality check behavior

[Ivan Levkivskyi]

On Tue, 7 May 2019 at 22:31, Jordan Adler  wrote:

> Hey folks!
>
> Through the course of work on the future polyfills
>  that mimic
> the behavior of Py3 builtins across versions of Python, we've discovered
> that the equality check behavior of at least some builtin types do not
> match the documented core data model
> .
>
> Specifically, a comparison between a primitive (int, str, float were
> tested) and an object of a different type always return False, instead of
> raising a NotImplementedError.  Consider `1 == '1'` as a test case.
>
> Should the data model be adjusted to declare that primitive types are
> expected to fallback to False, or should the cpython primitive type's
> __eq__ implementation fallback to raise NotImplementedError?
>
> Reasonable people could disagree about the right approach, but my distaste
> for silent failures leads me to recommend that the implementation be
> adjusted to return NotImplementedError as a fallback, and to document that
> the operands should not be coerced to the same type prior to comparison
> (enforcing a stricter equality check). This will of course require a
> deprecation protocol.
>
> Alternatively some new equality operator could be used to specify the
> level of coercion/type checking desired (currently Python has 'is' and
> '==').
>

I don't think there is a chance this can be changed at runtime. OTOH, mypy
has a (pretty recent, so better use master) flag --strict-equality that
uses some heuristics to detect suspicious comparisons, identity checks, and
container checks (all of these may return False at runtime while some
people want them to be an error, e.g. b'abc' == 'abc', '1' in [1, 2, 3],
etc).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any core dev event plans for EP19?

[Ivan Levkivskyi]

Hi,

I want to come to EP this year, but didn't register yet, is registration
already open?

--
Ivan



On Thu, 25 Apr 2019 at 15:01, Stefan Behnel  wrote:

> Hi core devs,
>
> there are several core dev events happening at the US PyCon this year, so I
> was wondering if we could organise something similar at EuroPython. Does
> anyone have any plans or ideas already? And, how many of us are planning to
> attend EP19 in Basel this year? Unless there's something already going on
> that I missed, I can (try to) set up a poll on dpo to count the interest
> and collect ideas.
>
> Sprints would probably be a straight-forward option, a mentoring session
> could be another, a language summit or PEP discussion/mentoring round would
> also be a possibility. More ideas welcome.
>
> Stefan
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concurrent.futures: no type discovery for PyCharm

[Ivan Levkivskyi]

Mypy doesn't use source code of stlib for analysis and instead uses stub
files from typeshed. IIUC PyCharm can also do that (i.e. use the typeshed
stubs).
The whole idea of typeshed is to avoid changing stlib solely for the sake
of static analysis. Please open an issue on typeshed an/or PyCharm tracker.

--
Ivan



On Tue, 23 Apr 2019 at 20:38, Ilya Kamenshchikov 
wrote:

> How would we answer the same question if it was not a part of stdlib?
> I am not sure it is fair to expect of Pycharm to parse / execute the
> __getattr__ on modules, as more elaborate implementation could even contain
> different types per some condition at the runtime or anything at all.
> The code:
>
> TYPE_CHECKING = False
> if TYPE_CHECKING:
> from .process import ProcessPoolExecutor
> from .thread import ThreadPoolExecutor
>
> works for type checking in PyCharm and is fast.
>
> This is how stdlib can be an example to how side libraries can be 
> implemented. If we can agree that this is the only clear, performant and 
> sufficient code - then perhaps modifying mypy is a reasonable price to pay.
>
> Perhaps this particular case can be just patched locally by PyCharm
> /JetBrains, but what is a general solution to this class of problems?
>
> Best Regards,
> --
> Ilya Kamenshchikov
>
>
> On Tue, Apr 23, 2019 at 7:05 PM Guido van Rossum  wrote:
>
>> In any case I think this should be filed (by the OP) as an issue against
>> JetBrains' PyCharm issue tracker. Who knows they may be able to
>> special-case this in a jiffy. I don't think we should add any clever hacks
>> to the stdlib for this.
>>
>> On Tue, Apr 23, 2019 at 9:59 AM Nathaniel Smith  wrote:
>>
>>> On Tue, Apr 23, 2019, 05:09 Andrew Svetlov 
>>> wrote:
>>>
 I agree that `from typing import TYPE_CHECKING` is not desirable from
 the import time reduction perspective.

 From my understanding code completion *can* be based on type hinting
 to avoid actual code execution.
 That's why I've mentioned that typeshed already has the correct type
 information.

 if TYPE_CHECKING:
 import ...

 requires mypy modification.

 if False:
 import ...

 Works right now for stdlib (mypy ignores stdlib code but uses typeshed
 anyway) but looks a little cryptic.
 Requires a comprehensive comment at least.

>>>
>>> Last time I looked at this, I'm pretty sure `if False` broke at least
>>> one popular static analysis tool (ie it was clever enough to ignore
>>> everything inside `if False`) – I think either pylint or jedi?
>>>
>>> I'd suggest checking any clever hacks against at least: mypy,
>>> pylint/astroid, jedi, pyflakes, and pycharm. They all have their own static
>>> analysis engines, and each one has its own idiosyncratic quirks.
>>>
>>> We've struggled with this a *lot* in trio, and eventually ended up
>>> giving up on all forms of dynamic export cleverness; we've even banned the
>>> use of __all__ entirely. Static analysis has gotten good enough that users
>>> won't accept it not working, but it hasn't gotten good enough to handle
>>> anything but the simplest static exports in a reliable way:
>>> https://github.com/python-trio/trio/pull/316
>>> https://github.com/python-trio/trio/issues/542
>>>
>>> The stdlib has more leeway because when tools don't work on the stdlib
>>> then they tend to eventually add workarounds. I'm just saying, think twice
>>> before diving into clever hacks to workaround static analysis limits, and
>>> if you're going to do it then be careful to be thorough. You're basically
>>> relying on undocumented bugs, and it gets really messy really quickly.
>>>
>>> -n
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> *Pronouns: he/him/his **(why is my pronoun here?)*
>> 
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Source of truth for C-API

[Ivan Levkivskyi]

I added to extra parameters to each, see
https://github.com/python/cpython/pull/11605/files#diff-d350c56a842065575842defb8aaa9f27

--
Ivan

On Wed, 23 Jan 2019 at 16:41, Victor Stinner  wrote:

> What is your change? Did you remove these functions? Change their
> parameters?
>
> Victor
>
> Le mer. 23 janv. 2019 à 16:24, Ivan Levkivskyi  a
> écrit :
> >
> > Thanks for advice Victor and Steve!
> >
> > I looked at the list, and the two functions I mentioned are not in the
> list. So I assume the best strategy for now is to wait until first
> alpha-beta releases are out, and see if anyone complains.
> >
> > --
> > Ivan
> >
> >
> >
> > On Wed, 23 Jan 2019 at 06:58, Steve Dower 
> wrote:
> >>
> >> On 22Jan.2019 1517, Victor Stinner wrote:
> >> > I'm not aware of any tool to automatically list the content of the C
> API.
> >>
> >> The shell script attached to https://bugs.python.org/issue23903 should
> >> be able to do it with different preprocessor values (we originally
> >> intended to detect inconsistencies in the stable API, but when we found
> >> lots of existing inconsistencies we couldn't agree on how to deal with
> >> them).
> >>
> >> Cheers,
> >> Steve
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Source of truth for C-API

[Ivan Levkivskyi]

Thanks for advice Victor and Steve!

I looked at the list, and the two functions I mentioned are not in the
list. So I assume the best strategy for now is to wait until first
alpha-beta releases are out, and see if anyone complains.

--
Ivan



On Wed, 23 Jan 2019 at 06:58, Steve Dower  wrote:

> On 22Jan.2019 1517, Victor Stinner wrote:
> > I'm not aware of any tool to automatically list the content of the C API.
>
> The shell script attached to https://bugs.python.org/issue23903 should
> be able to do it with different preprocessor values (we originally
> intended to detect inconsistencies in the stable API, but when we found
> lots of existing inconsistencies we couldn't agree on how to deal with
> them).
>
> Cheers,
> Steve
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Source of truth for C-API

[Ivan Levkivskyi]

Hi,

I recently modified signatures of two functions PyNode_AddChild() and
PyParser_AddToken().
These two functions are not listed in C-API docs on docs.python.org, and
are not included in Python.h. However, their names look like they may be
part of C-API. So there appeared a question, what is the source of truth
for C-API, is there an official list? Or is it just the content of Python.h?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Postponed annotations break inspection of dataclasses

[Ivan Levkivskyi]

Do we have a b.p.o. issue about this? If no, then I would recommend to open
one, so that we will not loose track of this.

--
Ivan



On Sat, 22 Sep 2018 at 16:32, David Hagen  wrote:

> The new postponed annotations have an unexpected interaction with
> dataclasses. Namely, you cannot get the type hints of any of the data
> classes methods.
>
> For example, I have some code that inspects the type parameters of a
> class's `__init__` method. (The real use case is to provide a default
> serializer for the class, but that is not important here.)
>
> ```
> from dataclasses import dataclass
> from typing import get_type_hints
>
> class Foo:
> pass
>
> @dataclass
> class Bar:
> foo: Foo
>
> print(get_type_hints(Bar.__init__))
> ```
>
> In Python 3.6 and 3.7, this does what is expected; it prints `{'foo':
> , 'return': }`.
>
> However, if in Python 3.7, I add `from __future__ import annotations`,
> then this fails with an error:
>
> ```
> NameError: name 'Foo' is not defined
> ```
>
> I know why this is happening. The `__init__` method is defined in the
> `dataclasses` module which does not have the `Foo` object in its
> environment, and the `Foo` annotation is being passed to `dataclass` and
> attached to `__init__` as the string `"Foo"` rather than as the original
> object `Foo`, but `get_type_hints` for the new annotations only does a name
> lookup in the module where `__init__` is defined not where the annotation
> is defined.
>
> I know that the use of lambdas to implement PEP 563 was rejected for
> performance reasons. I could be wrong, but I think this was motivated by
> variable annotations because the lambda would have to be constructed each
> time the function body ran. I was wondering if I could motivate storing the
> annotations as lambdas in class bodies and function signatures, in which
> the environment is already being captured and is code that usually only
> runs once.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] switch statement

[Ivan Levkivskyi]

> There have been several promising posts to python-ideas about the much
more powerful idea of a "match" statement

I actually even started working on a PEP about this (pattern matching), but
then decided to postpone it because it is unlikely that anything of this
size can be discussed/accepted in current situation.
We can return back to the idea when decision-making model will clarify.

--
Ivan



On Fri, 21 Sep 2018 at 22:12, Guido van Rossum  wrote:

> There's already a rejected PEP about a switch statement:
> https://www.python.org/dev/peps/pep-3103/. There's no point bringing this
> up again unless you have a new use case.
>
> There have been several promising posts to python-ideas about the much
> more powerful idea of a "match" statement. Please search for those before
> re-posting on python-ideas.
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Do we really need a ":" in ":="?

[Ivan Levkivskyi]

On 12 July 2018 at 16:41, Victor Stinner  wrote:

> 2018-07-12 17:14 GMT+02:00 Abdur-Rahmaan Janhangeer  >:
> > sorry for reviving the dead but community acceptance, a fundamental pep
> > principle has not been respected for 572
> >
> > also 29 core devs dislike vs 3 like
>
> [...] *as the PEP evolved* in the meanwhile.


Yes, the PEP has improved significantly since that time. My guess is the
same poll taken now could give an opposite result.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tone it down on Twitter?

[Ivan Levkivskyi]

Just to clarify, the club of three in _my_ twit are the three authors of
the PEP.

Also, how about you not telling me what to say?

--
Ivan



On 4 July 2018 at 16:48, Stefan Krah  wrote:

>
> Apparently I have made it into "club of three who don't care much about
> opinions of others" for the crime of a single +0.5 for PEP-572 without
> participating in the discussion at all (neither did Jason).
>
> https://twitter.com/SerhiyStorchaka/status/1014274554452209665
>
> This is a new high for Twitter gossip.  Well done.  Perhaps in the next
> vote
> the politbureau can indicate the intended outcome beforehand so we know how
> to vote.
>
>
>
> Thanks,
>
> Stefan Krah
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> levkivskyi%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 544 (Protocols): adding a protocol to a class post-hoc

[Ivan Levkivskyi]

On 30 June 2018 at 23:54, Tin Tvrtković  wrote:

> [...]
>
> An attrs class has a special class-level field, __attrs_attrs__, which
> holds the attribute definitions. So maybe we can define a protocol:
>
> class AttrsClass(Protocol):
> __attrs_attrs__: ClassVar[Tuple[Attribute, ...]]
>
> then we could define asdict as (simplified):
>
> def asdict(inst: AttrsClass) -> Dict[str, Any]:
> ...
>
> and it should work out. My question is how to actually add this protocol
> to attrs classes. Now, we currently have an attrs plugin in mypy so maybe
> some magic in there could make it happen in this particular case.
>

Just add a Var with an appropriate name and type to the TypeInfo. This is
literary a dozen lines of code, you can ask on mypy tracker or typing
Gitter chat for more help.


> My second use case is a small library I've developed for work, which
> basically wraps attrs and generates and sticks methods on a class for
> serialization/deserialization. Consider the following short program, which
> does not typecheck on the current mypy.
>
> class Serializable(Protocol):
> def __serialize__(self) -> int:
> ...
>
> def make_serializable(cl: Type) -> Type:
> cl = attr.s(cl)
> cl.__serialize__ = lambda self: 1
> return cl
>
>
> @make_serializable
> class A:
> a: int = attr.ib()
>
>
> def serialize(inst: Serializable) -> int:
> return inst.__serialize__()
>
>
> serialize(A(1))
>
> error: Argument 1 to "serialize" has incompatible type "A"; expected
> "Serializable"
> error: Too many arguments for "A"
>
> I have no desire to write a mypy plugin for this library. So I guess what
> is needed is a way to annotate the class decorator, telling mypy it's
> adding a protocol to a class. It seems to have trouble getting to this
> conclusion by itself.
>

A proper solution for this would be to introduce intersection types, and
type your decorator as following:

T = TypeVar('T')
def make_serializable(cls: Type[T]) -> Type[Intersection[T, Serializable]]:
...

However, intersection types are unlikely to appear in mypy this year. In
best case they could appear around mid-2019, so you are better with writing
a plugin for now.


> (The second error implies the attrs plugin doesn't handle wrapping attr.s,
> which is unfortunate but a different issue.)
>

 Your decorator is typed as (Type) -> Type, thats it, the function is a
black box for mypy (with few special exceptions), if some effect of a
function is not declared in its signature, then it is lost forever.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Intent to accept PEP 561 -- Distributing and Packaging Type Information

[Ivan Levkivskyi]

Congrats Ethan,

Well done! I think PEP 561 will significantly simplify typing third party
modules.

--
Ivan



On 28 June 2018 at 00:11, Guido van Rossum  wrote:

> Well, with that, I am hereby accepting PEP 561.
>
> Ethan has done a tremendous job writing this PEP and implementing it, and
> I am sure that package and stub authors will be very glad to hear that
> there are now officially supported ways other than typeshed to distribute
> type annotations.
>
> Congrats Ethan!
>
> --Guido
>
> On Mon, Jun 25, 2018 at 12:15 PM Guido van Rossum 
> wrote:
>
>> OK, last call! I'll accept the current draft tomorrow unless someone
>> pushes back.
>>
>> On Fri, Jun 22, 2018 at 8:37 AM Nick Coghlan  wrote:
>>
>>> On 23 June 2018 at 01:16, Guido van Rossum  wrote:
>>> > That sounds like you're supporting PEP 561 as is, right?
>>>
>>> Aye, I'm personally fine with it - we do need to do something about
>>> automatically reserving the derived names on PyPI, but I don't think
>>> that's a blocker for the initial PEP acceptance (instead, it will go
>>> the other way: PEP acceptance will drive Warehouse getting updated to
>>> handle the convention already being adopted by the client tools).
>>>
>>> > Excuse my
>>> > ignorance, but where are API testing stub interfaces described or used?
>>>
>>> They're not - it's just the context for Donald referring to "stubs" as
>>> being a general technical term with other meanings beyond the "type
>>> hinting stub file" one.
>>>
>>> As such, there's three parts to explaining why we're not worried about
>>> the terminology clash:
>>>
>>> - Ethan searched for projects called "*-stubs" or "*_stubs" and didn't
>>> find any, so the practical impact of any terminology clash will be low
>>> - there isn't an established need to automatically find testing stub
>>> libraries based on an existing project name the way there is for type
>>> hints
>>> - even if such a need did arise in the future, the "py.typed" marker
>>> file and the different file extension for stub files within a package
>>> still gives us an enormous amount of design flexibility
>>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> levkivskyi%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3.7 backwards incompatible

[Ivan Levkivskyi]

This particular breakage is explicitly listed in PEP 560, see an example
with List and List[int] in
https://www.python.org/dev/peps/pep-0560/#backwards-compatibility-and-impact-on-users-who-don-t-use-typing

In general, isinstance() with typing types should be avoided when possible
(Mark Shannon who is the BDFL delegate for PEP 484 wanted to prohibit it
completely, but in the end we kept only the bare minimum, like your first
example).

When designing/implementing PEP 560 I realised it will be impossible to
keep 100% backwards compatibility. I tried to preserve 99% of public APIs,
but since isinstance() is already discouraged, it fell into remaining 1%.

A possible workaround is to use `typing_inspect` library on PyPI
(disclaimer: I originally wrote it). You can use `get_origin()` function to
extract the runtime class that corresponds to a given type. It works with
both 3.6 and 3.7 and tries its best to return the relevant runtime class
(with few exceptions, see docstring), for every version, for example on
Python 3.7 `get_origin(List[int])` return `list`. The return of
get_origin() should be usable in all runtime context including
`isinstance()`.

Btw, I actually like the new behaviour. After PEP 560 types are no more
classes, which emphasises that they should be used in static context, if
one wants to do something in runtime, one needs to use an explicit
conversion to a runtime class.

--
Ivan



On 23 June 2018 at 23:10, Guido van Rossum  wrote:

> First, the typing module is still provisional, so there is no strict
> backwards compatibility guarantee.
>
> With that out of the way, I can reproduce your problem, and I assume it's
> caused by the implementation of PEP 560, which is meant to speed up the
> typing module (among other goals).
>
> I'm wondering about your claim that this breaks many "libraries that use
> annotations". Most code using annotations would not need to use
> issubclass() in this way. Where exactly did you encounter this?
>
> I'm CC'ing Ivan Levkivskyi, who knows the relevant code better and will be
> able to explain whether this is an oversight or an intentional regression.
>
> On Sat, Jun 23, 2018 at 2:59 PM Rokker Ruslan 
> wrote:
>
>> Python 3.7 in the status of RC, but I do not see information about the
>> fact that python3.7 is backwards incompatible with python3.5.
>>
>> $ ~ python3.5
>> Python 3.5.2 (default, Nov 23 2017, 16:37:01)
>> [GCC 5.4.0 20160609] on linux
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> from typing import List
>> >>> class MyList(list):
>> ... pass
>> ...
>> >>> issubclass(MyList, List)
>> True
>> >>> issubclass(List, MyList)
>> False
>> >>>
>>
>> $ ~ python3.7
>> Python 3.7.0rc1+ (heads/3.7:3747dd16d5, Jun 22 2018, 22:53:42)
>> [GCC 5.4.0 20160609] on linux
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> from typing import List
>> >>> class MyList(list):
>> ... pass
>> ...
>> >>> issubclass(MyList, List)
>> True
>> >>> issubclass(List, MyList)
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> TypeError: issubclass() arg 1 must be a class
>> >>>
>>
>> And so with all types of "typing" module.
>> This breaks down many libraries that use annotations as part of the
>> functionality if we now can't use typing.* into issubclass function.
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> guido%40python.org
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Ivan Levkivskyi]

On 19 June 2018 at 13:02, Nick Coghlan  wrote:

> On 19 June 2018 at 16:12, INADA Naoki  wrote:
> >
> > On Tue, Jun 19, 2018 at 2:56 PM Jeroen Demeyer 
> wrote:
> >>
> >> On 2018-06-18 16:55, INADA Naoki wrote:
> >> > Speeding up most python function and some bultin functions was very
> >> > significant.
> >> > But I doubt making some 3rd party call 20% faster can make real
> >> > applications significant faster.
> >>
> >> These two sentences are almost contradictory. I find it strange to claim
> >> that a given optimization was "very significant" in specific cases while
> >> saying that the same optimization won't matter in other cases.
> >
> >
> > It's not contradictory because there is basis:
> >
> >   In most real world Python application, number of calling Python
> methods or
> >   bulitin functions are much more than other calls.
> >
> > For example, optimization for bulitin `tp_init` or `tp_new` by FASTCALL
> was
> > rejected because it's implementation is complex and it's performance
> gain is
> > not significant enough on macro benchmarks.
> >
> > And I doubt number of 3rd party calls are much more than calling builtin
> > tp_init or tp_new.
>
> I don't think this assumption is correct, as scientific Python
> software spends a lot of time calling other components in the
> scientific Python stack, and bypassing the core language runtime
> entirely.
>
>
A recent Python survey by PSF/JetBrains shows that almost half of current
Python
users are using it for data science/ML/etc. For all these people most of
the time is spent
on calling C functions in extensions.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] FINAL WEEK FOR 3.7.0 CHANGES!

[Ivan Levkivskyi]

> But cases not supported before 3.7 (like List[int]) now produce fragile
pickles.

List[int] pickled in 3.7 can't be un-pickled in 3.6, but I wouldn't worry
too much about this because it never worked in 3.6.
I remember you proposed using __getitem__ in __reduce__, but I am not sure
it is a better way, although it will fix the above problem.

I don't think this one is a blocker and we can move this discussion back to
b.p.o., unless you have some particular concerns.

The AST one however looks more serious.

--
Ivan



On 24 May 2018 at 12:26, Serhiy Storchaka  wrote:

> 24.05.18 19:02, Ned Deily пише:
>
>> On May 24, 2018, at 11:35, Serhiy Storchaka  wrote:
>>
>>> I have doubts about two issues. I feel the responsibility for them
>>> because I had the opportunity to solve them before, but I lost it.
>>>
>> [...]
>>
>> Serhiy, what are the bugs.python.org issue numbers for these?  Are they
>> marked as "release blocker"?
>>
>
> For docstring in AST: https://bugs.python.org/issue32911
>
> Inada's patch looked complex (actually it mostly restored the code before
> his previous change). We didn't know about IPython and we decided that it
> is not worth to change this code at this stage (after beta2). And
> definitely it will be later to do this after rc1.
>
> For pickling of typing types: https://bugs.python.org/issue32873
>
> Ivan fixed cases supported before 3.7. They now are backward and forward
> compatible. But cases not supported before 3.7 (like List[int]) now produce
> fragile pickles.
>
>
> ___
> python-committers mailing list
> python-committ...@python.org
> https://mail.python.org/mailman/listinfo/python-committers
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] FINAL WEEK FOR 3.7.0 CHANGES!

[Ivan Levkivskyi]

> 2. Pickle support in typing is not perfect. I was going to fix it (I had
almost ready code), but lost a chance of doing this before. It can be
changed in 3.7.1, but this means that pickles of some derived typing types
created in 3.7.0 will be not compatible with future versions (may be 3.7.1
will not break compatibility, but it will be broken in future because we
will not specially supported compatibility with 3.7.0).

I think I had fixed this one. At least the examples reported on typing
tracker are now fixed.
Do you have some other examples that still fail?

--
Ivan



On 24 May 2018 at 12:02, Ned Deily  wrote:

> On May 24, 2018, at 11:35, Serhiy Storchaka  wrote:
> > I have doubts about two issues. I feel the responsibility for them
> because I had the opportunity to solve them before, but I lost it.
> [...]
>
> Serhiy, what are the bugs.python.org issue numbers for these?  Are they
> marked as "release blocker"?
>
> --
>   Ned Deily
>   n...@python.org -- []
>
> ___
> python-committers mailing list
> python-committ...@python.org
> https://mail.python.org/mailman/listinfo/python-committers
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Slow down...

[Ivan Levkivskyi]

On 7 May 2018 at 17:32, Brett Cannon  wrote:

>
>
> On Mon, 7 May 2018 at 08:18 João Santos  wrote:
>
>> Hi,
>>
>> I would like to see this go even further and have a tick-tock approach to
>> python versions, i.e. adopt new syntax and other large changes on one
>> version (for example odd versions) and polish everything up in the next
>> (even versions).
>>
>
> [...], so I would advise we stick to the discussion on a moratorium [...]
>
>
Btw the upcoming Language Summit may be a good opportunity for such
discussion.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Slow down...

[Ivan Levkivskyi]

On 7 May 2018 at 03:25, Nick Coghlan  wrote:

> On 7 May 2018 at 11:30, Dan Stromberg  wrote:
>
>> I'd very much like a live in a world where Jython and IronPython and
>> MicroPython and Cython and Pyjamas can all catch up and implement
>> Python 3.7, 3.8, and so forth.
>>
>
> I'm inclined to agree that a Python 3.8 PEP in the spirit of the PEP 3003
> language moratorium could be a very good idea. Between matrix
> multiplication, enhanced tuple unpacking, native coroutines, f-strings, and
> type hinting for variable assignments, we've had quite a bit of syntactic
> churn in the past few releases, and the rest of the ecosystem really hasn't
> caught up on it all yet (and that's not just other implementations - it's
> training material, online courses, etc, etc).
>
> If we're going to take such a step, now's also the time to do it, since
> 3.8 feature development is only just getting under way, and if we did
> decide to repeat the language moratorium, we could co-announce it with the
> Python 3.7 release.
>
>
These are all god points. I think it will be a good idea to take a little
pause with syntactic additions and other "cognitively loaded" changes. On
the other hand, I think it is fine to work on performance improvements
(start-up time, import system etc.), internal APIs (like simplifying
start-up sequence and maybe even C API), and polishing corner
cases/simplifying existing constructs (like scoping in comprehensions that
many people find confusing).

IOW, I think the PEP should describe precisely what is OK, and what is not
OK during the moratorium.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Ivan Levkivskyi]

On 24 April 2018 at 08:12, Greg Ewing  wrote:

> Chris Jerdonek wrote:
>
> if (diff := x - x_base) and (g := gcd(diff, n)) > 1:

>>>
> "if diff, which we let equal x - x_base, and g, which ..." or
>> "if diff, which we set equal to x - x_base, and g, which " or
>> "if diff, which we define to be x - x_base, and g, which " or
>> "if diff, which we define as x - x_base, and g, which ." etc.
>>
>
> How about "being" as a keyword:
>
>   if (diff being x - x_base) and (g being gcd(diff, n)) > 1:
> return g
>
> An advantage is that you're not likely to be tempted to write
>
>diff being x - x_base
>
> on its own as a statement.
>
>
I like this term, but I don't like having more reserved words. Or PEP can
make it an official way to read :=

(z := x + y) is _called_ a binding expression.
(z := x + y) _reads_ as "z being x + y"

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 561 implemented and minor clarification

[Ivan Levkivskyi]

On 12 April 2018 at 09:59, Ethan Smith  wrote:

> Hello,
>
> I've updated PEP 561 to clarify that any installed stub package should
> supersede an installed inline package. In other words if there is:
>
> /global/site-packages/pkg/
> /user/site-packages/pkg-stubs/
>
> Even if pkg in the global site packages is found first and marks that it
> supports types, the stub package should supersede it.
>
> I also point to mypy's docs on its implementation of the PEP (which can be
> read about here: https://mypy.readthedocs.io/en/latest/installed_packages.
> html). The implementation has been merged into master and will be
> available in the 0.590 release.
>
>
This clarification totally makes sense for me. I could easily imagine a
scenario where are third party provides more advanced/precise/detailed
types for a package that already supports typing. Also thanks for writing
the implementation, hopefully, this PEP will be accepted soon and it will
solve one of the major problems in typing ecosystem.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Ivan Levkivskyi]

Replying also to the list.

On 22 April 2018 at 09:14, Ivan Levkivskyi <levkivs...@gmail.com> wrote:

> On 20 April 2018 at 21:59, Guido van Rossum <gu...@python.org> wrote:
>
>> Does the PEP currently propose to *allow* that horrible example? I
>> thought Tim Peters successfully pleaded to *only* allow a single "NAME :=
>> ". You don't have to implement this restriction -- we know it's
>> possible to implement, and if specifying this alone were to pull enough
>> people from -1 to +0 there's a lot of hope!
>>
>>
> * FWIW I an -1 on anything but a simple name.
>
> * Also Tim proposed a good idea to call these "binding expressions".
> Because in contrasts the different purposes. Binding expressions would be
> probably typically used to (temporarily) name an expression, while
> assignment statements are actually creating "variables" -- long living
> names intended to be accessed externally to a class/module. The latter
> access can be programmed to trigger arbitrary complex code (see properties,
> __getattr__/__setattr__, etc).
>
> * Re implementing restrictions: there is a CST -> AST step that will allow
> to easily prohibit unwanted forms (FWIW this is how unpacking an chaining
> is prohibited for annotated assignments).
>
> * Re using plain "=": Although I am still using this in C quite often, I
> was bitten badly by this several times when I was younger, I don't want a
> similar experience when _learning_ Python.
>
> Modulo these points I would be +0 on the PEP.
>
> --
> Ivan
>
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should the dataclass frozen property apply to subclasses?

[Ivan Levkivskyi]

On 22 February 2018 at 10:55, Eric V. Smith  wrote:

> On 2/22/2018 1:56 AM, Raymond Hettinger wrote:
>
>> Other immutable classes in Python don't behave the same way:
>
>
>>  >>> class T(tuple):
>>  pass
>>
>>  >>> t = T([10, 20, 30])
>>  >>> t.cached = True
>>
>>  >>> class F(frozenset):
>>  pass
>>
>>  >>> f = F([10, 20, 30])
>>  >>> f.cached = True
>>
>>  >>> class B(bytes):
>>  pass
>>
>>  >>> b = B()
>>  >>> b.cached = True
>>
>
> The only way I can think of emulating this is checking in __setattr__ to
> see if the field name is a field of the frozen class, and only raising an
> error in that case.
>

How about checking that the type of self is the type where decorator was
applied? For example (pseudocode):

def dataclass(cls, ...):
def _set_attr(self, attr, value):
if type(self) is not cls:
 use super()
else:
raise AttributeError
cls.__setattr__ = _set_attr

It can be also more sophisticated, for example raising for all fields on
class where frozen=True was used, while only on frozen fields for
subclasses.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How is the GitHub workflow working for people?

[Ivan Levkivskyi]

> I'm personally hugely happy with the new workflow -- my only regret is
that we're not using GitHub for issue tracking yet.

I have the same feelings. I like the GitHub issue tracker, and it would be
great to migrate there (if possible).

--
Ivan



On 21 February 2018 at 21:22, Guido van Rossum  wrote:

> On Wed, Feb 21, 2018 at 9:53 AM, Brett Cannon  wrote:
>
>>
>>
>> On Wed, 21 Feb 2018 at 09:30 Yury Selivanov 
>> wrote:
>>
>>> FWIW I'm extremely happy with the current workflow. The recent
>>> improvements to @miss-islington (kudos to Mariatta!) allowing her to
>>> auto-backport PRs and commit them is a big time saver.
>>>
>>> I can only suggest a couple improvements:
>>>
>>> 1. Make our bots check the code style—fully enforce PEP 8, lint the
>>> code, and detect trailing whitespace on all lines that a PR modifies.
>>>
>>
>> Guido said "no" to this from the outset.
>>
>
> I'm willing to reconsider if there's a good enough tool. Ditto for C code
> (or do we already do it for C?).
>
> FWIW I'm personally hugely happy with the new workflow -- my only regret
> is that we're not using GitHub for issue tracking yet.
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> levkivskyi%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dataclasses and correct hashability

[Ivan Levkivskyi]

Just wanted to add my 5 cents here. I am a bit surprised how people are
scared by adding `__hash__` to mutable classes.
>From my experience it is quite normal, I was always thinking about `hash()`
as hashing a _value_,
and never as hashing _identity_, if I would need the latter, there is a
different function for this, `id()`.
Moreover, I often did this in situations where dataclasses would be useful:
a class with several fields,
necessary dunders, and few other methods (record-like classes).
My motivation was mostly speed-up by memorization.
To be fair my use cases were mostly about some tree-like strictures, but
this is probably a coincidence.

FWIW it is not a super-safe practice, but neither super-dangerous.

--
Ivan



On 5 February 2018 at 22:56, Nick Coghlan  wrote:

> On 6 February 2018 at 03:47, Guido van Rossum  wrote:
> > If there's going to be an API for it, it should be in the class, not
> > something that mutates the class afterwards.
>
> Something I realised after posting the __class__ setting idea is that
> you can actually use a comparable trick to inject an unsafe hash from
> the frozen version into the mutable version:
>
> >>> from dataclasses import dataclass
> >>> @dataclass
> ... class Example:
> ... a: int
> ... b: int
> ...
> >>> c = Example(1, 2)
> >>> hash(c)
> Traceback (most recent call last):
>  File "", line 1, in 
> TypeError: unhashable type: 'Example'
>
> >>> @dataclass(frozen=True)
> ... class LockedExample(Example):
> ... pass
> ...
> >>> Example.__hash__ = LockedExample.__hash__
> >>> hash(c)
> 3713081631934410656
>
> So "unsafe_hash=True" would just be a shorthand spelling of that which
> skips creating the full frozen version of the class (and with the
> explicit parameter, we can better document the risks of making
> something hashable without also freezing it post-creation).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> levkivskyi%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why wont duplicate methods be flagged as error (syntax or anything suitable error)

[Ivan Levkivskyi]

On 14 January 2018 at 08:20, Chris Angelico  wrote:

> On Sun, Jan 14, 2018 at 7:10 PM, joannah nanjekye
>  wrote:
> > Hello,
> >
> > Apparently when you implement two methods with the same name:
> >
> > def sub(x, y):
> >  print(x -y)
> >
> > def sub(x, y):
> >  print(x -y)
> >
> > Even with type hints.
> >
> > def sub(x: int, y:int) -> int:
> > return x - y
> >
> > def sub(x: float, y:float) -> float:
> > return 8
> >
> > If you are from another background, you will expect the syntax with type
> > hints to act as though method overloading but instead last
> implementation is
> > always called. If this is the required behavior,then just flag any
> duplicate
> > method implementations as syntax errors.
> >
> > Is this sort of method name duplication important in any cases?
> >
> > Not aimed at criticism, just to understand.
>
> This is not an error in the language for the same reason that any
> other assignment isn't an error:
>
> x = 5
> x = 6
>
> But you will find that a number of linters will flag this as a
> warning. You can configure your editor to constantly run a linter and
> show you when something's wrong.
>

For example mypy (and probably also PyCharm) warn about
variable/function/class re-definition.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Ivan Levkivskyi]

I like the Guido's proposal, i.e.

if '__repr__' not in cls.__dict__:
...  # generate the method

etc. I didn't find an issue to track this. Maybe we should open one?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Documenting types outside of typing

[Ivan Levkivskyi]

FWIW the same problem was discussed a year ago when documenting typing. At
that time the discussion was not conclusive,
so that some types use class:: directive while other use data:: directive.
At that time Guido was in favour of data:: and now in view of
PEP 560 many types in typing will stop being class objects, and will be
just (compact) objects. Therefore, my understanding is that
all special forms like Union, Any, ClassVar, etc. will use data:: in the
docs.

Concerning the question whether it makes to document types, I think it
makes sense if it is a publicly available type (or type alias)
that will be useful to annotate user code.

--
Ivan



On 27 December 2017 at 17:41, Barry Warsaw  wrote:

> In his review of PR#4911, Antoine points to the documentation of two type
> definitions in importlib.resources, Package and Resource.
>
> https://github.com/python/cpython/pull/4911/files#diff-
> 2a479c407f7177f3d7cb876f244e47bcR804
>
> One question is what markup to use for type definitions.  I’m using
> class:: because that’s what’s used in typing and there doesn’t seem to be
> any better alternative.
>
> More to the point, Antoine questions whether these two types should be
> documented at all:
>
> https://github.com/python/cpython/pull/4911#discussion_r158801065
>
> "What I mean is that a class is supposed to specify concrete behaviour,
> but being a type, Package doesn't have any methods or attributes of its
> own. So I don't see the point of mentioning it in the docs.”
>
> I suggest that they are worth documenting because they help to organize
> the discussion about what API is expected from the arguments to the
> functions, without having to duplicate that information in every function
> description.  I also think that since you’ll see those types in the code,
> they are worth documenting.  I don’t think you *lose* anything by including
> their documentation.
>
> But Antoine makes a good point that we probably don’t have a lot of
> precedence here, so suggests we discuss it on python-dev to come up with
> some useful conventions.  I haven’t kept up on the dataclasses discussion,
> but given that types are important in that API too, have the same issues
> come up there and if so, how are they being handled?
>
> Cheers,
> -Barry
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> levkivskyi%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Ivan Levkivskyi]

On 22 December 2017 at 20:55, Brett Cannon  wrote:

>
>
> On Fri, Dec 22, 2017, 11:38 Chris Barker,  wrote:
>
>> On Fri, Dec 22, 2017 at 8:49 AM, Brett Cannon  wrote:
>>
>>> I  think it's worth reminding people that if they don't like the fact
 dataclasses (ab)use type hints for their succinct syntax that you can
 always use attrs instead to avoid type hints.

>>>
>> sure -- but this doesn't really address the issue, the whole reason this
>> is even a discussion is because dataclasses is going into the standard
>> library. Third party packages can do whatever they want, of course.
>>
>> And the concern is that people (in particular newbies) will get confused
>> / the wrong impression / other-negative-response by the (semi) use of
>> typing in a standard library module.
>>
>
> I'm still not worried. Type hints are part of the syntax and so are no
> worse off than async/await and asyncio IMO.
>
>
>>
>>> As for those who feel dataclasses will force them to teach type hints
>>> and they simply don't want to, maybe we could help land protocols
>>>
>>
>> Could you please clarify what this is about ???
>>
>
> There's a PEP by Ivan (on my phone else I would look up the number).
>
>
If anyone is curious this is PEP 544. It is actually already fully
supported by mypy, so that one can play with it
(you will need to also install typing_extensions, where Protocol class
lives until the PEP is approved).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting functools.singledispatch with classes.

[Ivan Levkivskyi]

On 26 December 2017 at 01:41, Nick Coghlan  wrote:

> On 25 December 2017 at 12:32, Ethan Smith  wrote:
> > So at the moment, I don't think it is possible to implement
> singledispatch
> > on classmethod or staticmethod decorated functions.
>
> I've posted this to the PR, but adding it here as well: I think this
> is a situation very similar to the case with functools.partialmethod,
> where you're going to need to write a separate
> functools.singledispatchmethod class that's aware of the descriptor
> protocol, rather than trying to add the functionality directly to
> functools.singledispatch.
>

I agree with Nick here. Adding a separate decorator looks like the right
approach,
especially taking into account the precedent of @partialmethod.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Ivan Levkivskyi]

On 21 December 2017 at 11:22, Terry Reedy  wrote:

> On 12/21/2017 4:22 AM, Eric V. Smith wrote:
>
>> On 12/21/2017 1:46 AM, Chris Barker wrote:
>>
>
> I suggest that it be clear in the docs, and ideally in the PEP, that the
>>> dataclass decorator is using the *annotation" syntax, and that the the only
>>> relevant part it uses is that an annotation exists, but the value of the
>>> annotation is essentially (completely?) ignored.
>>>
>>
>> I think the PEP is very clear about this: "The dataclass decorator
>> examines the class to find fields. A field is defined as any variable
>> identified in __annotations__. That is, a variable that has a type
>> annotation. With two exceptions described below, none of the Data Class
>> machinery examines the type specified in the annotation."
>>
>
> This seems clear enough.  It could come after describing what a dataclass
> *is*.
>
> I agree the docs should also be clear about this.
>>
>
>
> So we should have examples like:
>>>
>>> @dataclass
>>> class C:
>>>  a: ...  # field with no default
>>>  b: ... = 0 # filed with a default value
>>>
>>> Then maybe:
>>>
>>> @dataclass
>>> class C:
>>>  a: "the a parameter" # field with no default
>>>  b: "another, different parameter" = 0.0 # field with a default
>>>
>>> Then the docs can go to say that if the user wants to specify a type for
>>> use with a static type checking pre-processor, they can do it like so:
>>>
>>> @dataclass
>>> class C:
>>>  a: int # integer field with no default
>>>  b: float = 0.0 # float field with a default
>>>
>>> And the types will be recognized by type checkers such as mypy.
>>>
>>> And I think the non-typed examples should go first in the docs.
>>>
>>
> Module some bike-shedding, the above seems pretty good to me.
>

For me, the three options for "don't care" have a bit different meaning:

* typing.Any: this class is supposed to be used with static type checkers,
but this field is too dynamic
* ... (ellipsis): this class may or may not be used with static type
checkers, use the inferred type in the latter case
* "field docstring": this class should not be used with static type checkers

Assuming this, the second option would be the "real" "don't care". If this
makes sense,
then we can go the way proposed in
https://github.com/python/typing/issues/276 and make ellipsis semantics
"official" in PEP 484.
(pending Guido's approval)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 -- Context Variables

[Ivan Levkivskyi]

On 13 December 2017 at 22:35, Yury Selivanov 
wrote:

> [..]
> >> A new standard library module ``contextvars`` is added
> >
> > Why not add this to contextlib instead of adding a new module?  IIRC
> > this was discussed relative to PEP 550, but I don't remember the
> > reason.  Regardless, it would be worth mentioning somewhere in the
> > PEP.
> >
>
> The mechanism is generic and isn't directly related to context
> managers.  Context managers can (and in many cases should) use the new
> APIs to store global state, but the contextvars APIs do not depend on
> context managers or require them.
>
>
This was the main point of confusion for me when reading the PEP.
Concept of TLS is independent of context managers, but using word "context"
everywhere leads to doubts like "Am I getting everything right?" I think
just adding the
two quoted sentences will clarify the intent.

Otherwise the PEP is easy to read, the proposed API looks simple, and this
definitely will be a useful feature.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Ivan Levkivskyi]

On 18 December 2017 at 20:38, Nick Coghlan  wrote:

>
> On 19 Dec. 2017 7:00 am, "Chris Barker"  wrote:
>
>
> Are there other options??
>
> plain old:
>
> @dataclass
> class C:
> a = 1
> b = 1.0
>
> would work, though then there would be no way to express fields without
> defaults:
>
>
> The PEP already supports using "a = field(); b = field()" (etc) to declare
> untyped fields without a default value.
>
>
The PEP is not 100% clear not this, but it is currently not the case and
this may be intentional (one obvious way to do it),
I just tried and this does not work:

@dataclass
class C:
x = field()

generates `__init__` etc. with no arguments. I think however that it is
better to generate an error than silently ignore it.
(Or if this a bug in the implementation, it should be just fixed.)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Ivan Levkivskyi]

@David
What you propose as `Infer` annotation was proposed some time ago (not only
for dataclasses, there are other use cases).
The discussion is here https://github.com/python/typing/issues/276

@Chris
People are still allowed not to use dataclasses if they really don't like
type hints :-)
Seriously however, annotations are just syntax. In this sense PEP 526 is
more like PEP 3107,
and less like PEP 484. People are still free to write:

@dataclass
class C:
x: "first coordinate"
y: "second coordinate"
plus: "I don't like types"

or

@dataclass
class C:
x: ...
y: ...

I don't see so big difference between hypothesis (testing lib) using
annotations for their purposes
from the situation with dataclasses. It is true that the syntax was chosen
to simplify
support in static type checkers (partially because users were often asking
for such feature),
but not more than this. If you don't use type checkers, there is no problem
in using one of the above forms.

If you have ideas about how to improve the dataclass docs, this can be
discussed in the issue https://bugs.python.org/issue32216

> ... the type will in fact be completely ignored by the implementation.
> Newbies are going to be confused by this -- they really are.

This is not different from

def f(x: int):
pass

f("What")  # OK

that exists starting from Python 3.0. Although I agree this is confusing,
the way forward could be just
explaining this better in the docs.

If you want my personal opinion about the current situation about type
hints _in general_, then I can say that
I have seen many cases where people use type hints where they are not needed
(for example in 10 line scripts or in highly polymorphic functions), so I
agree that some community
style guidance (like PEP 8) may be helpful. I had started such project an
the end of last year
(it was called pep-555, but I didn't have time to work on this and this
number is already taken).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make __class_getitem__ a class method

[Ivan Levkivskyi]

On 15 December 2017 at 19:35, Serhiy Storchaka  wrote:

> 15.12.17 20:02, Yury Selivanov пише:
>
>> But nobody decorates __new__ with a @staticmethod.  And making
>> __class_getitem__ a @classmethod will only confuse users -- that's all
>> I'm saying.
>>
>> So I'm +1 to keep the things exactly as they are now.  It would be
>> great do document that in order to implement __class_getitem__ in C
>> one should add it as METH_STATIC.  I also think we should merge your
>> PR that tests that it works the way it's expected.
>>
>
> In this case I suggest to make __class_getitem__ an automatic class method
> like __init_subclass__.
>
> The number of special cases bothers me.
>

I just want to clarify what is proposed. As I understand:

* From the point of view of a pure Python class there will be no difference
with the current behaviour, one just writes

  class C:
  def __class_getitem__(cls, item):
  ...

*  In `type_new`, `__class_getitem__` will be wrapped in classmethod
*  From the point of view of C extensions one will use METH_CLASS and no
tuple unpacking

If this is true that this looks reasonable. If no-one is against, then I
can make a PR.
The only downside to this that I see is that `type.__new__` will be
slightly slower.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] __init_subclass__ is a class method (Was: Make __class_getitem__ a class method)

[Ivan Levkivskyi]

On 15 December 2017 at 18:40, Serhiy Storchaka <storch...@gmail.com> wrote:

> 15.12.17 19:04, Ivan Levkivskyi пише:
>
>> Good point! Pure Python will be the primary use case and we have another
>> precedent
>> for "automatic" class method: __init_subclass__ (it does not need to be
>> decorated).
>>
>
> __init_subclass__ is very different beast, and parallels with it can be
> confusing. It is automatically decorated with classmethod if it is a
> regular function implemented in C. The following two examples are totally
> equivalent:
>
> class A:
> def __init_subclass__(cls): pass
>
> class B:
> @classmethod
> def __init_subclass__(cls): pass
>
> help(A) shows __init_subclass__() as a class method (in 3.7).
>
> But if you implement the class in C you need to make __init_subclass__ a
> class method.
>
> I think __init_subclass__ should be documented as a class method since it
> is a class method.
>

Thank you for clarification!
Actually documentation
https://docs.python.org/3.6/reference/datamodel.html#customizing-class-creation
already says `classmethod object.__init_subclass__(cls)`
I am not an expert in this, so I am not sure if the docs can be improved
here (maybe we can add how this works with C API?)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make __class_getitem__ a class method

[Ivan Levkivskyi]

On 15 December 2017 at 17:47, Yury Selivanov 
wrote:

> Shouldn't we optimize the usability for pure-Python first, and then for C
> API?
>
> Right now we have the '__new__' magic method, which isn't a
> @classmethod.  Making '__class_getitem__' a @classmethod will confuse
> regular Python users.  For example:
>
>class Foo:
>   def __new__(cls, ...): pass
>
>   @classmethod
>   def __class_getitem__(cls, item): pass
>
> To me it makes sense that type methods that are supposed to be called
> on type by the Python interpreter don't need the classmethod
> decorator.
>

Good point! Pure Python will be the primary use case and we have another
precedent
for "automatic" class method: __init_subclass__ (it does not need to be
decorated).


> METH_STATIC is a public working API, and in my opinion it's totally
> fine if we use it. It's not even hard to use it, it's just *mildly*
> inconvenient at most.
>

OK, then documenting this "recipe" (METH_STATIC plus tuple unpacking)
should be sufficient.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make __class_getitem__ a class method

[Ivan Levkivskyi]

I like this idea. I have few suggestions for the test cases you added, will
add them a bit later in the PR.

--
Ivan


On 15 December 2017 at 16:00, Serhiy Storchaka  wrote:

> The class itself always is passed as the first argument to
> __class_getitem__():
>
> cls.__class_getitem__(cls, item)
>
> I propose to make __class_getitem__ a class method. This will make simpler
> implementing it in C. Currently it should be declared with flags
> METH_VARARGS|METH_STATIC and implementing as
>
> static PyObject *
> generic_class_getitem(PyObject *Py_UNUSED(self), PyObject *args)
> {
> PyObject *type, *item;
> if (!PyArg_UnpackTuple(args, "__class_getitem__", 2, 2, , ))
> {
> return NULL;
> }
> ...
> }
>
> Note an unused parameter and the need of manual unpacking arguments.
>
> If use it as a class method it should be declared with flags
> METH_O|METH_CLASS and implemented as
>
> static PyObject *
> generic_class_getitem(PyObject *type, PyObject *item)
> {
> ...
> }
>
> See https://github.com/python/cpython/pull/4883 for sample.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/levkivsky
> i%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 560 -- Core support for typing module and generic types

[Ivan Levkivskyi]

On 15 December 2017 at 15:55, Yury Selivanov 
wrote:

> > I don't see any problems with implementing this on types defined in C.
> This isn't harder than implementing __sizeof__ or pickling support, and
> NumPy classes already have implemented both. Maybe Yury forgot about
> METH_STATIC and METH_CLASS?
>
> I just tested __class_getitem__ defined via METH_STATIC and it works.
> This means we don't need to add slots.  Thanks for the hint, Serhiy!
>
> Ivan, this might be worth mentioning in the PEP or in the docs.


I think it should be added to the PEP. Somehow I didn't think about C
extensions, but now I see that it is important, also recently there
appeared a lot of interest in better support of static typing for NumPy
arrays and numeric stack.
I will make a PR a bit later.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Last call for PEP approvals before the holidays

[Ivan Levkivskyi]

There is also PEP 544 (Structural subtyping, a.k.a. static duck typing),
but I think we discussed off-list that it is also not time sensitive,
given the (limited) provisional status of typing module.

(Also mypy already supports it, so the question is mainly when this support
is official after polishing few remaining issues.)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Ivan Levkivskyi]

On 10 December 2017 at 22:24, Raymond Hettinger  wrote:

> Without typing (only the first currently works):
>
> Point = namedtuple('Point', ['x', 'y', 'z'])  # underlying
> store is a tuple
> Point = make_dataclass('Point', ['x', 'y', 'z'])  # underlying
> store is an instance dict
>
>
Hm, I think this is a bug in implementation. The second form should also
work.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issues with PEP 526 Variable Notation at the class level

[Ivan Levkivskyi]

On 8 December 2017 at 19:28, Raymond Hettinger 
wrote:

>
> I'm hoping the typing experts will chime in here.  The question is
> straight-forward.  Where should we look for the signature and docstring for
> constructing instances?  Should they be attached to the class, to
> __init__(), or to __new__() when it used.
>
> It would be nice to have an official position on that before, it gets set
> in stone through arbitrary choices made by pycharm, pydoc, mypy,
> typing.NamedTuple, and dataclasses.dataclass.
>
>
Here are some thoughts about this:

1. Instance variables are given very little attention in pydoc. Consider
this example:

>>> class C:
... x: int = 1
... def meth(self, y: int) -> None:
... ...
>>> help(C)

Help on class C in module __main__:

class C(builtins.object)
 |  Methods defined here:
 |
 |  meth(self, y: int) -> None
 |
 |  --
 |  Data descriptors defined here:
 |
 |  __dict__
 |  dictionary for instance variables (if defined)
 |
 |  __weakref__
 |  list of weak references to the object (if defined)
 |
 |  --
 |  Data and other attributes defined here:
 |
 |  __annotations__ = {'x': }
 |
 |  x = 1

The methods defined are listed first and are nicely formatted, while
variables together with __annotations__ are
left at the very end. I think that a line like

x: int = 1

should appear for every instance variable should appear first, even before
methods, since this is how people write (and read) classes.
See also https://bugs.python.org/issue28519 for another problem with pydoc.

2. pydoc already extracts the signature of class from __init__ and __new__
(giving the preference to later if both are present) including
the type annotations. I think this can be kept as is, but the special
constructs like NamedTuple and dataclass that auto-generate methods should
add annotations to them. For example, there is an issue to add annotations
to __new__ by NamedTuple, see https://bugs.python.org/issue31006
and https://github.com/python/typing/issues/454

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations (Draft 3)

[Ivan Levkivskyi]

Congratulations, Łukasz!

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 560 -- Core support for typing module and generic types

[Ivan Levkivskyi]

Thank you! It looks like we have a bunch of accepted PEPs today.
It is great to see all this! Thanks everyone who participated in
discussions here, on python-ideas and
on typing tracker. Special thanks to Mark who started this discussion.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 562 -- Module __getattr__ and __dir__

[Ivan Levkivskyi]

Thank you Guido!

And thanks everyone for help, discussions, and ideas (in particular Larry
who started this discussion).
I will submit a PR with implementation soon.

--
Ivan


On 4 December 2017 at 17:58, Guido van Rossum  wrote:

> Ivan,
>
> Congrats on your PEP. I believe the outstanding issues are now resolved
> and I am hereby accepting it.
>
> PS. Sorry, Larry, PEP 549 is rejected. But that happened a while ago.
>
> --
> --Guido van Rossum (python.org/~guido)
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557 Data Classes 5th posting

[Ivan Levkivskyi]

Congratulations, Eric! This is a great PEP and I am looking forward to
implement support for it in mypy ;-)

--
Ivan


On 4 December 2017 at 18:17, Guido van Rossum  wrote:

> And with this, I'm accepting PEP 557, Data Classes.
>
> Eric, congrats with your efforts in proposing and implementing this PEP
> and guiding it through the discussion! It's been great to see this idea
> come to fruition. Thanks also to the many people who reviewed drafts or
> implementation code, including the very generous authors and maintainers of
> "attrs", from which this has taken many ideas.
>
> --
> --Guido van Rossum (python.org/~guido)
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 25 November 2017 at 16:57, Guido van Rossum <gu...@python.org> wrote:

> On Sat, Nov 25, 2017 at 6:55 AM, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
>
>> On 25 November 2017 at 04:30, Guido van Rossum <gu...@python.org> wrote:
>>
>>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum <gu...@python.org>
>>> wrote:
>>>
>>>> The more I hear about this topic, the more I think that `await`,
>>>> `yield` and `yield from` should all be banned from occurring in all
>>>> comprehensions and generator expressions. That's not much different from
>>>> disallowing `return` or `break`.
>>>>
>>>
>>> From the responses it seems that I tried to simplify things too far.
>>> Let's say that `await` in comprehensions is fine, as long as that
>>> comprehension is contained in an `async def`. While we *could* save `yield
>>> [from]` in comprehensions, I still see it as mostly a source of confusion,
>>> and the fact that the presence of `yield [from]` *implicitly* makes the
>>> surrounding `def` a generator makes things worse. It just requires too many
>>> mental contortions to figure out what it does.
>>>
>>
>> [...]
>> If the first example will be allowed, then one will be surprised why it
>> can't be rewritten as
>>
>> def pack_two():
>> return [(yield) for _ in range(2)]
>>
>
> And yet Nick's example shows that that is not equivalent!
>
> [...]
>
> In this example each thing that looks syntactically like a list
> comprehension becomes actually a generator expression at at runtime! And so
> does your example, so instead of a list of two items, it returns a
> generator that will produce two values when iterated over.
>
> That's not referential transparency to me, it feels more like a bug in the
> code generator.
>
> I want to ban this because apparently nobody besides Nick knows about this
> behavior (I certainly didn't, and from the above it seems you don't either).
>

This whole thread started as a proposal to fix this bug and to make the two
forms equivalent, so I don't know what you are talking about.

Also as there appeared arguments of authority (thanks Antoine) its time to
stop this discussion for me.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 25 November 2017 at 04:30, Guido van Rossum  wrote:

> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum 
> wrote:
>
>> The more I hear about this topic, the more I think that `await`, `yield`
>> and `yield from` should all be banned from occurring in all comprehensions
>> and generator expressions. That's not much different from disallowing
>> `return` or `break`.
>>
>
> From the responses it seems that I tried to simplify things too far. Let's
> say that `await` in comprehensions is fine, as long as that comprehension
> is contained in an `async def`. While we *could* save `yield [from]` in
> comprehensions, I still see it as mostly a source of confusion, and the
> fact that the presence of `yield [from]` *implicitly* makes the surrounding
> `def` a generator makes things worse. It just requires too many mental
> contortions to figure out what it does.
>

There were some arguments that `await` is like a function call, while
`yield` is like `return`.
TBH, I don't really like these arguments since to me they are to vague.
Continuing this logic one can say that
`return` is just a fancy function call (calling continuation with the
result). To me there is one clear distinction:
`return` and `break` are statements, while `yield`, `yield from`, and
`await` are expressions.

Continuing the topic of the ban, what exactly should be banned? For example
will this still be valid?

def pack_two():
return [(yield), (yield)]  # Just a list display

I don't see how this is controversial. It is clear that `pack_two` is a
generator.
If this is going to be prohibited, then one may be surprised by lack of
referential transparency, since this will be valid:

def pack_two():
first = (yield)
second = (yield)
return [first, second]

If the first example will be allowed, then one will be surprised why it
can't be rewritten as

def pack_two():
return [(yield) for _ in range(2)]

I have found several other examples where it is not clear whether they
should be prohibited with `yield` or not.

I still propose to rule out all of the above from generator expressions,
> because those can escape from the surrounding scope.
>

Here I agree. Also note that the above problem does not apply to generator
expressions since (x, x) and (x for _ in range(2)) are
two very different expressions.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 25 November 2017 at 01:22, Guido van Rossum  wrote:

> The more I hear about this topic, the more I think that `await`, `yield`
> and `yield from` should all be banned from occurring in all comprehensions
> and generator expressions. That's not much different from disallowing
> `return` or `break`.
>
>
IIUC this would essentially mean rejecting PEP 530.
What do you think about banning `await`, `yield` and `yield from` only from
generator expressions?
Comprehensions can be fixed (just make them equivalent to for-loops without
leaks as discussed).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

OK, so my 24 hours are over :-)


On 24 November 2017 at 01:50, Nick Coghlan <ncogh...@gmail.com> wrote:

> On 23 November 2017 at 23:04, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
>
>> I don't see why this particular case qualifies for such a radical measure
>> as an exception to syntactic rules,
>> instead of just fixing it (sorry Nick :-)
>>
>
> I've posted in more detail about this to the issue tracker, but the
> argument here is: because making it behave differently from the way it does
> now while still hiding the loop iteration variable potentially requires
> even more radical revisions to the lexical scoping rules :)
>
If somebody can come up with a clever trick to allow yield inside a
> comprehension to jump levels in a relatively intuitive way, that would
> actually be genuinely cool, but the lexical scoping rules mean it's
> trickier than it sounds.
>

"potentially" is the key word here. The plan is to avoid "more radical
revisions".


> Now that I frame the question that way, though, I'm also remembering that
> we didn't have "yield from" yet when I wrote the current comprehension
> implementation, and given that, it may be as simple as having an explicit
> yield expression in a comprehension imply delegation to a subgenerator.
>

My understanding is that this is exactly how async comprehensions are
currently implemented and why they work as one would naively expect, i.e.
`await` is "bound" to the surrounding async def, not to the implicit scope
async def. So that an async comprehension is just equivalent to a for-loop.
However, although "implicit yield from" solution is simpler than the one
proposed by Serhiy, I still more like the latter one. The former has its
strange cases, for example I mentioned before in this thread:

>>> async def f():
... for i in range(3):
... yield i
...
>>> async def g():
... return [(yield i) async for i in f()]
...
>>> g().send(None)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in g
TypeError: object async_generator can't be used in 'await' expression

My understanding is that the strange error is exactly because of the
implicit `yield from`. With Serhiy's approach this would work.
Another minor bonus of Serhiy's idea is a performance gain: we will not
push an execution frame.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 15:30, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 23 November 2017 at 14:24, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> >> My main concern is that comprehension is not equivalent to a for loop
> >> for a specific reason - the scope issue. Has anyone looked back at the
> >> original discussions to confirm *why* a function was used?
> >>
> >> My recollection:
> >>
> >> >>> i = 1
> >> >>> a = [i for i in (1,2,3)]
> >> >>> print(i)
> >> 1
> >>
> >> Serihy's approach (and your described expansion) would have print(i)
> >> return NameError.
> >
> >
> > Absolutely no, it will still print 1. The internal implementation will
> use
> > unique ids internally (see https://bugs.python.org/issue10544 for
> details).
> >
>
> Ok, cool. My main point still applies though - has anyone confirmed
> why a function scope was considered necessary at the time of the
> original implementation, but it's apparently not now? I'm pretty sure
> it was a deliberate choice, not an accident.


>From what Nick explained on b.p.o. I understand that this is closer to the
"accident" definition.
Also the original issue https://bugs.python.org/issue1660500 doesn't have
any discussion of the implementation _strategy_.
So I tried to dig the mailing list, in the latest Guido's message I have
found
https://mail.python.org/pipermail/python-3000/2006-December/005218.html
he still likes the idea of unique hidden ids (like Serhiy proposes now) and
no function scopes. After that there is Nick's message
https://mail.python.org/pipermail/python-3000/2006-December/005229.html
where he says that he still likes pseudo-scopes more.
Then I lost the track of discussion.

It may well be Nick's intentional decision (and it has its merits) but I am
not sure it was a conscious consensus.
Nick could probably add more. Also I propose to wait and see when Serhiy
will show us his complete implementation.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 15:21, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 23 November 2017 at 13:04, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > Let us forget for a moment about other problems and focus on this one:
> list
> > comprehension is currently not equivalent to a for-loop.
> > There are two options:
> > - Fix this, i.e. make comprehension equivalent to a for-loop even in edge
> > cases (Serhiy seems ready to do this)
> > - Prohibit all cases when they are not equivalent
> >
> > I still prefer option one. But I see your point, option two is also an
> > acceptable fix.
> > Note that there were not so many situations when some code became
> > SyntaxError later.
> > I don't see why this particular case qualifies for such a radical
> measure as
> > an exception to syntactic rules,
> > instead of just fixing it (sorry Nick :-)
>
> My main concern is that comprehension is not equivalent to a for loop
> for a specific reason - the scope issue. Has anyone looked back at the
> original discussions to confirm *why* a function was used?
>
> My recollection:
>
> >>> i = 1
> >>> a = [i for i in (1,2,3)]
> >>> print(i)
> 1
>
> Serihy's approach (and your described expansion) would have print(i)
> return NameError.
>

Absolutely no, it will still print 1. The internal implementation will use
unique ids internally (see https://bugs.python.org/issue10544 for details).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 13:45, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 23 November 2017 at 12:28, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > On 23 November 2017 at 13:11, Paul Moore <p.f.mo...@gmail.com> wrote:
> >>
> >> On 23 November 2017 at 12:01, Ivan Levkivskyi <levkivs...@gmail.com>
> >> wrote:
> >>
> >> > "I don't use it, therefore it is not needed"  is a great argument,
> >> > thanks.
> >> > Lets just forget about two SO questions and dozens people who up-voted
> >> > it.
> >> > Do you use async comprehensions? If not, then we don't need them
> either.
> >>
> >> For those of us trying to keep up with the discussion who don't have
> >> time to chase the various references, and in the interest of keeping
> >> the discussion in one place, can you summarise the real-world use
> >> cases from the SO questions here? (I assume they are real world cases,
> >> and not just theoretical questions)
> [...]
> >
> > My understanding is that none of the case is _pressing_, since they all
> > start with a for-loop, but
> > following this logic comprehensions themselves are not needed.
> Nevertheless
> > people use them because they like it.
> > The problem in all four cases is that they got hard to debug problem,
> since
> > calling `f()` returns a generator,
> > just not the one they would expect.
>
> OK, thanks. I can see why someone would want to do this. However, it
> seems to me that the problem (a hard to debug error) could be solved
> by disallowing yield in comprehensions and generator expressions
> (giving an *easy* to debug error). I don't think the above is a
> compelling argument that we have to support the one-line form. If
> there was a non-trivial body of actual user code that uses the loop
> form, which would be substantially improved by being able to use
> comprehensions, that would be different. To put it another way, the
> example you gave is still artificial. The second link is a real use
> case, but as you say seems to be more a question about "why did this
> not work as I expected" which could be solved with a SyntaxError
> saying "yield expression not allowed in comprehensions".
>

The level of "artificialness" is quite subjective, this is rather matter of
taste (see the tornado example).
Let us forget for a moment about other problems and focus on this one: list
comprehension is currently not equivalent to a for-loop.
There are two options:
- Fix this, i.e. make comprehension equivalent to a for-loop even in edge
cases (Serhiy seems ready to do this)
- Prohibit all cases when they are not equivalent

I still prefer option one. But I see your point, option two is also an
acceptable fix.
Note that there were not so many situations when some code became
SyntaxError later.
I don't see why this particular case qualifies for such a radical measure
as an exception to syntactic rules,
instead of just fixing it (sorry Nick :-)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 13:30, Antoine Pitrou  wrote:

> On Thu, 23 Nov 2017 14:17:32 +0200
> Serhiy Storchaka  wrote:
> >
> > I used the "yield" statement, but I never used the "yield" expressions.
> > And I can't found examples. Could you please present a real-world use
> > case for the "yield" (not "yield from") expression?
>
> Of course I can.  "yield" expressions are important for writing
> Python 2-compatible asynchronous code while avoiding callback hell:
>
> See e.g. http://www.tornadoweb.org/en/stable/gen.html
>
> 


Great, so I open this page and see this code:

results = []
for future in list_of_futures:
results.append(yield future)

Interesting, why don't they use a comprehension for this and instead need
to invent a whole `tornado.gen.multi` function?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 13:11, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 23 November 2017 at 12:01, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
>
> > "I don't use it, therefore it is not needed"  is a great argument,
> thanks.
> > Lets just forget about two SO questions and dozens people who up-voted
> it.
> > Do you use async comprehensions? If not, then we don't need them either.
>
> For those of us trying to keep up with the discussion who don't have
> time to chase the various references, and in the interest of keeping
> the discussion in one place, can you summarise the real-world use
> cases from the SO questions here? (I assume they are real world cases,
> and not just theoretical questions)
>

OK, here are the links:

https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions
https://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin
https://bugs.python.org/issue10544
https://bugs.python.org/issue3267

In all four cases pattern is the same, people were trying to refactor
something like this:

def f():
res = []
for x in y:
r = yield x
res.append(r)
return res

into something like this:

def f():
return [(yield x) for x in y]

My understanding is that none of the case is _pressing_, since they all
start with a for-loop, but
following this logic comprehensions themselves are not needed. Nevertheless
people use them because they like it.
The problem in all four cases is that they got hard to debug problem, since
calling `f()` returns a generator,
just not the one they would expect.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 12:49, Antoine Pitrou <solip...@pitrou.net> wrote:

> On Thu, 23 Nov 2017 12:39:46 +0100
> Ivan Levkivskyi <levkivs...@gmail.com> wrote:
> >
> > Also I think it makes sense to keep discussion in one place, i.e. either
> > here xor at https://bugs.python.org/issue10544
>
> The bug tracker can be used for implementation discussions, but general
> language design decisions (such as whether to allow or not a certain
> construct) should take place on python-dev.
>
> I'm still in favour of deprecating and then disallowing.  Nobody seems
> to have presented a real-world use case that is made significantly
> easier by trying to "fix" the current behaviour (as opposed to
> spelling the loop explicitly).  I do asynchronous programming using
> "yield" every day in may job (because of compatibility requirements
> with Python 2) and I've never had once the need to write a "yield"
> inside a comprehension or generator expression.
>
>
"I don't use it, therefore it is not needed"  is a great argument, thanks.
Lets just forget about two SO questions and dozens people who up-voted it.
Do you use async comprehensions? If not, then we don't need them either.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 10:50, Paul Moore  wrote:

> On 23 November 2017 at 09:14, Steve Holden  wrote:
> > I would urge developers, in their improvements to the language to support
> > asynchronous programming, to bear in mind that this is (currently) a
> > minority use case. Why the rush to set complex semantics in stone?
>
> +1
>
> Also, given that languages like C# have similar async/await
> functionality, I'd be interested to know how they address questions
> like this. If they have a parallel, we should probably follow it. If
> they don't that would be further indication that no-one has much
> experience of the "best answers" yet, and caution is indicated.
>
>
Keeping this open for indefinite time is also not a good option.
Note that the issue about `yield` in comprehensions
https://bugs.python.org/issue10544 is 7 (seven) years old.
I don't say that we should fix it _right now_, but I think it makes sense
to spend some time and finally resolve it.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 12:38, Ivan Levkivskyi <levkivs...@gmail.com> wrote:

> On 23 November 2017 at 11:55, Nick Coghlan <ncogh...@gmail.com> wrote:
>
>> On 23 November 2017 at 18:11, Greg Ewing <greg.ew...@canterbury.ac.nz>
>> wrote:
>>
>>> Ivan Levkivskyi wrote:
>>>
>>>> "People sometimes want to refactor for-loops containing `yield` into a
>>>> comprehension but that doesn't work (particularly because of the hidden
>>>> function scope) - lets make it a SyntaxError"
>>>>
>>>
>>> Personally I'd be fine with removing the implicit function
>>> scope from comprehensions and allowing yield in them, since
>>> the semantics of that are clear.
>>>
>>
>> People keep saying this, but seriously, those semantics aren't clear at
>> all once you actually start trying to implement it.
>>
>>
> If Serhiy will implement his idea (emitting for-loop bytecode inside a
> try-finally), then I see no problems accepting it as a fix.
>

Also I think it makes sense to keep discussion in one place, i.e. either
here xor at https://bugs.python.org/issue10544

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 11:55, Nick Coghlan <ncogh...@gmail.com> wrote:

> On 23 November 2017 at 18:11, Greg Ewing <greg.ew...@canterbury.ac.nz>
> wrote:
>
>> Ivan Levkivskyi wrote:
>>
>>> "People sometimes want to refactor for-loops containing `yield` into a
>>> comprehension but that doesn't work (particularly because of the hidden
>>> function scope) - lets make it a SyntaxError"
>>>
>>
>> Personally I'd be fine with removing the implicit function
>> scope from comprehensions and allowing yield in them, since
>> the semantics of that are clear.
>>
>
> People keep saying this, but seriously, those semantics aren't clear at
> all once you actually start trying to implement it.
>
>
If Serhiy will implement his idea (emitting for-loop bytecode inside a
try-finally), then I see no problems accepting it as a fix.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 09:17, Greg Ewing <greg.ew...@canterbury.ac.nz>
wrote:

> Ivan Levkivskyi wrote:
>
>> "People sometimes want to refactor for-loops containing `yield` into a
>> comprehension
>>
>
> By the way, do we have any real-life examples of people wanting to
> do this? It might help us decide what the semantics should be.
>
>
Yes, there are two SO questions in two first posts here, also there are
some b.p.o. issues. It looks like in all case people expect:

def f():
return [(yield i) for i in range(3)]

to be roughly equivalent to

def f():
res = []
for i in range(3):
r = yield i
res.append(r)
return res

See Serhiy's original post for more detailed proposed semantic equivalence.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 09:11, Greg Ewing <greg.ew...@canterbury.ac.nz>
wrote:

> Ivan Levkivskyi wrote:
>
>> "People sometimes want to refactor for-loops containing `yield` into a
>> comprehension but that doesn't work (particularly because of the hidden
>> function scope) - lets make it a SyntaxError"
>>
>
> Personally I'd be fine with removing the implicit function
> scope from comprehensions and allowing yield in them, since
> the semantics of that are clear.
>
> But I don't see a way to do anything equivalent with
> generator expressions. Since the current effect of
> yield in a generator expression is pretty useless,
> it seems best just to disallow it.
>
> That means a list comprehension won't be equivalent
> to list(generator_expression) in all cases, but I
> don't think there's any great need for it to be.
>

I am also fine with this. Generator expressions are indeed less clear.
Also different people have different mental model about them (especially re
implicit scope and equivalence to comprehensions).

On the contrary, vast majority agrees that comprehensions are just
for-loops without leaking variables.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 09:15, Greg Ewing <greg.ew...@canterbury.ac.nz>
wrote:

> Ivan Levkivskyi wrote:
>
>> On 23 November 2017 at 05:44, Greg Ewing <greg.ew...@canterbury.ac.nz
>> <mailto:greg.ew...@canterbury.ac.nz>> wrote:
>>
>>def g():
>>   return ((yield i) for i in range(10))
>>
>>
>> I think this code should be just equivalent to this code
>>
>> def g():
>> temp = [(yield i) for i in range(10)]
>> return (v for v in temp)
>>
>
> But then you get a non-lazy iterable, which defeats the
> purpose of using a generator expression -- you might as
> well have used a comprehension to begin with.
>
>
This could be just a semantic equivalence (mental model), not how it should
be internally implemented.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 05:44, Greg Ewing <greg.ew...@canterbury.ac.nz>
wrote:

> Ivan Levkivskyi wrote:
>
>> The key idea is that neither comprehensions nor generator expressions
>> should create a function scope surrounding the `expr`
>>
>
> I don't see how you can avoid an implicit function scope in
> the case of a generator expression, though. And I can't see
> how to make yield in a generator expression do anything
> sensible.
>
> Consider this:
>
>def g():
>   return ((yield i) for i in range(10))
>
> Presumably the yield should turn g into a generator, but...
> then what? My brain is hurting trying to figure out what
> it should do.
>
>
I think this code should be just equivalent to this code

def g():
temp = [(yield i) for i in range(10)]
return (v for v in temp)

Semantics of the comprehension should be clear here (just an equivalent
for-loop without name leaking)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 01:00, Yury Selivanov <yselivanov...@gmail.com>
wrote:

> On Wed, Nov 22, 2017 at 6:46 PM, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> [..]
> > Just found another example of intuitive behaviour:
> >
> >>>> async def f():
> > ... for i in range(3):
> > ... yield i
> > ...
> >>>> async def g():
> > ... return [(yield i) async for i in f()]
> > ...
> >>>> g().send(None)
> > Traceback (most recent call last):
> >   File "", line 1, in 
> >   File "", line 2, in g
> > TypeError: object async_generator can't be used in 'await' expression
> >
> > of course it is obvious for anyone who writes async code, but anyway an
> > interesting example.
>
> I wouldn't say that it's obvious to anyone...
>
> I think this thread has started to discuss the use of 'yield'
> expression in comprehensions, and the outcome of the discussion is
> that everyone thinks that we should deprecate that syntax in 3.7,
> remove in 3.8.  Let's start with that? :)


I am not sure everyone agrees with this. My main obstacle is following,
consider motivation for the `await` part of PEP 530
which is in my understanding is roughly like this:

"People sometimes want to refactor for-loops containing `await` into a
comprehension but that doesn't work (particularly because of the hidden
function scope) - lets fix this"

I don't see how this compare to:

"People sometimes want to refactor for-loops containing `yield` into a
comprehension but that doesn't work (particularly because of the hidden
function scope) - lets make it a SyntaxError"

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 21:07, Yury Selivanov <yselivanov...@gmail.com>
wrote:

> On Wed, Nov 22, 2017 at 2:37 PM, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > On 22 November 2017 at 20:33, Guido van Rossum <gu...@python.org> wrote:
> >>
> >> On Wed, Nov 22, 2017 at 11:12 AM, Ivan Levkivskyi <levkivs...@gmail.com
> >
> >> wrote:
> >>>
> >>> On 22 November 2017 at 20:05, Guido van Rossum <gu...@python.org>
> wrote:
> >>>>
> >>>> On Wed, Nov 22, 2017 at 10:54 AM, Jelle Zijlstra
> >>>> <jelle.zijls...@gmail.com> wrote
> >>>>>
> >>>>> 2017-11-22 9:58 GMT-08:00 Guido van Rossum <gu...@python.org>:
> >>>>>
> >>>>> (OTOH, await in the same position must keep working since it's not
> >>>>> broken and not unintuitive either.)
> >>>>
> >>>>
> >>>
> >>>
> >>> This is very questionable IMO.
> >>> So do you think that [await x for y in z] and list(await x for y in z)
>
> Comprehensions are declarative, and that's why [], and {} work with
> async/await.  When you're using parens () you *explicitly* tell Python
> compiler that you want a generator expression.
>
> And the distinction between comprehensions and generator expressions
> also exists for synchronous code:
>
>x = [a for a in range(10)]
>x[0]
>
> and
>
>x = (a for a in range(10))
>x[0]  # TypeError
>
> Is the above "intuitive" for all Python users?  Probably not.  Write
> it once, get your TypeError, read the error message and you understand
> what's going on here.
>
> Is the difference between "[await x for y in z ]" and "list(await x
> for y in z)" intuitive for all Python users?  Again, probably not.
> But for those who write async code it is.
>

Just found another example of intuitive behaviour:

>>> async def f():
... for i in range(3):
... yield i
...
>>> async def g():
... return [(yield i) async for i in f()]
...
>>> g().send(None)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in g
TypeError: object async_generator can't be used in 'await' expression

of course it is obvious for anyone who writes async code, but anyway an
interesting example.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

22 Лис 2017 22:33 "Ethan Furman" <et...@stoneleaf.us> пише:

On 11/22/2017 01:25 PM, Ivan Levkivskyi wrote:

> 22 Лис 2017 22:19 "Ethan Furman" пише:
>

Whether it's inside or outside a function should be irrelevant -- a
>> comprehension / generator expression should have
>> no influence on the type of the resulting function (and at least
>> synchronous comprehensions / generator expressions
>> should be able to live outside of a function).
>>
>> def fun_gen():
>>   return list((yield i) for i in range(3))
>>
>> The return says it's returning a list, so that's what it should be
>> returning
>>
>
> def f():
>  yield 1
>  return list()
>
> Here return also says it should return a list, so this is not an argument.
>

Right, the argument is that calling the `list` constructor should return a
list -- not a database proxy, not a web page, and not a generator.


Then you didn't read my example carefully, since the whole point is that
the list constructor there returns a list.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

22 Лис 2017 22:19 "Ethan Furman" <et...@stoneleaf.us> пише:

On 11/22/2017 11:01 AM, Ivan Levkivskyi wrote:

I think how to formulate these rules more "precisely" so that they will be
> all consistent even if there is a
> `yield` inside.
> The key idea is that neither comprehensions nor generator expressions
> should create a function scope surrounding the
> `expr` in (expr for ind in iterable) and [expr for ind in iterable].
> (this still can be some hidden implementation detail)
>
> So as Serhiy proposed in one of his first posts any comprehensions and
> generator expressions with `yield` are not valid
> outside functions.
> If such comprehension or generator expression is inside a function, then
> it makes it a generator, and all the `yield`s
>
> are yielded from that generator, for example:
>

Whether it's inside or outside a function should be irrelevant -- a
comprehension / generator expression should have no influence on the type
of the resulting function (and at least synchronous comprehensions /
generator expressions should be able to live outside of a function).


def fun_gen():
>  return list((yield i) for i in range(3))
>

The return says it's returning a list, so that's what it should be returning


def f():
yield 1
return list()

Here return also says it should return a list, so this is not an argument.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 20:33, Guido van Rossum <gu...@python.org> wrote:

> On Wed, Nov 22, 2017 at 11:12 AM, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
>
>> On 22 November 2017 at 20:05, Guido van Rossum <gu...@python.org> wrote:
>>
>>> On Wed, Nov 22, 2017 at 10:54 AM, Jelle Zijlstra <
>>> jelle.zijls...@gmail.com> wrote
>>>
>>>> 2017-11-22 9:58 GMT-08:00 Guido van Rossum <gu...@python.org>:
>>>>
>>> (OTOH, await in the same position must keep working since it's not
>>>> broken and not unintuitive either.)
>>>>
>>>
>>>
>>
>> This is very questionable IMO.
>> So do you think that [await x for y in z] and list(await x for y in z)
>> being not equivalent is intuitive?
>>
>
> I see, that's why this is such a long thread. :-(
>
> But are they different? I can't find an example where they don't give the
> same outcome.
>
>
I think this is a minimal example https://bugs.python.org/issue32113
Also Yury explains there why [await x for y in z ] is different from
list(await x for y in z).
Although I understand why it works this way, TBH it is not very intuitive.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 20:05, Guido van Rossum  wrote:

> On Wed, Nov 22, 2017 at 10:54 AM, Jelle Zijlstra  > wrote
>
>> 2017-11-22 9:58 GMT-08:00 Guido van Rossum :
>>
> (OTOH, await in the same position must keep working since it's not broken
>> and not unintuitive either.)
>>
>
>

This is very questionable IMO.
So do you think that [await x for y in z] and list(await x for y in z)
being not equivalent is intuitive?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 20:15, Guido van Rossum <gu...@python.org> wrote:

> On Wed, Nov 22, 2017 at 11:08 AM, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
>
>> On 22 November 2017 at 19:54, Jelle Zijlstra <jelle.zijls...@gmail.com>
>> wrote:
>>
>>>
>>> One proposal is to make it so `g` gets assigned a list, and the `yield`
>>> happens in the enclosing scope (so the enclosing function would have to be
>>> a generator). This was the way things worked in Python 2, I believe.
>>>
>>> Another proposal is to make this code a syntax error, because it's
>>> confusing either way. (For what it's worth, that would be my preference.)
>>>
>>>
>> Concerning this two options it looks like me and Serhiy like the first
>> one, Paul is undecided (), and Antoine is in favor of option 2.
>>
>
> While that may be the right thing to do, it's a silent change in
> semantics, which I find pretty disturbing -- how would people debug such a
> failure?
>

Some may call this just fixing a bug (At least in two mentioned
Stackoverflow questions and in two b.p.o. issues the current behaviour is
considered a bug).
But anyway, it is not me who decides :-)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 19:54, Jelle Zijlstra 
wrote:

>
> One proposal is to make it so `g` gets assigned a list, and the `yield`
> happens in the enclosing scope (so the enclosing function would have to be
> a generator). This was the way things worked in Python 2, I believe.
>
> Another proposal is to make this code a syntax error, because it's
> confusing either way. (For what it's worth, that would be my preference.)
>
>
Concerning this two options it looks like me and Serhiy like the first one,
Paul is undecided (), and Antoine is in favor of option 2.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 18:15, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 22 November 2017 at 16:47, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > On 22 November 2017 at 17:43, Paul Moore <p.f.mo...@gmail.com> wrote:
> >>
> >> On 22 November 2017 at 16:30, Ivan Levkivskyi <levkivs...@gmail.com>
> >> wrote:
> >> > On 22 November 2017 at 17:24, Antoine Pitrou <solip...@pitrou.net>
> >> > wrote:
> >> >> Given a comprehension (e.g. list comprehension) is expected to work
> >> >> nominally as `constructor(generator expression)`
> >> >
> >> > As Yury just explained, these two are not equivalent if there is an
> >> > `await`
> >> > in the comprehension/generator expression.
> >>
> >> As Antoine said, people *expect* them to work the same.
> >
> >
> > The difference is that a generator expression can be used independently,
> one
> > can assign it to a variable etc. not necessary to wrap it into a list()
> > Anyway, can you propose an equivalent "defining" code for both?
> Otherwise it
> > is not clear what you are defending.
>
> [...]
>
> 1. List comprehensions expand into nested for/if statements in the
> "obvious" way - with an empty list created to start and append used to
> add items to it.
>1a. Variables introduced in the comprehension don't "leak" (see below).
> 2. Generator expressions expand into generators with the same "nested
> loop" behaviour, and a yield of the generated value.
> 3. List comprehensions are the same as list(the equivalent generator
> expression).
>
>
Paul, OK, I think how to formulate these rules more "precisely" so that
they will be all consistent even if there is a `yield` inside.
The key idea is that neither comprehensions nor generator expressions
should create a function scope surrounding the `expr` in (expr for ind in
iterable) and [expr for ind in iterable].
(this still can be some hidden implementation detail)

So as Serhiy proposed in one of his first posts any comprehensions and
generator expressions with `yield` are not valid outside functions.
If such comprehension or generator expression is inside a function, then it
makes it a generator, and all the `yiled`s are yielded from that generator,
for example:

def fun_gen():
return list((yield i) for i in range(3))

should work as following:

g = func_gen()

g.send(42)
0
g.send(43)
1
g.send(44)
2
try:
g.send(45)
except StopIteration as e:
assert e.value  == [42, 43, 44]

And exactly the same with

def fun_comp():
[(yield i) for i in range(3)]

I hope with this we can close the no-async part of the problem.
Currently this is not how it works, and should be fixed. Do you agree?

The async part can then be considered separately.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 18:15, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 22 November 2017 at 16:47, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > On 22 November 2017 at 17:43, Paul Moore <p.f.mo...@gmail.com> wrote:
> >>
> >> On 22 November 2017 at 16:30, Ivan Levkivskyi <levkivs...@gmail.com>
> >> wrote:
> >> > On 22 November 2017 at 17:24, Antoine Pitrou <solip...@pitrou.net>
> >> > wrote:
> >> >> Given a comprehension (e.g. list comprehension) is expected to work
> >> >> nominally as `constructor(generator expression)`
> >> >
> >> > As Yury just explained, these two are not equivalent if there is an
> >> > `await`
> >> > in the comprehension/generator expression.
> >>
> >> As Antoine said, people *expect* them to work the same.
> >
> >
> > The difference is that a generator expression can be used independently,
> one
> > can assign it to a variable etc. not necessary to wrap it into a list()
> > Anyway, can you propose an equivalent "defining" code for both?
> Otherwise it
> > is not clear what you are defending.
>
> [...snip...]
>
> 1. List comprehensions expand into nested for/if statements in the
> "obvious" way - with an empty list created to start and append used to
> add items to it.
>1a. Variables introduced in the comprehension don't "leak" (see below).
> 2. Generator expressions expand into generators with the same "nested
> loop" behaviour, and a yield of the generated value.
> 3. List comprehensions are the same as list(the equivalent generator
> expression).
>

Great, I agree with all three rules.
But there is a problem, it is hard to make these three rules consistent in
some corner cases even _without async_.
For example, with the original problematic example, it is not clear to me
how to apply the rule 2 so that it is consistent with 3:

def fun_comp():
return [(yield i) for i in range(3)]

def fun_gen():
return list((yield i) for i in range(3))

I think the solution may be to formulate the rules in terms of the iterator
protocol (__iter__ and __next__).
I will try to think more about this.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 17:50, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 22 November 2017 at 16:38, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > On 22 November 2017 at 17:16, Paul Moore <p.f.mo...@gmail.com> wrote:
> >>
> >> Docs more importantly than PEP IMO. And are you implying that there's
> >> a difference between generator expressions and comprehensions? I
> >> thought both were intended to behave as if expanded to a function
> >> containing nested for loops? Nothing said in this thread so far (about
> >> semantics, as opposed to about current behaviour) implies there's a
> >> deliberate difference.
> >
> >
> > I think there may be a difference:
> >
> > comprehension `g = [(yield i) for i in range(3)]` is defined as this
> code:
> >
> > __result = []
> > __i = None
> > try:
> > for __i in range(3):
> > __result.append(yield __i)
> > g = __result
> > finally:
> > del __result, __i
>
> Not in the docs, it isn't...


Yes, since there is almost nothing there, this is what I _propose_ (or
actually Serhiy proposed it first)


> The docs explicitly state that a new
> scope is involved.
>

But docs don't say it is a _function_ scope. The meaning of that statement
(as I understand it) is just that the loop variable doen't leak from
comprehension.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 17:43, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 22 November 2017 at 16:30, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > On 22 November 2017 at 17:24, Antoine Pitrou <solip...@pitrou.net>
> wrote:
> >> Given a comprehension (e.g. list comprehension) is expected to work
> >> nominally as `constructor(generator expression)`
> >
> > As Yury just explained, these two are not equivalent if there is an
> `await`
> > in the comprehension/generator expression.
>
> As Antoine said, people *expect* them to work the same.
>

The difference is that a generator expression can be used independently,
one can assign it to a variable etc. not necessary to wrap it into a list()
Anyway, can you propose an equivalent "defining" code for both? Otherwise
it is not clear what you are defending.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

Sorry, forgot some details in the second "definition":

 try:
 def __gen():
 for i in range(3):
 yield (yield i)
 g = list(__gen())
 finally:
 del __gen

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 17:16, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 22 November 2017 at 16:08, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > On 22 November 2017 at 16:56, Yury Selivanov <yselivanov...@gmail.com>
> > wrote:
> >>
> >> On Wed, Nov 22, 2017 at 10:10 AM, Ivan Levkivskyi <levkivs...@gmail.com
> >
> >> wrote:
> >> > On 22 November 2017 at 15:47, Paul Moore <p.f.mo...@gmail.com> wrote:
> >> [...]
> >> I'm all for prohibiting using 'yield' expression in generator
> >> expressions or comprehensions.  The semantics is way to hard to
> >> understand and hence be of any value.
> >>
> >> Making 'await' a SyntaxError is absolutely not an option.  Async
> >> generator expressions are a shorthand syntax for defining asynchronous
> >> generators (PEP 525), and it's already being used in the wild.
> >
> >
> > OK, makes sense, so it looks like we may have the following plan:
> >
> > - fix `yield` in comprehensions
>
> I'm still not clear what "fix" would actually mean, but you propose
> clarifying the docs below, so I assume it means "according to whatever
> the updated docs say"...
>
>
I mean the initial proposal: make comprehensions equivalent to a for-loop


> > - update PEP 530 and docs re generator expressions vs comprehensions
>
> Docs more importantly than PEP IMO. And are you implying that there's
> a difference between generator expressions and comprehensions? I
> thought both were intended to behave as if expanded to a function
> containing nested for loops? Nothing said in this thread so far (about
> semantics, as opposed to about current behaviour) implies there's a
> deliberate difference.
>

I think there may be a difference:

comprehension `g = [(yield i) for i in range(3)]` is defined as this code:

__result = []
__i = None
try:
for __i in range(3):
__result.append(yield __i)
g = __result
finally:
del __result, __i

while `g = list((yield i) for i in range(3))` is defined as this code:

def __gen():
   for i in range(3):
 yield (yield i)
g = list(__gen())

 Although these two definitions are equivalent in simple cases (like having
`f(i)` instead of `yield i`)

But this is debatable, I think before we move to other points we need to
agree on the clear definitions of semantics of generator expressions and
comprehensions.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 17:24, Antoine Pitrou <solip...@pitrou.net> wrote:

> On Wed, 22 Nov 2017 17:08:14 +0100
> Ivan Levkivskyi <levkivs...@gmail.com> wrote:
> > On 22 November 2017 at 16:56, Yury Selivanov <yselivanov...@gmail.com>
> > wrote:
> >
> > > On Wed, Nov 22, 2017 at 10:10 AM, Ivan Levkivskyi <
> levkivs...@gmail.com>
> > > wrote:
> > > > On 22 November 2017 at 15:47, Paul Moore <p.f.mo...@gmail.com>
> wrote:
> > > [...]
> > > I'm all for prohibiting using 'yield' expression in generator
> > > expressions or comprehensions.  The semantics is way to hard to
> > > understand and hence be of any value.
> > >
> > > Making 'await' a SyntaxError is absolutely not an option.  Async
> > > generator expressions are a shorthand syntax for defining asynchronous
> > > generators (PEP 525), and it's already being used in the wild.
> > >
> >
> > OK, makes sense, so it looks like we may have the following plan:
> >
> > - fix `yield` in comprehensions
> > - update PEP 530 and docs re generator expressions vs comprehensions
> > - make `yield` in generator expressions a SyntaxError
>
> Given a comprehension (e.g. list comprehension) is expected to work
> nominally as `constructor(generator expression)`
>

As Yury just explained, these two are not equivalent if there is an `await`
in the comprehension/generator expression.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 16:56, Yury Selivanov <yselivanov...@gmail.com>
wrote:

> On Wed, Nov 22, 2017 at 10:10 AM, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > On 22 November 2017 at 15:47, Paul Moore <p.f.mo...@gmail.com> wrote:
> [...]
> I'm all for prohibiting using 'yield' expression in generator
> expressions or comprehensions.  The semantics is way to hard to
> understand and hence be of any value.
>
> Making 'await' a SyntaxError is absolutely not an option.  Async
> generator expressions are a shorthand syntax for defining asynchronous
> generators (PEP 525), and it's already being used in the wild.
>

OK, makes sense, so it looks like we may have the following plan:

- fix `yield` in comprehensions
- update PEP 530 and docs re generator expressions vs comprehensions
- make `yield` in generator expressions a SyntaxError

If everyone agrees, then I propose to open a separate issue on b.p.o. to
coordinate the efforts.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 16:53, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 22 November 2017 at 15:10, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > I think there may be a small misunderstanding here. The situation is
> > different for comprehensions and generator expressions,
> > let me summarize the current state:
> [...]
> > What do you think?
>
> Can you point me to where in the docs it explains the semantic
> difference between a generaor expression and a list comprehension?
> Ignoring the trivial point that they return different values - think
> of [a for a in expr] vs list(a for a in expr) if you like - these
> should be semantically the same, I believe?
>
> TBH, the docs aren't super-precise in this area. Before we start
> debating the "correct" behaviour of edge cases, I think we need to
> agree (and document) the semantics more precisely.
>
> Paul
>

Looking at
https://docs.python.org/3/reference/expressions.html?highlight=generator%20expression#generator-expressions
it is indeed not super clear what it _should_ mean.
I agree this requires documentation updates.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 15:47, Paul Moore  wrote:

> I generally don't understand "await" in any context, so I deferred
> judgement on that :-) Based on your comment that they are equally
> tricky, I'd suggest we prohibit them both ;-)
>
> Less facetiously, comprehensions are defined in the language reference
> in terms of a source translation to nested loops. That description
> isn't 100% precise, but nevertheless, if yield/async in a
> comprehension doesn't behave like that, I'd consider it a bug. So
> current behaviour (for both yield and await) is a bug, and your
> proposed semantics for yield is correct.
>

I think there may be a small misunderstanding here. The situation is
different for comprehensions and generator expressions,
let me summarize the current state:

- yield in comprehensions works "wrong" (a shorthand for not according to
the docs/naive expectations, i.e. not equivalent to for loop)
- await in comprehensions works "right"
- yield in generator expressions works "wrong"
- await in generator expressions works "wrong"

After some thinking, both `yield` and `await` look quite mind bending in
_generator expressions_, so maybe the right compromise strategy is:

- fix yield in comprehensions
- await in comprehensions already works
- make both `yield` and `await` a SyntaxError in generator expressions.

What do you think?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 15:46, Antoine Pitrou <solip...@pitrou.net> wrote:

> On Wed, 22 Nov 2017 15:15:49 +0100
> Ivan Levkivskyi <levkivs...@gmail.com> wrote:
> > There are many things that I would reject in code review, but they are
> > still allowed in Python,
> > this is one of the reasons why code reviews exist. Also I am not sure how
> > `yield` in a comprehension
> > is more tricky than `await` in a comprehension.
>
> I am not sure either, but do note that "yield" and "await" are two
> different things with different semantics, so allowing "await" while
> disallowing "yield" wouldn't strike me as inconsistent.
>
> The exact semantics of "yield" inside a comprehension is a common
> source of surprise or bewilderment, and the only actual use I've seen
> of it is to show it off as a "clever trick".  So instead of fixing (and
> perhaps complicating) the implementation to try and make it do the
> supposedly right thing, I am proposing to simply disallow it so that
> we are done with the controversy :-)
>
>
Actually, I am not sure there is really a controversy, I have not yet met a
person who _expects_
`yield` in a comprehension to work as it works now, instead everyone thinks
it is just equivalent to a for-loop.

Anyway, I have some compromise idea, will send it in a separate e-mail.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 15:09, Paul Moore <p.f.mo...@gmail.com> wrote:

> On 22 November 2017 at 13:53, Ivan Levkivskyi <levkivs...@gmail.com>
> wrote:
> > On 22 November 2017 at 14:38, Antoine Pitrou <solip...@pitrou.net>
> wrote:
> >>
> >> On Wed, 22 Nov 2017 15:03:09 +0200
> >> Serhiy Storchaka <storch...@gmail.com> wrote:
> >> >  From
> >> >
> >> > https://stackoverflow.com/questions/45190729/
> differences-between-generator-comprehension-expressions.
> >> >
> >> >  g = [(yield i) for i in range(3)]
> >> >
> >> > Syntactically this looks like a list comprehension, and g should be a
> >> > list, right? But actually it is a generator. This code is equivalent
> to
> >> > the following code:
> >> >
> >> >  def _make_list(it):
> >> >  result = []
> >> >  for i in it:
> >> >  result.append(yield i)
> >> >  return result
> >> >  g = _make_list(iter(range(3)))
> >> >
> >> > Due to "yield" in the expression _make_list() is not a function
> >> > returning a list, but a generator function returning a generator.
> >> >
> >> > This change in semantic looks unintentional to me. It looks like
> leaking
> >> > an implementation detail.
> >>
> >> Perhaps we can deprecate the use of "yield" in comprehensions and make
> >> it a syntax error in a couple versions?
> >>
> >> I don't see a reason for writing such code rather than the more
> >> explicit variants.  It looks really obscure, regardless of the actual
> >> semantics.
> >
> >
> > People actually try this (probably simply because they like
> comprehensions)
> > see two mentioned Stackoverflow questions, plus there are two b.p.o.
> issues.
> > So this will be a breaking change. Second, recent PEP 530 allowed
> writing a
> > similar comprehensions with `await`:
> >
> > async def process(funcs):
> > result = [await fun() for fun in funcs]  # OK
> > ...
> >
> > Moreover, it has the semantics very similar to the proposed by Serhiy for
> > `yield` (i.e. equivalent to for-loop without name leaking into outer
> scope).
> > Taking into account that the actual fix is not so hard, I don't think it
> > makes sense to have all the hassles of deprecation period.
>
> I agree with Antoine. This (yield in a comprehension) seems far too
> tricky, and I'd rather it just be rejected. If I saw it in a code
> review, I'd certainly insist it be rewritten as an explicit loop.
>
>
There are many things that I would reject in code review, but they are
still allowed in Python,
this is one of the reasons why code reviews exist. Also I am not sure how
`yield` in a comprehension
is more tricky than `await` in a comprehension. Anyway, this looks more
like a taste question.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 14:53, Serhiy Storchaka <storch...@gmail.com> wrote:

> 22.11.17 15:25, Ivan Levkivskyi пише:
>
>> I think this is indeed a problem.. For me the biggest surprise was that
>> `yield` inside a comprehension does not turn a surrounding function into
>> comprehension, see also
>> https://stackoverflow.com/questions/29334054/why-am-i-gettin
>> g-different-results-when-using-a-list-comprehension-with-coroutin
>>
>> In fact there is a b.p.o. issue for this https://bugs.python.org/issue1
>> 0544, it is assigned to me since July, but I was focused on other things
>> recently.
>> My plan was to restore the Python 2 semantics while still avoiding the
>> leak of comprehension variable to the enclosing scope (the initial reason
>> of introducing auxiliary "_make_list" function IIUC).
>> So that:
>>
>> 1) g = [(yield i) for i in range(3)] outside a function will be a
>> SyntaxError (yield outside a function)
>> 2) g = [(yield i) for i in range(3)] inside a function will turn that
>> enclosing function into generator.
>> 3) accessing i after g = [(yield i) for i in range(3)] will give a
>> NameError: name 'i' is not defined
>>
>> If you have time to work on this, then I will be glad if you take care of
>> this issue, you can re-assign it.
>>
>
> I have the same plan. I know how implement this for comprehensions, but
> the tricky question is what to do with generator expressions? Ideally
>
> result = [expr for i in iterable]
>
> and
>
> result = list(expr for i in iterable)
>
> should have the same semantic. I.e. if expr is "(yield i)", this should
> turn the enclosing function into a generator function, and fill the list
> with values passed to the generator's .send(). I have no idea how implement
> this.
>
>
Yes, generator expressions are also currently problematic with `await`:

>>> async def g(i):
... print(i)
...
>>> async def f():
... result = list(await g(i) for i in range(3))
... print(result)
...
>>> f().send(None)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in f
TypeError: 'async_generator' object is not iterable

Maybe Yury can say something about this?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 14:38, Antoine Pitrou  wrote:

> On Wed, 22 Nov 2017 15:03:09 +0200
> Serhiy Storchaka  wrote:
> >  From
> > https://stackoverflow.com/questions/45190729/
> differences-between-generator-comprehension-expressions.
> >
> >  g = [(yield i) for i in range(3)]
> >
> > Syntactically this looks like a list comprehension, and g should be a
> > list, right? But actually it is a generator. This code is equivalent to
> > the following code:
> >
> >  def _make_list(it):
> >  result = []
> >  for i in it:
> >  result.append(yield i)
> >  return result
> >  g = _make_list(iter(range(3)))
> >
> > Due to "yield" in the expression _make_list() is not a function
> > returning a list, but a generator function returning a generator.
> >
> > This change in semantic looks unintentional to me. It looks like leaking
> > an implementation detail.
>
> Perhaps we can deprecate the use of "yield" in comprehensions and make
> it a syntax error in a couple versions?
>
> I don't see a reason for writing such code rather than the more
> explicit variants.  It looks really obscure, regardless of the actual
> semantics.
>

People actually try this (probably simply because they like comprehensions)
see two mentioned Stackoverflow questions, plus there are two b.p.o. issues.
So this will be a breaking change. Second, recent PEP 530 allowed writing a
similar comprehensions with `await`:

async def process(funcs):
result = [await fun() for fun in funcs]  # OK
...

Moreover, it has the semantics very similar to the proposed by Serhiy for
`yield` (i.e. equivalent to for-loop without name leaking into outer scope).
Taking into account that the actual fix is not so hard, I don't think it
makes sense to have all the hassles of deprecation period.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

Serhiy,

I think this is indeed a problem. For me the biggest surprise was that
`yield` inside a comprehension does not turn a surrounding function into
comprehension, see also
https://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin

In fact there is a b.p.o. issue for this https://bugs.python.org/issue10544,
it is assigned to me since July, but I was focused on other things recently.
My plan was to restore the Python 2 semantics while still avoiding the leak
of comprehension variable to the enclosing scope (the initial reason of
introducing auxiliary "_make_list" function IIUC).
So that:

1) g = [(yield i) for i in range(3)] outside a function will be a
SyntaxError (yield outside a function)
2) g = [(yield i) for i in range(3)] inside a function will turn that
enclosing function into generator.
3) accessing i after g = [(yield i) for i in range(3)] will give a
NameError: name 'i' is not defined

If you have time to work on this, then I will be glad if you take care of
this issue, you can re-assign it.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Comments on PEP 560 (Core support for typing module and generic types)

[Ivan Levkivskyi]

On 20 November 2017 at 10:22, Mark Shannon <m...@hotpy.org> wrote:

> On 19/11/17 22:36, Ivan Levkivskyi wrote:
>
>> Except that there is no such thing as object._getitem__. Probably you
>> mean PyObject_GetItem (which is just what is done by BINARY_SUBSCR opcode).
>>
> In fact, I initially implemented type.__getitem__, but I didn't like it
>> for various reasons.
>
>
> Could you elaborate?
>

I didn't like that although type.__getitem__ would exist, it would almost
always raise, which can be confusing. Also dir(type) is already long,
I don't want to make it even longer. Maybe there were other reasons that I
forgot.

Anyway, I propose to stop here, since this is not about the PEP, this is
about implementation.
This is a topic of a separate discussion.


> Given the power and flexibility of the built-in data structures, defining
> custom containers is relatively rare. I'm not saying that it should not be
> considered,

but a few minor hurdles are acceptable to keep the rest of the language
> (including more common uses of type-hints) clean.
>

This essentially means changing decisions already made in PEP 484 and not a
topic of this PEP.
Also,

@Generic[T]
class C:
...

is currently a syntax error (only names and function calls are allowed in a
decorator).
Finally, it is too late to change how generics are declared, since it will
break
existing code.

Should `isinstance` and `issubclass` call `__mro_entries__` before
>> raising an error if the second argument is not a class?
>> In other words, if `List` implements `__mro_entries__` to return
>> `list` then should `issubclass(x, List)` act like `issubclass(x,
>> list)`?
>> (IMO, it shouldn't) The reasoning behind this decision should be
>> made explicit in the PEP.
>>
>>
>> I think this is orthogonal to the PEP. There are many situations where a
>> class is expected,
>> and IMO it is clear that all that are not mentioned in the PEP stay
>> unchanged.
>>
>
> Indeed, but you do mention issubclass in the PEP. I think a few extra
> words of explanation would be helpful.
>

OK, I will add a comment to the PEP.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Comment on PEP 562 (Module __getattr__ and __dir__)

[Ivan Levkivskyi]

On 20 November 2017 at 20:51, Guido van Rossum  wrote:

> Yeah, I don't think there's an action item here except *maybe* changes to
> the wording of the PEP. Ivan?
>

Yes, I will make a small PR with a more detailed description of how
__getattr__ works.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com