[issue47006] PEP 646: Decide on substitution behavior
Matthew Rahtz added the comment: Ok, https://github.com/python/cpython/pull/32341/files is a reference of how the current implementation behaves. Fwiw, it *is* mostly correct - with a few minor tweaks it might be alright for at least the 3.11 release. In particular, instead of dealing with the thorny issue of what to do about splitting unpacked arbitrary-length tuples over multiple type variables - e.g. C[T, *Ts][*tuple[int, ...]] - instead either deciding to try and evaluate it properly and living with the complexity, or leaving it unsimplified and living with the __args__, __parameters__ and __origin__ problem - for now, we could just raise an exception for any substitutions which involve an unpacked arbitrary-length tuple, since I'd guess it's going to be an extremely rare use-case. -- ___ Python tracker <https://bugs.python.org/issue47006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47006] PEP 646: Decide on substitution behavior
Change by Matthew Rahtz : -- keywords: +patch pull_requests: +30396 stage: -> patch review pull_request: https://github.com/python/cpython/pull/32341 ___ Python tracker <https://bugs.python.org/issue47006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47006] PEP 646: Decide on substitution behavior
Matthew Rahtz added the comment: [Guido] > 1. Some edge case seems to be that if *tuple[...] is involved on either side > we will never simplify. Alright, let me think this through with some examples to get my head round it. It would prohibit the following difficult case: class C(Generic[*Ts]): ... Alias = C[T, *Ts] Alias[*tuple[int, ...]] # Does not simplify; stays C[T, *Ts][*tuple[int, ...]] That seems pretty reasonable. It would also prohibit these other relatively simple cases, but I guess that's fine: Alias = C[*Ts] Alias[*tuple[int, ...]] # Does not simplify; stays C[*Ts][*tuple[int, ...]] Alias = C[T, *tuple[int, ...]] Alias[str] # Does not simplify; stays C[T, *tuple[int, ...]][str] > Or perhaps a better rule is that *tuple[...] is never simplified away (but > fixed items before and after it may be). Is this to say that we effectively prohibit binding *tuple[...] to anything? If we can simplify without binding *tuple[...] to anything, then we do simplify, but otherwise, we don't simplify? So under this rule, the following WOULD work? Alias = C[T, *tuple[int, ...]] Alias[str] # Simplifies to C[str, *tuple[int, ...]], because we didn't have to bind *tuple[int, ...] to do it > 2. Another edge case is that if neither side has any starred items we will > always simplify (since this is the existing behavior in 3.10). This may raise > an error if the number of subscripts on the right does not match the number > of parameters on the left. Alright, so this is business as usual. > 3. If there's a single *Ts on the left but not on the right, we should be > able to simplify, which again may raise an error if there are not enough > values on the right, but if there are more than enough, the excess will be > consumed by *Ts (in fact that's the only way *Ts is fed). So then: class C(Generic[*Ts]): ... Alias = C[T, *Ts] Alias[()] # Raises error Alias[int] # Simplifies to C[int, *Ts] Alias[int, str]# Simplifies to C[int, str] Alias[int, str, bool] # Simplifies to C[int, str, bool] Yup, seems straightforward. > 4. If there's a *Ts on the right but not on the left, we should _not_ > simplify, since whatever we have on the left serves as a constraint for *Ts. Ok, so this is about the following situations: class C(Generic[*Ts]): ... Alias = C[T1, T2] Alias[*Ts] # Does not simplify; stays C[T1, T2][*Ts] Yikes - in fact, this is actually super hairy; I hadn't thought about this edge case at all in the PEP. Agreed that it seems reasonable not to simplify here. > E.g. tuple[int, int][*Ts] constrains *Ts to being (int, int). Was that a typo? Surely tuple[int, int][*Ts] isn't valid - since tuple[int, int] doesn't have any free parameters? > 5. If there's exactly one *Ts on the left and one on the right, we _might__ > be able to simplify if the prefix and suffix of the __parameters__ match the > prefix and suffix of the subscript on the right. E.g. C[int, T, *Ts, > float][str, *Ts] can be simplified to C[int, str, *Ts, float]. OTOH C[int, T, > *Ts, float][*Ts] cannot be simplified -- but we cannot flag it as an error > either. Note that __parameters__ in this example is (T, Ts); we have to > assume that typevartuples in __parameters__ are always used as *Ts (since the > PEP recognizes no valid unstarred uses of Ts). Ok, this also makes sense. --- Still, though, doesn't the point that Serhiy brought up about __origin__, __parameters__ and __args__ still apply? In cases where we *don't* simplify, there'd still be the issue of what we'd set these things to be. This evening I'll also revisit the PRs adding tests for substitution to try and make them a comprehensive reference as to what's currently possible. -- ___ Python tracker <https://bugs.python.org/issue47006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47006] PEP 646: Decide on substitution behavior
Matthew Rahtz added the comment: Apologies for the slow reply - coming back to this now that the docs and pickling issues are mostly sorted. [Serhiy] > > Alias = C[T, *Ts] > > Alias2 = Alias[*tuple[int, ...]] > > # Alias2 should be C[int, *tuple[int, ...]] > > tuple[int, ...] includes also an empty tuple, and in this case there is no > value for T. This was my initial intuition too, but Pradeep pointed out to me in https://github.com/python/cpython/pull/31021#discussion_r815853784 that for tuple[int, ...], Python has chosen the opposite mindset: instead of assuming the worst-case scenario, we assume the best-case scenario. Thus, the following type-checks correctly with mypy (https://mypy-play.net/?mypy=latest=3.10=b9ca66fb7d172f939951a741388836a6): def return_first(tup: tuple[int, ...]) -> int: return tup[0] tup: tuple[()] = () return_first(tup) > > We actually deliberately chose not to unpack concrete tuple types - see the > > description of https://github.com/python/cpython/pull/30398, under the > > heading 'Starred tuple types'. (If you see another way around it, though, > > let me know.) > > You assumed that *tuple[str, bool] in def foo(*args: *tuple[str, bool]) > should give foo.__annotations__['args'] = tuple[str, bool], but it should > rather give (str, bool). No confusion with tuple[str, bool]. Fair point, we could *technically* distinguish between tuple[str, bool] and (str, bool). But if I was a naive user and I saw `foo.__annotations__['args'] == (str, bool)`, I don't think it'd be immediately obvious to me that the type of `args` was `*tuple[str, bool]`. Also though, there's a second reason mentioned in https://github.com/python/cpython/pull/30398 why `(str, bool)` wouldn't be the best choice. We decided that the runtime behaviour of `*args: *something` should be that we essentially do `(*something,)[0]`. If we made `tuple[int, str]` unpack to `(int, str)`, then we'd end up with `__annotations__['args'] == (int,)`. > And one of PEP 646 options is to implement star-syntax only in subscription, > not in var-parameter type annotations. As in, we would allow `Generic[*Ts]`, but not `*args: *Ts`? That'd be a *major* change to the PEP - not an option I'm willing to consider at this stage in the process. > > I'm also not sure about this one; disallowing unpacked TypeVarTuples in > > argument lists to generic aliases completely (if I've understood right?) > > No, it will only be disallowed in substitution of a VarType. Tuple[T][*Ts] -- > error. Tuple[*Ts][*Ts2] -- ok. Ah, gotcha. My mistake. [Guido] I ran out of time this evening :) Will reply properly soon. -- ___ Python tracker <https://bugs.python.org/issue47006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Matthew Rahtz added the comment: > 1. Finish writing docs Done once https://github.com/python/cpython/pull/32103 is merged. > 2. Implement support for pickling of unpacked native tuples Done once https://github.com/python/cpython/pull/32159 is merged. 4. Resolve the issue of how we implement type substitution (https://bugs.python.org/issue47006) Will re-visit this now. -- ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- pull_requests: +30237 pull_request: https://github.com/python/cpython/pull/32159 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Matthew Rahtz added the comment: Since things are piling up, here's a quick record of what I think the remaining tasks are: (in approximate order of priority) 1. Finish writing docs (is updating library/typing.html sufficient? https://github.com/python/cpython/pull/32103) 2. Implement support for pickling of unpacked native tuples 3. Implement support and add tests for copy() of TypeVarTuple and unpacked tuple 4. Resolve the issue of how we implement type substitution (https://bugs.python.org/issue47006) -- ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- pull_requests: +30197 pull_request: https://github.com/python/cpython/pull/32119 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47097] Document PEP 646
Change by Matthew Rahtz : -- keywords: +patch pull_requests: +30183 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/32103 ___ Python tracker <https://bugs.python.org/issue47097> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47097] Document PEP 646
Matthew Rahtz added the comment: Ooh, thanks for the reminder! I'll start drafting this now. -- nosy: +matthew.rahtz ___ Python tracker <https://bugs.python.org/issue47097> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47006] PEP 646: Decide on substitution behavior
Matthew Rahtz added the comment: P.s. To be clear, (I think?) these are all substitutions that are computable. We *could* implement the logic to make all these evaluate correctly if we wanted to. It's just a matter of how much complexity we want to allow in typing.py (or in the runtime in general, if we say farmed some of this logic out to a separate module). -- ___ Python tracker <https://bugs.python.org/issue47006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47006] PEP 646: Decide on substitution behavior
Matthew Rahtz added the comment:
[Guido]
> What would be an example of a substitution that's too complex to do?
We also need to remember the dreaded arbitrary-length tuple. For example, I
think it should be the case that:
```python
T = TypeVar('T')
Ts = TypeVarTuple('Ts')
class C(Generic[*Ts]): pass
Alias = C[T, *Ts]
Alias2 = Alias[*tuple[int, ...]]
# Alias2 should be C[int, *tuple[int, ...]]
```
Ok, this is a bit of a silly example, but if we're committing to evaluating
substitutions correctly, we should probably make even this kind of example
behave correctly so that users who accidentally do something silly can debug
what's gone wrong.
[Serhiy]
> A repr can be less readable.
Definitely true.
> It will break equality comparison and hashing. Good bye caching.
Huh, I didn't know about this one. Fair enough, this is totally a downside.
> What about __origin__, __parameters__, __args__? How will they be calculated?
This could admittedly be thorny. We'd have to think it through carefully.
Admittedly also a downside.
> It can break code which uses annotations for something. For example it can
> break dataclasses.
Oh, also interesting - I didn't know about this one either. Could you give an
example?
> The first case will be practically fixed by GH 32030 after chenging the
> grammar to allow unpacking in index tuple: A[*B].
We actually deliberately chose not to unpack concrete tuple types - see the
description of https://github.com/python/cpython/pull/30398, under the heading
'Starred tuple types'. (If you see another way around it, though, let me know.)
> Two other cases will be fixed by GH 32031. It does not require any C code.
I'm also not sure about this one; disallowing unpacked TypeVarTuples in
argument lists to generic aliases completely (if I've understood right?) seems
like too restrictive a solution. I can imagine there might be completely
legitimate cases where the ability to do this would be important. For example:
```python
DType = TypeVar('DType')
Shape = TypeVarTuple('Shape')
class Tensor(Generic[DType, *Shape]): ...
Uint8Tensor = Tensor[uint8, *Shape]
Unit8BatchTensor = Uint8Tensor[Batch, *Shape]
```
> Note that the alternative proposition is even more lenient to errors.
True, but at least it's predictably lenient to errors - I think the repr makes
it very clear that "Woah, you're doing something advanced here. You're on your
own!" I think it better fits the principle of least astonishment to have
something that consistently lets through all errors of a certain class than
something that sometimes catches errors and sometimes doesn't.
--
___
Python tracker
<https://bugs.python.org/issue47006>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47006] PEP 646: Decide on substitution behavior
Matthew Rahtz added the comment: (Having said that, to be clear: my preferred solution currently would still be the solution where we just return a new GenericAlias for anything involving a TypeVarTuple. The crux is what Serhiy is happy with.) -- ___ Python tracker <https://bugs.python.org/issue47006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47006] PEP 646: Decide on substitution behavior
Matthew Rahtz added the comment: Thanks for starting this, Jelle - I was a bit unsure about how to proceed here. Given that https://github.com/python/cpython/pull/31800 is already merged, I'd also propose something halfway between the two extremes: return a sensible substitution when the logic to compute that isn't too onerous, and a new GenericAlias object when it is. The upsides are that we'd probably be able to return reasonable substitutions for the vast majority of cases, and that we wouldn't have to remove what's already been merged. The downsides would be lack of consistency, and the potential for changing rules about what does and doesn't return a full substitution as time goes on and new features are added. -- nosy: +matthew.rahtz ___ Python tracker <https://bugs.python.org/issue47006> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- pull_requests: +29945 pull_request: https://github.com/python/cpython/pull/31846 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- pull_requests: +29944 pull_request: https://github.com/python/cpython/pull/31845 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- pull_requests: +29943 pull_request: https://github.com/python/cpython/pull/31844 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- pull_requests: +29905 pull_request: https://github.com/python/cpython/pull/31804 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46589] Improve documentation for typing._GenericAlias
New submission from Matthew Rahtz : There's currently not much documentation in `typing.py` for `_GenericAlias`. Some fairly weird things go on in there, so it would be great to have more info in the class about what's going on and why various edge cases are necessary. -- components: Library (Lib) messages: 412171 nosy: matthew.rahtz priority: normal pull_requests: 29210 severity: normal status: open title: Improve documentation for typing._GenericAlias type: enhancement versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46589> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- pull_requests: +29202 pull_request: https://github.com/python/cpython/pull/31021 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- pull_requests: +29200 pull_request: https://github.com/python/cpython/pull/31019 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- pull_requests: +29199 pull_request: https://github.com/python/cpython/pull/31018 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- pull_requests: +28607 pull_request: https://github.com/python/cpython/pull/30398 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646
Change by Matthew Rahtz : -- components: +Parser, Tests nosy: +lys.nikolaou, pablogsal title: Add support for PEP 646 (Variadic Generics) to typing.py -> Add support for PEP 646 versions: +Python 3.11 -Python 3.10 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646 (Variadic Generics) to typing.py
Change by Matthew Rahtz : -- keywords: +patch nosy: +matthew.rahtz nosy_count: 1.0 -> 2.0 pull_requests: +23313 stage: -> patch review pull_request: https://github.com/python/cpython/pull/24527 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43224] Add support for PEP 646 (Variadic Generics) to typing.py
Change by Matthew Rahtz : -- components: Library (Lib) nosy: mrahtz priority: normal severity: normal status: open title: Add support for PEP 646 (Variadic Generics) to typing.py versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue43224> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
