[Python-Dev] RFC on Callable Type Syntax
Hello all, Typing-sig has been discussing user-friendly syntax for the type used to represent callables. [1] Since this affects the Python language syntax, we wanted to get some high-level feedback from you before putting up a detailed PEP. TL;DR: We want to propose syntax for Callable types, e.g., `(int, str) -> bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are there concerns we need to keep in mind about such a syntax change? 2. Should we propose syntax for additional features in the same PEP or in a future PEP? # Motivation Guido has pointed out that `Callable` is one of the most frequently used type forms but has the least readable syntax. For example, say we have a callback `formatter` that accepts a record and a list of permissions: ```python def print_purchases( user, formatter, # <-- callback ): <...> output = formatter(record, permissions) print(output) ``` To give it a type, we currently have to write: ```python from typing import Callable def print_purchases( user: User, formatter: Callable[[PurchaseRecord, List[AuthPermission]], FormattedItem] # Hard to read. ) -> None: <...> output = formatter(record, permissions) print(output) ``` `Callable` can be hard to understand for new users since it doesn't resemble existing function signatures and there can be multiple square brackets. It also requires an import from `typing`, which is inconvenient. Around 40% of the time [2], users just give up on precisely typing the parameters and return type of their callbacks and just leave it as a blank Callable. In other cases, they don't add a type for their callback at all. Both mean that they lose the safety guarantees from typing and leave room for bugs. We believe that adding friendly syntax for Callable types will improve readability and encourage users to type their callbacks more precisely. Other modern, gradually-typed languages like TypeScript (JS), Hack (PHP), etc. have special syntax for Callable types. (As a precedent, PEP 604 recently added clean, user-friendly syntax for the widely-used `Union` type. Instead of importing `Union` and writing `expr: Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr | SubtractExpr`.) # Proposal and Questions We have a two-part proposal and questions for you: 1. Syntax to replace Callable After a lot of discussion, there is strong consensus in typing-sig about adding syntax to replace Callable. So, the above example would be written as: ```python def print_purchases( user: User, formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem, ) -> None: <...> output = formatter(record, permissions) print(output) ``` This feels more natural and succinct. Async callbacks currently need to be written as ``` from typing import Callable async_callback: Callable[[HttpRequest], Awaitable[HttpResponse]] ``` With the new syntax, they would be written as ``` async_callback: async (HttpRequest) -> HttpResponse ``` which again seems more natural. There is similar syntax for the type of decorators that pass on *args and **kwargs to the decorated function. Note that we considered and rejected using a full def-signature syntax like (record: PurchaseRecord, permissions: List[AuthPermission], /) -> FormattedItem because it would be more verbose for common cases and could lead to subtle bugs; more details in [3]. The Callable type is also usable as an expression, like in type aliases `IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls. **Question 1**: Are there concerns we should keep in mind about such a syntax proposal? 2. Syntax for callback types beyond Callable `Callable` can't express the type of all possible callbacks. For example, it doesn't support callbacks where some parameters have default values: `formatter(record)` (the user didn't pass in `permissions`). It *is* possible to express these advanced cases using Callback Protocols (PEP 544) [4] but it gets verbose. There are two schools of thought on typing-sig on adding more syntax on top of (1): (a) Some, including Guido, feel that it would be a shame to not have syntax for core Python features like default values, keyword arguments, etc. One way to represent default values or optionally name parameters would be: ``` # permissions is optional formatter: (PurchaseRecord, List[AuthPermission]=...) -> FormattedItem # permissions can be called using a keyword argument. formatter: (PurchaseRecord, permissions: List[AuthPermission]) -> FormattedItem ``` There are also alternative syntax proposals. (b) Others want to wait till we have more real-world experience with the syntax in (1). The above cases occur < 5% of the time in typed or untyped code [5]. And the syntax in (1) is forward-compatible with the additional proposals. So we could add them later if needed or leave them out, since we can always use callback protocols. **Questi
[Python-Dev] Re: RFC on Callable Type Syntax
Thanks for the responses! 1. Chris Angelico: > it looks like the parentheses are mandatory, and they are what define it? Yes, we always expect parens. Like you said, this keeps things consistent, resembles the function signature, and avoids ambiguities. It also won't look like Haskell function types `int -> str -> bool` (which can be confusing for non-functional programmers). This was pretty explicitly rejected when we polled people at the Typing Summit. > Which of these will be valid? (int) -> int (int, int) -> int (int,) -> int should also be fine, but I don't have strong opinions here. e.g., what about (,) -> int? We'll make sure to address this edge case explicitly, thanks. > There are parallel proposals to support a lambda syntax like "x => x.spam" as equivalent to "lambda x: x.spam", so I'm curious what the rules would be for the two types of arrows, and whether they'd be consistent with each other. Yeah, this had come up in our discussions. Like in other languages (such as Hack), we believe that lambdas should have a different arrow type to disambiguate the two. So, lambdas could be (x, y) => x + y. 2. Piotr Duda > Did you considered full def-signature with optional argument names, so the common cases would look like > > > (:PurchaseRecord, :List[AuthPermission]) -> FormattedItem > > Bare name signatures like '(record) -> FormattedItem' could be disallowed to prevent bugs. The additional-features proposal in (2) actually does have optional parameter names. There are two variants: (a) the "hybrid" proposal, which will let you write (int, str) -> bool, but also hybrids like (int, y: str) -> bool; (x: int, y: str) -> bool; (int, str=...) -> bool. (b) the "xor" proposal, which will let you use the shorthand syntax (int, str) -> bool for the most frequently-used cases and the full def-signature for all other advanced cases using familiar syntax: (x: int, y: str=...) -> bool. We haven't reached a strong consensus on the above two yet (flexibility vs familiarity). We decided to get some thoughts here about whether we need to propose additional-features at all (2) in our initial PEP. 3. Patrick Reader > How would this work for a function that returns a function? Would you just put it on the def line like this? We will allow using the callable syntax in the return type without parens: def foo() -> (int) -> str: return int_to_str The reason for not making that a syntax error is that it would be pretty bad UX. Users would see the callable syntax being used without parens everywhere: f: (int) -> str, but when they try to use that as the return type, they would get a confusing syntax error. So, given how frequently functions return functions, we decided to not require parens. People can specify parens in their coding guidelines (and auto-formatters) if they find parens more readable. 4. Paul Moore +1 to what Jelle said. > That's one of the differences between the proposals discussed here. The basic proposal Pradeep pointed out would not support named arguments, the more complete syntax favored by Guido (and also by me; 2(a) in Pradeep's email) would support it. > > Pradeep has done some empirical analysis that shows that in practice it is not very common to use such types, though. Yeah, that's basically Question 2 - whether to just replace Callable in the initial PEP (option 1) or to specify a more complete syntax from the beginning (option 2). 5. Serhiy Storchaka > How could you replace Callable[..., int] and Callable[Concatenate[str, P], int] ? To represent a Callable that accepts arbitrary arguments: (...) -> int To represent a Callable that accepts ParamSpec (for decorators, etc.), we're leaning towards the below: (**P) -> int To represent your Concatenate example: (str, **P) -> int We would no longer need the Concatenate operator. We can naturally just add types up front. -- S Pradeep Kumar On Fri, Oct 8, 2021 at 8:48 AM Guido van Rossum wrote: > On Fri, Oct 8, 2021 at 8:31 AM Jelle Zijlstra > wrote: > >> El vie, 8 oct 2021 a las 0:54, Paul Moore () >> escribió: >> >>> Also, note that I automatically used a type of int->int up there. As >>> someone else asked, is that form allowed, or is it required to be >>> (int)->int? In my view, if we require the latter, I expect there will >>> be a *lot* of people making that mistake and having to correct it. >>> >> >> That's not something we discussed before. I'd be OK with allowing this >> unless it makes the grammar too ambiguous. >> > > Even if it didn't make the grammar ambiguous (and I think it doesn't, as > long as the argument type isn't a union or another callable type
[Python-Dev] Re: RFC on PEP 673: Self Type
On Mon, Jan 17, 2022 at 7:02 AM Jelle Zijlstra wrote: > > > El lun, 17 ene 2022 a las 6:25, Petr Viktorin () > escribió: > >> On Wed, Nov 17, 2021 at 8:31 AM Pradeep Kumar Srinivasan >> wrote: >> > >> > This PEP [1] introduces a simple and intuitive way to annotate methods >> and classmethods that return an instance of their class. Such methods and >> classmethods occur quite frequently, but the existing way to annotate them >> correctly is quite arcane and error-prone. The PEP introduces a special >> type `Self` to represent the type of the `self` parameter, similar to the >> `this` type in TypeScript and the `Self` type in Rust. We have >> implementations for mypy and pyright. The PEP does not affect CPython >> directly except for the addition of one special form (Self) to typing.py >> [2]. >> > >> > Since we have reached consensus on the PEP in typing-sig [3], we wanted >> to get your comments and suggestions before submitting to the Steering >> Council. >> > >> > Thanks, >> > Pradeep Kumar Srinivasan >> > James Hilton-Balfe >> > >> > [1]: https://www.python.org/dev/peps/pep-0673/ >> > [2]: Adding `Self` to typing_extensions.py: >> https://github.com/python/typing/pull/933 >> > [3]: See the comments from typing-sig members on the Google doc: >> https://docs.google.com/document/d/1ujuSMXDmSIOJpiZyV7mvBEC8P-y55AgSzXcvhrZciuI/edit?usp=sharing >> >> Hello, and thanks for the PEP! >> Sorry I'm late, but I have two curious questions about the PEP. >> I don't think they should hold back accepting the PEP, but I'm >> interested in the answers. >> >> The PEP uses `reveal_type`, a function that's appeared in a few PEPs >> already, but was never described. Is it a standard function in typing >> tools, something specific to mypy, or pseudocode? >> > > It's a function that doesn't exist at runtime, but when a type checker > sees a call, it emits the inferred type of the argument. It originated with > mypy but I believe has been adopted by all type checkers. > > There's been some talk of adding it to the `typing` module, but that > hasn't happened so far. I opened https://bugs.python.org/issue46414 to > suggest adding it. > Yes, it is a pseudo function that has been adopted afaik by all type checkers, e.g., Mypy, Pyre, Pyright. It's useful for debugging type errors by using `reveal_type()` and running the type checker. I honestly didn't even realize that it wasn't specified in PEP 484 until you asked. Do you suggest adding an explanation in the PEP or is it reasonably self-explanatory? I see a few other PEPs using it. > >> >> The PEP says "we reject Self in metaclasses." >> "Metaclass" can mean "subclass of `type`", or it can refer to how a >> value is used -- for example, you can write `class >> Foo(metaclass=print): ...`. >> In the PEP's example, is MyMetaclass rejected because: >> - it's used as a metaclass in a class statement, or >> - it's a subclass of `type` (so it's rejected even if unused), or >> - it becomes a class of a class? >> Or is the exact interpretation best left to the type checker? >> > The PEP rejects `Self` used in `MyMetaclass` because the class is a subclass of `type`. So, there would be a type error even if `MyMetaclass` were unused. In general, the `# Rejected` comment indicates the position at which the type checker would emit a type error, as per the convention in other typing PEPs. So, in the PEP's metaclass example, we suggest that type errors be emitted at the uses of `Self` in the return annotations of `__new__` and `__mul__`. (The exact error and position is left up to the type checker.) Thanks for the questions! ___ >> Python-Dev mailing list -- python-dev@python.org >> To unsubscribe send an email to python-dev-le...@python.org >> https://mail.python.org/mailman3/lists/python-dev.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-dev@python.org/message/DVOQK3U6MOYXPRXF5OVLVLJBPEJRUIM5/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -- S Pradeep Kumar ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KSFAFKT6Q7CGD4V6B5WXSPPJHATPYGIA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: RFC on PEP 675: Arbitrary Literal Strings
On Mon, Feb 7, 2022 at 3:55 PM Brett Cannon wrote: > Can I suggest adding a "Specification" section (see > https://www.python.org/dev/peps/pep-0012/#suggested-sections for the > suggested sections to have in a PEP)? > Brett: I'm sorry about this oversight. I've fixed it in https://github.com/python/peps/pull/2313 -- S Pradeep Kumar ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IBIFCXEBTXJ4YM6PVC24BO6MRSG2DZ6R/ Code of Conduct: http://python.org/psf/codeofconduct/
