[Python-Dev] Re: Python multithreading without the GIL

[Guido van Rossum]

On Fri, Oct 8, 2021 at 8:55 PM Sam Gross  wrote:

> the "nogil" interpreter stays within the same interpreter loop for many
> Python function calls, while upstream CPython
> recursively calls into _PyEval_EvalFrameDefault.
>

Not for much longer though. https://github.com/python/cpython/pull/28488

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5AK2VA5FAFJ3NPYWRJCWNIIV77SEUIO2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Sam Gross]

On Fri, Oct 8, 2021 at 12:24 PM Pablo Galindo Salgado 
wrote:

> When you mean "an order of magnitude less overhead than the current
> CPython implementation" do you mean compared with the main branch? We
> recently implemented already almost everything is listed in this paragraph.
>

I think I wrote that in August when "current CPython" meant something
different from today :) I'll update it.

Thanks for the links to the PRs. I'll need to look at them more closely,
but one I think one remaining difference is that
the "nogil" interpreter stays within the same interpreter loop for many
Python function calls, while upstream CPython
recursively calls into _PyEval_EvalFrameDefault.

I've been using this mini-benchmark to measure the overhead of Python
function calls for various numbers of
arguments and keywords:
https://github.com/colesbury/nogil/blob/fb6aabede5f7f1936a21c2f48ec7fcc0848d74bf/benchmarks/call_benchmark.py

For zero, two, and four argument functions, I get:
nogil (nogil/fb6aabed): 10ns, 14ns, 18ns
3.11 (main/b108db63): 47ns, 54ns, 63ns
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UZKOY4Y3QWT76TCXJ3QXMEGRODN2DOGB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[S Pradeep Kumar]

Thanks for the responses!

1. Chris Angelico:

> it looks like the parentheses are mandatory, and they are what define it?

Yes, we always expect parens. Like you said, this keeps things consistent,
resembles the function signature, and avoids ambiguities. It also won't
look like Haskell function types `int -> str -> bool` (which can be
confusing for non-functional programmers).

This was pretty explicitly rejected when we polled people at the Typing
Summit.

> Which of these will be valid?

(int) -> int
(int, int) -> int

(int,) -> int should also be fine, but I don't have strong opinions here.
e.g., what about (,) -> int? We'll make sure to address this edge case
explicitly, thanks.

> There are parallel proposals to support a lambda syntax like "x =>
x.spam" as equivalent to "lambda x: x.spam", so I'm curious what the rules
would be for the two types of arrows, and whether they'd be consistent with
each other.

Yeah, this had come up in our discussions. Like in other languages (such as
Hack), we believe that lambdas should have a different arrow type to
disambiguate the two. So, lambdas could be (x, y) => x + y.

2. Piotr Duda

> Did you considered full def-signature with optional argument names, so
the common cases would look like
>
> 
> (:PurchaseRecord, :List[AuthPermission]) -> FormattedItem
> 
> Bare name signatures like '(record) -> FormattedItem' could be disallowed
to prevent bugs.

The additional-features proposal in (2) actually does have optional
parameter names. There are two variants:

(a) the "hybrid" proposal, which will let you write (int, str) -> bool, but
also hybrids like (int, y: str) -> bool; (x: int, y: str) -> bool; (int,
str=...) -> bool.

(b) the "xor" proposal, which will let you use the shorthand syntax (int,
str) -> bool for the most frequently-used cases and the full def-signature
for all other advanced cases using familiar syntax: (x: int, y: str=...) ->
bool.

We haven't reached a strong consensus on the above two yet (flexibility vs
familiarity). We decided to get some thoughts here about whether we need to
propose additional-features at all (2) in our initial PEP.

3. Patrick Reader

> How would this work for a function that returns a function? Would you
just put it on the def line like this?

We will allow using the callable syntax in the return type without parens:

def foo() -> (int) -> str:
return int_to_str

The reason for not making that a syntax error is that it would be pretty
bad UX. Users would see the callable syntax being used without parens
everywhere: f: (int) -> str, but when they try to use that as the return
type, they would get a confusing syntax error. So, given how frequently
functions return functions, we decided to not require parens.

People can specify parens in their coding guidelines (and auto-formatters)
if they find parens more readable.

4. Paul Moore

+1 to what Jelle said.

> That's one of the differences between the proposals discussed here. The
basic proposal Pradeep pointed out would not support named arguments, the
more complete syntax favored by Guido (and also by me; 2(a) in Pradeep's
email) would support it.
>
> Pradeep has done some empirical analysis that shows that in practice it
is not very common to use such types, though.

Yeah, that's basically Question 2 - whether to just replace Callable in the
initial PEP (option 1) or to specify a more complete syntax from the
beginning (option 2).

5. Serhiy Storchaka

> How could you replace Callable[..., int] and Callable[Concatenate[str,
P], int] ?

To represent a Callable that accepts arbitrary arguments:

(...) -> int

To represent a Callable that accepts ParamSpec (for decorators, etc.),
we're leaning towards the below:

(**P) -> int

To represent your Concatenate example:

(str, **P) -> int

We would no longer need the Concatenate operator. We can naturally just add
types up front.
--
S Pradeep Kumar

On Fri, Oct 8, 2021 at 8:48 AM Guido van Rossum  wrote:

> On Fri, Oct 8, 2021 at 8:31 AM Jelle Zijlstra 
> wrote:
>
>> El vie, 8 oct 2021 a las 0:54, Paul Moore ()
>> escribió:
>>
>>> Also, note that I automatically used a type of int->int up there. As
>>> someone else asked, is that form allowed, or is it required to be
>>> (int)->int? In my view, if we require the latter, I expect there will
>>> be a *lot* of people making that mistake and having to correct it.
>>>
>>
>> That's not something we discussed before. I'd be OK with allowing this
>> unless it makes the grammar too ambiguous.
>>
>
> Even if it didn't make the grammar ambiguous (and I think it doesn't, as
> long as the argument type isn't a union or another callable type), I'd be
> against this.
>
> An argument list is not a tuple, and we don't allow omitting the
> parentheses in other call-related situations: you can't write "def f x:
> return x+1" and you can't write "f 42". Allowing the omission of the
> parentheses here would be inconsistent (even if some other languages allow
> it).
>
> --
> --Guido van 

[Python-Dev] We should accept Final as indicating ClassVar for dataclasses

[Gregory Beauregard via Python-Dev]

(copied from typing on advice since this affects runtime behavior)
PEP 591 for Final states "Type checkers should infer a final attribute that is 
initialized in a class body as being a class variable. Variables should not be 
annotated with both ClassVar and Final."

ClassVar is used to indicate library behavior in dataclasses. Therefore, I 
propose accepting the Final attribute as an indicator of a ClassVar in 
dataclass class bodies in order to be better compatible with the Final PEP.

One edge case that would need to be handled is when someone might explicitly 
mark a dataclass field Final:
a: Final[int] = dataclasses.field(init=False, default=10)

I've opened an issue as well and would greatly appreciate any feedback: 
https://bugs.python.org/issue45384

I appreciate your time,
Gregory Beauregard
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HS2UEMTQJYPHXOI6FPU523JMCCHHRNGH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Sergei Lebedev]

Thanks for summarizing the PSC position, Barry!

I agree that type annotation syntax should be consistent with the rest of
the language, but I'm curious why it has to be "standard Python syntax"?
Could you elaborate a bit more on the reasoning behind that constraint?

Cheers,
Sergei

On Fri, Oct 8, 2021 at 8:38 PM Barry Warsaw  wrote:

> Thanks for thinking more deeply about how we can make type annotations for
> callables more user friendly.
>
> The intersection between the syntax for general Python code and type
> annotations has been a topic of discussion within the Steering Council (on
> and off) for quite some time.  We intend to provide more clarity about our
> thinking, probably in the form of an informational PEP, around this topic,
> and my apologies because I haven’t had time to draft it yet.
>
> But since this came up, I want to at least say that we are unanimous in
> our belief that the type annotation language must also be standard Python
> syntax.  In other words, we are not in favor of letting type annotation
> syntax deviate from Python syntax.  As a effect of this, the PSC does not
> intend to issue a blanket delegation of type-focused PEPs.  Of course,
> individual typing PEPs can be delegated just like any other PEP, but
> especially where syntax change proposals are made, I expect that the PSC
> will want keep final decision making.
>
> This means that any PEP which proposes syntactic changes to support typing
> features must also address the implications of those syntax changes for the
> general Python language.  PEP 646 (Variadic Generics) is a good example of
> this.  Early on we recognized that the proposed syntax change had
> implications for Python, and we asked the PEP authors to address those
> implications for Python, which they added:
>
> https://www.python.org/dev/peps/pep-0646/#grammar-changes
>
> This is the kind of analysis the PSC needs in order to weigh the cost and
> benefits of making such changes to Python.
>
> TL;RA - One Syntax to Rule Them All
>
> Cheers,
> -Barry
>
> > On Oct 7, 2021, at 09:41, S Pradeep Kumar  wrote:
> >
> > Hello all,
> >
> > Typing-sig has been discussing user-friendly syntax for the type used to
> represent callables. [1] Since this affects the Python language syntax, we
> wanted to get some high-level feedback from you before putting up a
> detailed PEP.
> >
> > TL;DR: We want to propose syntax for Callable types, e.g., `(int, str)
> -> bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are
> there concerns we need to keep in mind about such a syntax change? 2.
> Should we propose syntax for additional features in the same PEP or in a
> future PEP?
> >
> >
> > # Motivation
> >
> > Guido has pointed out that `Callable` is one of the most frequently used
> type forms but has the least readable syntax. For example, say we have a
> callback `formatter` that accepts a record and a list of permissions:
> >
> > ```python
> > def print_purchases(
> > user,
> > formatter,  # <-- callback
> > ):
> > <...>
> > output = formatter(record, permissions)
> > print(output)
> > ```
> >
> > To give it a type, we currently have to write:
> >
> > ```python
> > from typing import Callable
> >
> > def print_purchases(
> > user: User,
> > formatter: Callable[[PurchaseRecord, List[AuthPermission]],
> FormattedItem]  # Hard to read.
> > ) -> None:
> >
> > <...>
> > output = formatter(record, permissions)
> > print(output)
> > ```
> >
> > `Callable` can be hard to understand for new users since it doesn't
> resemble existing function signatures and there can be multiple square
> brackets. It also requires an import from `typing`, which is inconvenient.
> Around 40% of the time [2], users just give up on precisely typing the
> parameters and return type of their callbacks and just leave it as a blank
> Callable. In other cases, they don't add a type for their callback at all.
> Both mean that they lose the safety guarantees from typing and leave room
> for bugs.
> >
> > We believe that adding friendly syntax for Callable types will improve
> readability and encourage users to type their callbacks more precisely.
> Other modern, gradually-typed languages like TypeScript (JS), Hack (PHP),
> etc. have special syntax for Callable types.
> >
> > (As a precedent, PEP 604 recently added clean, user-friendly syntax for
> the widely-used `Union` type. Instead of importing `Union` and writing
> `expr: Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr |
> SubtractExpr`.)
> >
> >
> > # Proposal and Questions
> >
> > We have a two-part proposal and questions for you:
> >
> > 1. Syntax to replace Callable
> >
> > After a lot of discussion, there is strong consensus in typing-sig about
> adding syntax to replace Callable. So, the above example would be written
> as:
> > ```python
> > def print_purchases(
> > user: User,
> > formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem,
> > ) -> None:
> 

[Python-Dev] Re: Python multithreading without the GIL

[Sam Gross]

On Fri, Oct 8, 2021 at 12:55 PM Daniel Pope  wrote:

> I'm a novice C programmer, but I'm unsure about the safety of your
> thread-safe collections description.
>

The "list" class uses a slightly different strategy than "dict", which I
forgot about
when writing the design overview. List relies on the property that the
backing array
of a given list can only grow (never shrink) [1]. This is different from
upstream CPython.

Dict stores the capacity inside PyDictKeysObject (the backing array). The
capacity
never changes, so if you have a valid pointer to the PyDictKeysObject you
load the
correct capacity.

I've been meaning to change "list" to use the same strategy as "dict". I
think that would
simplify the overall design and let "list" shrink the backing array again.

[1]
https://github.com/colesbury/nogil/blob/fb6aabede5f7f1936a21c2f48ec7fcc0848d74bf/Objects/listobject.c#L46-L49
(
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/46TGB2MXWJ37VUQH3R5LW6BOGLIE3PGG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Larry Hastings]

On 10/7/21 8:52 PM, Sam Gross wrote:
I've been working on changes to CPython to allow it to run without the 
global interpreter lock.



Before anybody asks: Sam contacted me privately some time ago to pick my 
brain a little.  But honestly, Sam didn't need any help--he'd already 
taken the project further than I'd ever taken the Gilectomy.  I have 
every confidence in Sam and his work, and I'm excited he's revealed it 
to the world!



Best wishes,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CCGH6COYQGCAFZWD32ROUOHRSE4BUL3P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Guido van Rossum]

On Fri, Oct 8, 2021 at 3:36 PM Barry Warsaw  wrote:

> On Oct 8, 2021, at 13:02, Sergei Lebedev 
> wrote:
> >
> > I agree that type annotation syntax should be consistent with the rest
> of the language, but I'm curious why it has to be "standard Python syntax"?
> Could you elaborate a bit more on the reasoning behind that constraint?
>
>
> Hi Sergei.  I don’t mean anything more than just that there should be a
> single syntax for all of Python. I was just trying to describe “the bits of
> Python that aren’t type annotations”.
>

I interpret this as follows:

- Whatever we choose (say, "(int, int) -> int") should be allowed where
expressions are allowed, so that e.g.

foo = (int, int) -> int

assigns *something* at runtime -- it shouldn't be a syntax error and it
shouldn't raise an exception.

My preference would be for it to return something that's introspectable,
i.e. an object lets one recover the original info. That's what we did when
we introduced "A | B" as union syntax for types. (It's slightly different
because that was already valid syntax, but the principle that it must
return something introspectable is the same.) Union objects have an
`__args__` attribute that gives the underlying types, and a
`__parameters__` attribute giving any type variables.

Note that the implementation of this introspectable type should ultimately
be in C.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5X2XE52DDSCOFYVDO2IMBUKBQPU4HQUD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Barry Warsaw]

On Oct 8, 2021, at 13:02, Sergei Lebedev  wrote:
> 
> I agree that type annotation syntax should be consistent with the rest of the 
> language, but I'm curious why it has to be "standard Python syntax"? Could 
> you elaborate a bit more on the reasoning behind that constraint?


Hi Sergei.  I don’t mean anything more than just that there should be a single 
syntax for all of Python. I was just trying to describe “the bits of Python 
that aren’t type annotations”.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KGDCVTFBVYIP4AXECDVZMDP5MAAJVTTW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Barry Warsaw]

Thanks for thinking more deeply about how we can make type annotations for 
callables more user friendly.

The intersection between the syntax for general Python code and type 
annotations has been a topic of discussion within the Steering Council (on and 
off) for quite some time.  We intend to provide more clarity about our 
thinking, probably in the form of an informational PEP, around this topic, and 
my apologies because I haven’t had time to draft it yet.

But since this came up, I want to at least say that we are unanimous in our 
belief that the type annotation language must also be standard Python syntax.  
In other words, we are not in favor of letting type annotation syntax deviate 
from Python syntax.  As a effect of this, the PSC does not intend to issue a 
blanket delegation of type-focused PEPs.  Of course, individual typing PEPs can 
be delegated just like any other PEP, but especially where syntax change 
proposals are made, I expect that the PSC will want keep final decision making.

This means that any PEP which proposes syntactic changes to support typing 
features must also address the implications of those syntax changes for the 
general Python language.  PEP 646 (Variadic Generics) is a good example of 
this.  Early on we recognized that the proposed syntax change had implications 
for Python, and we asked the PEP authors to address those implications for 
Python, which they added:

https://www.python.org/dev/peps/pep-0646/#grammar-changes

This is the kind of analysis the PSC needs in order to weigh the cost and 
benefits of making such changes to Python.

TL;RA - One Syntax to Rule Them All

Cheers,
-Barry

> On Oct 7, 2021, at 09:41, S Pradeep Kumar  wrote:
> 
> Hello all,
> 
> Typing-sig has been discussing user-friendly syntax for the type used to 
> represent callables. [1] Since this affects the Python language syntax, we 
> wanted to get some high-level feedback from you before putting up a detailed 
> PEP.
> 
> TL;DR: We want to propose syntax for Callable types, e.g., `(int, str) -> 
> bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are there 
> concerns we need to keep in mind about such a syntax change? 2. Should we 
> propose syntax for additional features in the same PEP or in a future PEP?
> 
> 
> # Motivation
> 
> Guido has pointed out that `Callable` is one of the most frequently used type 
> forms but has the least readable syntax. For example, say we have a callback 
> `formatter` that accepts a record and a list of permissions:
> 
> ```python
> def print_purchases(
> user,
> formatter,  # <-- callback
> ):
> <...>
> output = formatter(record, permissions)
> print(output)
> ```
> 
> To give it a type, we currently have to write:
> 
> ```python
> from typing import Callable
> 
> def print_purchases(
> user: User,
> formatter: Callable[[PurchaseRecord, List[AuthPermission]], 
> FormattedItem]  # Hard to read.
> ) -> None:
> 
> <...>
> output = formatter(record, permissions)
> print(output)
> ```
> 
> `Callable` can be hard to understand for new users since it doesn't resemble 
> existing function signatures and there can be multiple square brackets. It 
> also requires an import from `typing`, which is inconvenient. Around 40% of 
> the time [2], users just give up on precisely typing the parameters and 
> return type of their callbacks and just leave it as a blank Callable. In 
> other cases, they don't add a type for their callback at all. Both mean that 
> they lose the safety guarantees from typing and leave room for bugs.
> 
> We believe that adding friendly syntax for Callable types will improve 
> readability and encourage users to type their callbacks more precisely. Other 
> modern, gradually-typed languages like TypeScript (JS), Hack (PHP), etc. have 
> special syntax for Callable types.
> 
> (As a precedent, PEP 604 recently added clean, user-friendly syntax for the 
> widely-used `Union` type. Instead of importing `Union` and writing `expr: 
> Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr | 
> SubtractExpr`.)
> 
> 
> # Proposal and Questions
> 
> We have a two-part proposal and questions for you:
> 
> 1. Syntax to replace Callable
> 
> After a lot of discussion, there is strong consensus in typing-sig about 
> adding syntax to replace Callable. So, the above example would be written as:
> ```python
> def print_purchases(
> user: User,
> formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem,
> ) -> None:
> 
> <...>
> output = formatter(record, permissions)
> print(output)
> ```
> This feels more natural and succinct.
> 
> Async callbacks currently need to be written as
> ```
> from typing import Callable
> 
> async_callback: Callable[[HttpRequest], Awaitable[HttpResponse]]
> ```
> 
> With the new syntax, they would be written as
> ```
> async_callback: async (HttpRequest) -> HttpResponse
> ```
> which again seems more natural. There is similar 

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2021-10-01 - 2021-10-08)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7374 (+33)
  closed 49772 (+38)
  total  57146 (+71)

Open issues with patches: 2920 


Issues opened (51)
==

#33125: Windows 10 ARM64 platform support
https://bugs.python.org/issue33125  reopened by steve.dower

#45163: Haiku build fix
https://bugs.python.org/issue45163  reopened by ned.deily

#45344: Have zipapp respect SOURCE_DATE_EPOCH
https://bugs.python.org/issue45344  opened by bign8

#45347: datetime subject to rounding?
https://bugs.python.org/issue45347  opened by piro

#45351: asyncio doc: List all sockets in TCP echo server using streams
https://bugs.python.org/issue45351  opened by olafvdspek

#45352: Move documentation for typed generic forms of standard collect
https://bugs.python.org/issue45352  opened by pxeger

#45353: sys.modules: dictionary changed size during iteration
https://bugs.python.org/issue45353  opened by idan57

#45355: Use sizeof(_Py_CODEUNIT) instead of literal 2 for the size of 
https://bugs.python.org/issue45355  opened by serhiy.storchaka

#45356: Calling `help` executes @classmethod @property decorated metho
https://bugs.python.org/issue45356  opened by randolf.scholz

#45357: Idle does not check user config for extention configuration
https://bugs.python.org/issue45357  opened by CoolCat467

#45358: Bogus cookie generated after invalid cookie attribute is input
https://bugs.python.org/issue45358  opened by greob

#45359: TopologicalSorter is not Generic at runtime (but is in typeshe
https://bugs.python.org/issue45359  opened by JacobHayes

#45361: Provide a more convenient way to set an exception as "active",
https://bugs.python.org/issue45361  opened by njs

#45363: Use instruction offsets in co_lnotab
https://bugs.python.org/issue45363  opened by serhiy.storchaka

#45364: Add more documentation for ModuleSpec.loader_state.
https://bugs.python.org/issue45364  opened by eric.snow

#45365: concurrent.futures.Future should be suitable for use outside o
https://bugs.python.org/issue45365  opened by xmorel

#45366: dataclass init=False field with default works but default_fact
https://bugs.python.org/issue45366  opened by simple_coder878

#45367: Specialize BINARY_MULTIPLY
https://bugs.python.org/issue45367  opened by Dennis Sweeney

#45369: Remove LibreSSL workarounds
https://bugs.python.org/issue45369  opened by ramikg

#45371: distutil's runtime_library_dir (rpath) option doesn't work wit
https://bugs.python.org/issue45371  opened by christian.heimes

#45373: ./configure --enable-optimizations should enable LTO
https://bugs.python.org/issue45373  opened by vstinner

#45374: sqlite3: Add configure option to set or auto-detect rpath to s
https://bugs.python.org/issue45374  opened by chaz6

#45376: Run Windows release docs build on regular CI
https://bugs.python.org/issue45376  opened by steve.dower

#45378: Can't find "map" with search on docs.python.org
https://bugs.python.org/issue45378  opened by gvanrossum

#45379: Improve errors related to frozen modules.
https://bugs.python.org/issue45379  opened by eric.snow

#45380: help() appears confused about the module of typing.Annotated
https://bugs.python.org/issue45380  opened by AlexWaygood

#45381: IDLE: "restart shell" while receiving output disables program
https://bugs.python.org/issue45381  opened by capsicumw

#45382: platform() is not able to detect windows 11
https://bugs.python.org/issue45382  opened by sahsariga111

#45383: PyType_FromSpec API fails to use metaclass of bases
https://bugs.python.org/issue45383  opened by seberg

#45384: Accept Final as indicating ClassVar for dataclass
https://bugs.python.org/issue45384  opened by GBeauregard

#45386: xmlrpc.client unimportable due to strfmt ValueError
https://bugs.python.org/issue45386  opened by rtobar2

#45387: GzipFile.write should be buffered
https://bugs.python.org/issue45387  opened by rhpvorderman

#45388: Use JUMP_FORWARD for all forward jumps.
https://bugs.python.org/issue45388  opened by Mark.Shannon

#45389: https://docs.python.org/3/objects.inv still points to 3.9
https://bugs.python.org/issue45389  opened by gaborjbernat

#45390: asyncio.Task doesn't propagate CancelledError() exception corr
https://bugs.python.org/issue45390  opened by pagliaricci.m

#45391: 3.10 objects.inv classifies UnionType as data
https://bugs.python.org/issue45391  opened by gaborjbernat

#45392: docstring of "type" could use an update
https://bugs.python.org/issue45392  opened by mark.dickinson

#45393: help() on operator precedence has confusing entries "await" "x
https://bugs.python.org/issue45393  opened by MFH

#45395: Frozen stdlib modules are discarded if custom frozen modules a
https://bugs.python.org/issue45395  opened by eric.snow

#45396: Custom frozen modules get ignored.
https://bugs.python.org/issue45396  opened by 

[Python-Dev] CPython is participating in hacktoberfest (with caveat)

[Mariatta]

Hello,

I'm happy to announce that CPython project is participating in
hacktoberfest.
I have just added the hacktoberfest topic to CPython's GitHub Repo (
https://github.com/python/cpython).

Caveat: if we end up receiving too many spammy, invalid, and low-quality
PRs which just adds burden to maintainers, we will opt-out of this
immediately. I will be watching incoming PRs for the next week to determine
whether we should opt-out.

If you're participating in hacktoberfest, your contribution to the repo
will count.
Instead of receiving a t-shirt, you can also choose to have a tree planted
on your behalf, which I personally recommend.

If you're a core dev, you can also participate and get rewarded by
reviewing and merging pull requests. If you spot spammy PRs, please add
either the "spam" or "invalid" label.

Another way to participate is by donating to participating open source
projects. You can donate to CPython development through GitHub Sponsors (
https://github.com/sponsors/python)

Hacktoberfest  is an annual event
organized by Digital Ocean, now in the 8th year, where aspiring
contributors are encouraged to contribute to open source projects.
I'm a member of the Hacktoberfest Advisory council
. If you have
concerns, feedback, and suggestions about this event, please do let me know
and I'll be sure to pass it along.

Thank you.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EXNW6XT4LNAVJYTB6TELVJEAWZWI5W6J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Daniel Pope]

On Fri, 8 Oct 2021 at 03:50, Sam Gross  wrote:
> My goal with the proof-of-concept is to demonstrate that removing the GIL is 
> feasible and worthwhile, and that the technical ideas of the project could 
> serve as a basis of such an effort.

I'm a novice C programmer, but I'm unsure about the safety of your
thread-safe collections description. You describe an algorithm for
lock-free read access to list items as

1. Load the version counter from the collection
2. Load the “backing array” from the collection
3. Load the address of the item (from the “backing array”)
4. Increment the reference count of the item, if it is non-zero
(otherwise retry)
5. Verify that the item still exists at the same location in the
collection (otherwise retry)
6. Verify that the version counter did not change (otherwise retry)
7. Return the address of the item

But you do the bounds check for the index before this, here[1]. If the
thread is suspended after this and before you read the address of the
backing array [2], the list could have been resized (shrunk), and the
backing array reallocated from a new memory block. So the pointer you
read at 3 could be from uninitialized memory that is beyond the size
of the array (or within the array but larger than the current number
of items). And then you write to it at 4 which is then a write into a
random memory location.

[1] 
https://github.com/colesbury/nogil/blob/fb6aabede5f7f1936a21c2f48ec7fcc0848d74bf/Objects/listobject.c#L137
[2] 
https://github.com/colesbury/nogil/blob/fb6aabede5f7f1936a21c2f48ec7fcc0848d74bf/Objects/listobject.c#L141
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6J6XFEACF2C6XPLZRVABUFFHJICUTZCS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Pablo Galindo Salgado]

>
> To speed-up function calls, the interpreter uses a linear, resizable stack
> to store function call frames, an idea taken from LuaJIT. The stack stores
> the interpreter registers (local variables + space for temporaries) plus
> some extra information per-function call. This avoids the need for
> allocating PyFrameObjects for each call. For compatibility, the
> PyFrameObject type still exists, but they are created lazily as-needed
> (such as for exception handling and for sys._getframe).

The optimized function calls have about an order of magnitude less overhead
> than the current CPython implementation.

The change also simplifies the use of deferred reference counting with the
> data that is stored per-call like the function object. The interpreter can
> usually avoid incrementing the reference count of the function object
> during a call. Like other objects on the stack, a borrowed reference to the
> function is indicated by setting the least-significant-bit.


Congrats Sam! This is incredible work! One quick question after reading the
design doc:

When you mean "an order of magnitude less overhead than the current CPython
implementation" do you mean compared with the main branch? We recently
implemented already almost everything is listed in this paragraph:

https://github.com/python/cpython/pull/27077

We also pack some extra similar optimizations in this other PR, including
stealing the frame arguments from python to python calls:

https://github.com/python/cpython/pull/28488

This could explain why the performance is closer to the current master
branch as you indicate:

 It gets about the same average performance as the “main” branch of CPython
> 3.11 as of early September 2021.


Cheers from cloudy London,
Pablo Galindo Salgado

On Fri, 8 Oct 2021 at 03:49, Sam Gross  wrote:

> Hi,
>
> I've been working on changes to CPython to allow it to run without the
> global interpreter lock. I'd like to share a working proof-of-concept that
> can run without the GIL. The proof-of-concept involves substantial changes
> to CPython internals, but relatively few changes to the C-API. It is
> compatible with many C extensions: extensions must be rebuilt, but usually
> require small or no modifications to source code. I've built compatible
> versions of packages from the scientific Python ecosystem, and they are
> installable through the bundled "pip".
>
> Source code:
> https://github.com/colesbury/nogil
>
> Design overview:
>
> https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit
>
> My goal with the proof-of-concept is to demonstrate that removing the GIL
> is feasible and worthwhile, and that the technical ideas of the project
> could serve as a basis of such an effort.
>
> I'd like to start a discussion about these ideas and gauge the community's
> interest in this approach to removing the GIL.
>
> Regards,
> Sam Gross
> colesb...@gmail.com / sgr...@fb.com
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ABR2L6BENNA6UPSPKV474HCS4LWT26GY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K4SHFGDVAZUMAAKX5ZANAQIVYBPUSLEI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Nathaniel Smith]

On Thu, Oct 7, 2021 at 7:54 PM Sam Gross  wrote:
> Design overview:
> https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit

Whoa, this is impressive work.

I notice the fb.com address -- is this a personal project or something
facebook is working on? what's the relationship to Cinder, if any?

Regarding the tricky lock-free dict/list reads: I guess the more
straightforward approach would be to use a plain ol' mutex that's
optimized for this kind of fine-grained per-object lock with short
critical sections and minimal contention, like WTF::Lock. Did you try
alternatives like that? If so, I assume they didn't work well -- can
you give more details?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XAZYWRYXKIVUSMRSMFAETKQDLGL27L7X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Guido van Rossum]

Pradeep,

Thanks for all the clarifications. I think it's time that you put together
a formal grammar. That should also make clear that "(,) -> int" is
disallowed while "(int,) -> int" is allowed.

--Guido

On Fri, Oct 8, 2021 at 8:50 AM S Pradeep Kumar  wrote:

> Thanks for the responses!
>
> 1. Chris Angelico:
>
> > it looks like the parentheses are mandatory, and they are what define it?
>
> Yes, we always expect parens. Like you said, this keeps things consistent,
> resembles the function signature, and avoids ambiguities. It also won't
> look like Haskell function types `int -> str -> bool` (which can be
> confusing for non-functional programmers).
>
> This was pretty explicitly rejected when we polled people at the Typing
> Summit.
>
> > Which of these will be valid?
>
> (int) -> int
> (int, int) -> int
>
> (int,) -> int should also be fine, but I don't have strong opinions here.
> e.g., what about (,) -> int? We'll make sure to address this edge case
> explicitly, thanks.
>
> > There are parallel proposals to support a lambda syntax like "x =>
> x.spam" as equivalent to "lambda x: x.spam", so I'm curious what the rules
> would be for the two types of arrows, and whether they'd be consistent with
> each other.
>
> Yeah, this had come up in our discussions. Like in other languages (such
> as Hack), we believe that lambdas should have a different arrow type to
> disambiguate the two. So, lambdas could be (x, y) => x + y.
>
> 2. Piotr Duda
>
> > Did you considered full def-signature with optional argument names, so
> the common cases would look like
> >
> > 
> > (:PurchaseRecord, :List[AuthPermission]) -> FormattedItem
> > 
> > Bare name signatures like '(record) -> FormattedItem' could be
> disallowed to prevent bugs.
>
> The additional-features proposal in (2) actually does have optional
> parameter names. There are two variants:
>
> (a) the "hybrid" proposal, which will let you write (int, str) -> bool,
> but also hybrids like (int, y: str) -> bool; (x: int, y: str) -> bool;
> (int, str=...) -> bool.
>
> (b) the "xor" proposal, which will let you use the shorthand syntax (int,
> str) -> bool for the most frequently-used cases and the full def-signature
> for all other advanced cases using familiar syntax: (x: int, y: str=...) ->
> bool.
>
> We haven't reached a strong consensus on the above two yet (flexibility vs
> familiarity). We decided to get some thoughts here about whether we need to
> propose additional-features at all (2) in our initial PEP.
>
> 3. Patrick Reader
>
> > How would this work for a function that returns a function? Would you
> just put it on the def line like this?
>
> We will allow using the callable syntax in the return type without parens:
>
> def foo() -> (int) -> str:
> return int_to_str
>
> The reason for not making that a syntax error is that it would be pretty
> bad UX. Users would see the callable syntax being used without parens
> everywhere: f: (int) -> str, but when they try to use that as the return
> type, they would get a confusing syntax error. So, given how frequently
> functions return functions, we decided to not require parens.
>
> People can specify parens in their coding guidelines (and auto-formatters)
> if they find parens more readable.
>
> 4. Paul Moore
>
> +1 to what Jelle said.
>
> > That's one of the differences between the proposals discussed here. The
> basic proposal Pradeep pointed out would not support named arguments, the
> more complete syntax favored by Guido (and also by me; 2(a) in Pradeep's
> email) would support it.
> >
> > Pradeep has done some empirical analysis that shows that in practice it
> is not very common to use such types, though.
>
> Yeah, that's basically Question 2 - whether to just replace Callable in
> the initial PEP (option 1) or to specify a more complete syntax from the
> beginning (option 2).
>
> 5. Serhiy Storchaka
>
> > How could you replace Callable[..., int] and Callable[Concatenate[str,
> P], int] ?
>
> To represent a Callable that accepts arbitrary arguments:
>
> (...) -> int
>
> To represent a Callable that accepts ParamSpec (for decorators, etc.),
> we're leaning towards the below:
>
> (**P) -> int
>
> To represent your Concatenate example:
>
> (str, **P) -> int
>
> We would no longer need the Concatenate operator. We can naturally just
> add types up front.
> --
> S Pradeep Kumar
>
> On Fri, Oct 8, 2021 at 8:48 AM Guido van Rossum  wrote:
>
>> On Fri, Oct 8, 2021 at 8:31 AM Jelle Zijlstra 
>> wrote:
>>
>>> El vie, 8 oct 2021 a las 0:54, Paul Moore ()
>>> escribió:
>>>
 Also, note that I automatically used a type of int->int up there. As
 someone else asked, is that form allowed, or is it required to be
 (int)->int? In my view, if we require the latter, I expect there will
 be a *lot* of people making that mistake and having to correct it.

>>>
>>> That's not something we discussed before. I'd be OK with allowing this
>>> unless it makes the grammar too ambiguous.
>>>
>>
>> 

[Python-Dev] Re: RFC on Callable Type Syntax

[Paul Moore]

On Fri, 8 Oct 2021 at 16:29, Jelle Zijlstra  wrote:
> That's one of the differences between the proposals discussed here. The basic 
> proposal Pradeep pointed out would not support named arguments, the more 
> complete syntax favored by Guido (and also by me; 2(a) in Pradeep's email) 
> would support it.
>
> Pradeep has done some empirical analysis that shows that in practice it is 
> not very common to use such types, though.

I believe resolvelib (used by pip) makes a point of always calling
callbacks with named arguments, to ensure compatibility. I thought PEP
517 hooks also required callers to use keywords, but when I checked
the PEP, apparently not. I think with hindsight that we should have,
though.

But yes, I can imagine that simpler callbacks (like key= in sort)
would be more common, and would be fine with positional arguments.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EUXX7JBUXTNYZWKJ2XG7BXJST65DHAVI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Guido van Rossum]

On Fri, Oct 8, 2021 at 8:31 AM Jelle Zijlstra 
wrote:

> El vie, 8 oct 2021 a las 0:54, Paul Moore ()
> escribió:
>
>> Also, note that I automatically used a type of int->int up there. As
>> someone else asked, is that form allowed, or is it required to be
>> (int)->int? In my view, if we require the latter, I expect there will
>> be a *lot* of people making that mistake and having to correct it.
>>
>
> That's not something we discussed before. I'd be OK with allowing this
> unless it makes the grammar too ambiguous.
>

Even if it didn't make the grammar ambiguous (and I think it doesn't, as
long as the argument type isn't a union or another callable type), I'd be
against this.

An argument list is not a tuple, and we don't allow omitting the
parentheses in other call-related situations: you can't write "def f x:
return x+1" and you can't write "f 42". Allowing the omission of the
parentheses here would be inconsistent (even if some other languages allow
it).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UI2JGJ56RIUV65ULATP3JP4FJGUQSZAI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Chris Jerdonek]

On Fri, Oct 8, 2021 at 8:11 AM Guido van Rossum  wrote:

> To be clear, Sam’s basic approach is a bit slower for single-threaded
> code, and he admits that.
>

Is it also slower even when running with PYTHONGIL=1? If it could be made
the same speed for single-threaded code when running in GIL-enabled mode,
that might be an easier intermediate target while still adding value.

—Chris


But to sweeten the pot he has also applied a bunch of unrelated speedups
> that make it faster in general, so that overall it’s always a win. But
> presumably we could upstream the latter easily, separately from the
> GIL-freeing part.
>
> On Fri, Oct 8, 2021 at 07:42 Łukasz Langa  wrote:
>
>>
>> > On 8 Oct 2021, at 10:13, Steven D'Aprano  wrote:
>> >
>> > Hi Sam,
>> >
>> > On Thu, Oct 07, 2021 at 03:52:56PM -0400, Sam Gross wrote:
>> >
>> >> I've been working on changes to CPython to allow it to run without the
>> >> global interpreter lock. I'd like to share a working proof-of-concept
>> that
>> >> can run without the GIL.
>> >
>> > Getting Python to run without the GIL has never been a major problem for
>> > CPython (and of course some other Python interpreters don't have a GIL
>> > at all).
>>
>> On the first page of Sam's design overview he references Gilectomy by
>> name.
>>
>> > Single threaded code is still, and always will be, an important part of
>> > Python's ecosystem. A lot of people would be annoyed if the cost of
>> > speeding up heavily threaded Python by a small percentage would be to
>> > slow down single-threaded Python by a large percentage.
>>
>> Quoting that same design document, Sam writes: "The new interpreter
>> (together with the GIL changes) is about 10% faster than CPython 3.9
>> on the single-threaded pyperformance benchmarks."
>>
>> - Ł
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/JO7OQCHZKIFNKSXTTXT2JBCF5H47M7OO/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> --
> --Guido (mobile)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XQOOGKH5PIFBHJRK7W2LMX32DIGIH4KX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/STEMG6WAORYZ2WVMXZZPYSQVEUNNXCSW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Jelle Zijlstra]

El vie, 8 oct 2021 a las 0:54, Paul Moore () escribió:

> On Fri, 8 Oct 2021 at 03:45, S Pradeep Kumar  wrote:
>
> > Typing-sig has been discussing user-friendly syntax for the type used to
> represent callables. [1] Since this affects the Python language syntax, we
> wanted to get some high-level feedback from you before putting up a
> detailed PEP.
>
> Disclaimer: I personally find that use of complex type annotations
> like Callable make code unreadable for the *human* reader, as compared
> to out of line information like a clearly-written comment or
> documentation. Therefore I'd probably be extremely reluctant to use
> this new syntax in real-life projects, regardless of the form that it
> takes. However, I do have some experience working on projects that use
> type annotations like this for callbacks, so I will answer in spite of
> my personal reservations.
>

The point of this change is to make Callable annotations more readable:
(int) -> int is a lot less dense than Callable[[int], int]. We hope that
the new syntax will make it easier to use more complex annotations. Of
course, if you think the annotations are still too complex, that's your
choice.


>
> > **Question 1**: Are there concerns we should keep in mind about such a
> syntax proposal?
>
> In my experience, a common pattern for functions that take non-trivial
> callbacks is to call them with named arguments rather than positional,
> in order to preserve future compatibility. So I would like callback
> annotations to be able to express the type "a function returning X,
> taking arguments `arg1` of type Y and `arg2` of type Z". The lack of
> keyword argument support in this proposal makes it unusable for APIs
> like this.
>
That's one of the differences between the proposals discussed here. The
basic proposal Pradeep pointed out would not support named arguments, the
more complete syntax favored by Guido (and also by me; 2(a) in Pradeep's
email) would support it.

Pradeep has done some empirical analysis that shows that in practice it is
not very common to use such types, though.


>
> Conversely, for callbacks with single arguments like f(cb: int->int)
> -> int, in my experience a lot of uses are lambda expressions, so
> something like f(lambda x: x+1). I would be a strong -1 on having to
> add types to lambda expressions (they are ugly enough already) so how
> does this interact with the annotation? Are type checkers able to
> correctly infer the type of lambda x: x+1, or would the whole
> expression end up being untyped? Similarly with functools.partial -
> would that correctly match the type? If common uses of callbacks end
> up being untyped in practice, that drastically reduces the value of
> the new syntax. (Note that this point is equally applicable to
> Callable - so I guess it's not so much about the new syntax as it is
> about whether annotating callbacks is sufficiently useful to be
> *worth* improving).
>

Type checkers should be able to infer this type. You would not need to
write any explicit type when you call f(). However, the definition of `f`
would look something like `def f(callback: (int) -> int) -> None: ...`.
Currently, you'd write that as `def f(callback: Callable[[int], int]) ->
None: ...`.


>
> Also, note that I automatically used a type of int->int up there. As
> someone else asked, is that form allowed, or is it required to be
> (int)->int? In my view, if we require the latter, I expect there will
> be a *lot* of people making that mistake and having to correct it.
>
That's not something we discussed before. I'd be OK with allowing this
unless it makes the grammar too ambiguous.


>
> > **Question 2**: Do you have preferences either way? Do we propose (1)
> alone or (1) + (2)?
>
> See above. My experience is that using named arguments for callbacks
> is a useful approach, and should probably be encouraged rather than
> made difficult, so I'd say that named argument support is important.
> Having said that, I think that making simple types like int->int or
> (int, int)->int more awkward to write would be a disadvantage - they
> are used a *lot* in things like the `key` argument of `sort`. So I'm
> -1 on things like (for example) requiring a colon as in (:int, :int)
> -> int.
>
> But to reiterate, I'm probably not likely to ever be a significant
> consumer of this feature, so consider the above with that in mind ;-)
>
> Paul
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ANGLWMP4X7AIBFI5SII7RBCOHRIZZ7TH/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org

[Python-Dev] Re: Python multithreading without the GIL

[Guido van Rossum]

To be clear, Sam’s basic approach is a bit slower for single-threaded code,
and he admits that. But to sweeten the pot he has also applied a bunch of
unrelated speedups that make it faster in general, so that overall it’s
always a win. But presumably we could upstream the latter easily,
separately from the GIL-freeing part.

On Fri, Oct 8, 2021 at 07:42 Łukasz Langa  wrote:

>
> > On 8 Oct 2021, at 10:13, Steven D'Aprano  wrote:
> >
> > Hi Sam,
> >
> > On Thu, Oct 07, 2021 at 03:52:56PM -0400, Sam Gross wrote:
> >
> >> I've been working on changes to CPython to allow it to run without the
> >> global interpreter lock. I'd like to share a working proof-of-concept
> that
> >> can run without the GIL.
> >
> > Getting Python to run without the GIL has never been a major problem for
> > CPython (and of course some other Python interpreters don't have a GIL
> > at all).
>
> On the first page of Sam's design overview he references Gilectomy by name.
>
> > Single threaded code is still, and always will be, an important part of
> > Python's ecosystem. A lot of people would be annoyed if the cost of
> > speeding up heavily threaded Python by a small percentage would be to
> > slow down single-threaded Python by a large percentage.
>
> Quoting that same design document, Sam writes: "The new interpreter
> (together with the GIL changes) is about 10% faster than CPython 3.9
> on the single-threaded pyperformance benchmarks."
>
> - Ł
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/JO7OQCHZKIFNKSXTTXT2JBCF5H47M7OO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XQOOGKH5PIFBHJRK7W2LMX32DIGIH4KX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Łukasz Langa]

> On 8 Oct 2021, at 10:13, Steven D'Aprano  wrote:
> 
> Hi Sam,
> 
> On Thu, Oct 07, 2021 at 03:52:56PM -0400, Sam Gross wrote:
> 
>> I've been working on changes to CPython to allow it to run without the
>> global interpreter lock. I'd like to share a working proof-of-concept that
>> can run without the GIL.
> 
> Getting Python to run without the GIL has never been a major problem for
> CPython (and of course some other Python interpreters don't have a GIL
> at all).

On the first page of Sam's design overview he references Gilectomy by name.

> Single threaded code is still, and always will be, an important part of
> Python's ecosystem. A lot of people would be annoyed if the cost of
> speeding up heavily threaded Python by a small percentage would be to
> slow down single-threaded Python by a large percentage.

Quoting that same design document, Sam writes: "The new interpreter
(together with the GIL changes) is about 10% faster than CPython 3.9
on the single-threaded pyperformance benchmarks."

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JO7OQCHZKIFNKSXTTXT2JBCF5H47M7OO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] svn.python.org require an HTTP authentication: r77062 links at bugs.python.org

[Victor Stinner]

Hi,

I'm digging into the Python bug tracker history and I found links to
Subversion commits:
"Fixed in r77062 (trunk), r77063 (py3k)."
https://bugs.python.org/issue1811#msg96910

Roundup adds links which are redirected:

* https://hg.python.org/lookup/r77062 ->
https://svn.python.org/view?view=revision=77062
* https://hg.python.org/lookup/r77063 ->
https://svn.python.org/view?view=revision=77063

Problem: the svn.python.org links require me to log in with an
username and password, and I don't know how to log in. If I cancel the
prompt, it fails with: an HTTP Error 401 "Unauthorized".

Who manages this Subversion server? Would it be possible to remove
this authentication?

I know that the Misc/svnmap.txt file in Python contains a mapping from
Subversion to Mercurial commits, but Python now uses Git, so it's not
convenient. Maybe a new file mapping Subversion to Git commits should
be created? Or the https://hg.python.org/lookup/ service should
redirect to Mercurial commits at least.

Using this map, I was able to manually retrieve the r77062 commit:
https://hg.python.org/cpython/rev/c4e60988834df0fd437ae3422da0cab4acd6fce0

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UR5SKKQ3XJTP7Q2IBMMZTNICZ7BDOKVZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Patrick Reader]

How would this work for a function that returns a function? Would you just put 
it on the def line like this?

def foo() -> (int) -> str:

    return some_function_from_int_to_str

Or would it have to be:

def foo() -> ((int) -> str): ...

The former is a little confusing to read - at first glance, it looks like it 
might return an int; the latter is ugly.

I'd prefer a (soft?) keyword, like `(func (int) -> str)`. That keyword could be 
`def` or `lambda`, but those look like they might preclude custom callables. 
And just using `callable` seems like little improvement...

On 07/10/2021 17:41, S Pradeep Kumar wrote:
> Hello all,
>
> Typing-sig has been discussing user-friendly syntax for the type used to 
> represent callables. [1] Since this affects the Python language syntax, we 
> wanted to get some high-level feedback from you before putting up a detailed 
> PEP.
>
> TL;DR: We want to propose syntax for Callable types, e.g., `(int, str) -> 
> bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are there 
> concerns we need to keep in mind about such a syntax change? 2. Should we 
> propose syntax for additional features in the same PEP or in a future PEP?
>
>
> # Motivation
>
> Guido has pointed out that `Callable` is one of the most frequently used type 
> forms but has the least readable syntax. For example, say we have a callback 
> `formatter` that accepts a record and a list of permissions:
>
> ```python
> def print_purchases(
>     user,
>     formatter,  # <-- callback
> ):
>     <...>
>     output = formatter(record, permissions)
>     print(output)
> ```
>
> To give it a type, we currently have to write:
>
> ```python
> from typing import Callable
>
> def print_purchases(
>     user: User,
>     formatter: Callable[[PurchaseRecord, List[AuthPermission]], 
> FormattedItem]  # Hard to read.
> ) -> None:
>
>     <...>
>     output = formatter(record, permissions)
>     print(output)
> ```
>
> `Callable` can be hard to understand for new users since it doesn't resemble 
> existing function signatures and there can be multiple square brackets. It 
> also requires an import from `typing`, which is inconvenient. Around 40% of 
> the time [2], users just give up on precisely typing the parameters and 
> return type of their callbacks and just leave it as a blank Callable. In 
> other cases, they don't add a type for their callback at all. Both mean that 
> they lose the safety guarantees from typing and leave room for bugs.
>
> We believe that adding friendly syntax for Callable types will improve 
> readability and encourage users to type their callbacks more precisely. Other 
> modern, gradually-typed languages like TypeScript (JS), Hack (PHP), etc. have 
> special syntax for Callable types.
>
> (As a precedent, PEP 604 recently added clean, user-friendly syntax for the 
> widely-used `Union` type. Instead of importing `Union` and writing `expr: 
> Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr | 
> SubtractExpr`.)
>
>
> # Proposal and Questions
>
> We have a two-part proposal and questions for you:
>
> 1. Syntax to replace Callable
>
> After a lot of discussion, there is strong consensus in typing-sig about 
> adding syntax to replace Callable. So, the above example would be written as:
> ```python
> def print_purchases(
>     user: User,
>     formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem,
> ) -> None:
>
>     <...>
>     output = formatter(record, permissions)
>     print(output)
> ```
> This feels more natural and succinct.
>
> Async callbacks currently need to be written as
> ```
> from typing import Callable
>
> async_callback: Callable[[HttpRequest], Awaitable[HttpResponse]]
> ```
>
> With the new syntax, they would be written as
> ```
> async_callback: async (HttpRequest) -> HttpResponse
> ```
> which again seems more natural. There is similar syntax for the type of 
> decorators that pass on *args and **kwargs to the decorated function.
>
> Note that we considered and rejected using a full def-signature syntax like
> 
> (record: PurchaseRecord, permissions: List[AuthPermission], /) -> 
> FormattedItem
> 
> because it would be more verbose for common cases and could lead to subtle 
> bugs; more details in [3].
>
> The Callable type is also usable as an expression, like in type aliases 
> `IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls.
>
> **Question 1**: Are there concerns we should keep in mind about such a syntax 
> proposal?
>
>
>
> 2. Syntax for callback types beyond Callable
>
> `Callable` can't express the type of all possible callbacks. For example, it 
> doesn't support callbacks where some parameters have default values: 
> `formatter(record)` (the user didn't pass in `permissions`). It *is* possible 
> to express these advanced cases using Callback Protocols (PEP 544) [4] but it 
> gets verbose.
>
> There are two schools of thought on typing-sig on adding more syntax on top 
> of (1):
>
> (a) Some, 

[Python-Dev] Re: RFC on Callable Type Syntax

[Serhiy Storchaka]

07.10.21 19:41, S Pradeep Kumar пише:
> 1. Syntax to replace Callable
> 
> After a lot of discussion, there is strong consensus in typing-sig about
> adding syntax to replace Callable. So, the above example would be
> written as:
> ```python
> def print_purchases(
>     user: User,
>     formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem,
> ) -> None:
> 
>     <...>
>     output = formatter(record, permissions)
>     print(output)
> ```

How could you replace Callable[..., int] and Callable[Concatenate[str,
P], int] ?

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BJHVYIMVXYRGRTR77TNX77HEQKPZ2A4Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Steven D'Aprano]

Hi Sam,

On Thu, Oct 07, 2021 at 03:52:56PM -0400, Sam Gross wrote:

> I've been working on changes to CPython to allow it to run without the
> global interpreter lock. I'd like to share a working proof-of-concept that
> can run without the GIL.

Getting Python to run without the GIL has never been a major problem for 
CPython (and of course some other Python interpreters don't have a GIL 
at all). I think the first attempt was in 1999, a mere handful of years 
after Python was released.

https://www.artima.com/weblogs/viewpost.jsp?thread=214235

The problem has been removing the GIL without seriously degrading 
performance. How does your GIL-less CPython fork perform? Especially for 
single-threaded code.

Have you been following progress of the GILectomy?

https://pythoncapi.readthedocs.io/gilectomy.html

Single threaded code is still, and always will be, an important part of 
Python's ecosystem. A lot of people would be annoyed if the cost of 
speeding up heavily threaded Python by a small percentage would be to 
slow down single-threaded Python by a large percentage.


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CFPTMVFVPYHCGUPXYVBUYZBPLEVLSFIM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Paul Moore]

On Fri, 8 Oct 2021 at 03:45, S Pradeep Kumar  wrote:

> Typing-sig has been discussing user-friendly syntax for the type used to 
> represent callables. [1] Since this affects the Python language syntax, we 
> wanted to get some high-level feedback from you before putting up a detailed 
> PEP.

Disclaimer: I personally find that use of complex type annotations
like Callable make code unreadable for the *human* reader, as compared
to out of line information like a clearly-written comment or
documentation. Therefore I'd probably be extremely reluctant to use
this new syntax in real-life projects, regardless of the form that it
takes. However, I do have some experience working on projects that use
type annotations like this for callbacks, so I will answer in spite of
my personal reservations.

> **Question 1**: Are there concerns we should keep in mind about such a syntax 
> proposal?

In my experience, a common pattern for functions that take non-trivial
callbacks is to call them with named arguments rather than positional,
in order to preserve future compatibility. So I would like callback
annotations to be able to express the type "a function returning X,
taking arguments `arg1` of type Y and `arg2` of type Z". The lack of
keyword argument support in this proposal makes it unusable for APIs
like this.

Conversely, for callbacks with single arguments like f(cb: int->int)
-> int, in my experience a lot of uses are lambda expressions, so
something like f(lambda x: x+1). I would be a strong -1 on having to
add types to lambda expressions (they are ugly enough already) so how
does this interact with the annotation? Are type checkers able to
correctly infer the type of lambda x: x+1, or would the whole
expression end up being untyped? Similarly with functools.partial -
would that correctly match the type? If common uses of callbacks end
up being untyped in practice, that drastically reduces the value of
the new syntax. (Note that this point is equally applicable to
Callable - so I guess it's not so much about the new syntax as it is
about whether annotating callbacks is sufficiently useful to be
*worth* improving).

Also, note that I automatically used a type of int->int up there. As
someone else asked, is that form allowed, or is it required to be
(int)->int? In my view, if we require the latter, I expect there will
be a *lot* of people making that mistake and having to correct it.

> **Question 2**: Do you have preferences either way? Do we propose (1) alone 
> or (1) + (2)?

See above. My experience is that using named arguments for callbacks
is a useful approach, and should probably be encouraged rather than
made difficult, so I'd say that named argument support is important.
Having said that, I think that making simple types like int->int or
(int, int)->int more awkward to write would be a disadvantage - they
are used a *lot* in things like the `key` argument of `sort`. So I'm
-1 on things like (for example) requiring a colon as in (:int, :int)
-> int.

But to reiterate, I'm probably not likely to ever be a significant
consumer of this feature, so consider the above with that in mind ;-)

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ANGLWMP4X7AIBFI5SII7RBCOHRIZZ7TH/
Code of Conduct: http://python.org/psf/codeofconduct/