[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Barry Warsaw
Thank you Sam, this additional detail really helps me understand your proposal. -Barry > On Oct 11, 2021, at 12:06, Sam Gross wrote: > > I’m unclear what is actually retried. You use this note throughout the > document, so I think it would help to clarify exactly what is retried and why >

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread Erik Demaine
My apologies! Must be an issue with my email client rendering. Erik On Mon, 11 Oct 2021, Guido van Rossum wrote: On Mon, Oct 11, 2021 at 3:22 PM Erik Demaine wrote: On Mon, 11 Oct 2021, Guido van Rossum wrote: No, I didn't write that, Lukasz did. > I always found the

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread Guido van Rossum
On Mon, Oct 11, 2021 at 3:22 PM Erik Demaine wrote: > On Mon, 11 Oct 2021, Guido van Rossum wrote: > No, I didn't write that, Lukasz did. > I always found the following more obvious: > > > > def data_to_table(d: Iterable[Mapping[str, float]], *, sort: bool = > False, reversed: bool = False) ->

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread Erik Demaine
On Mon, 11 Oct 2021, Guido van Rossum wrote: I always found the following more obvious: def data_to_table(d: Iterable[Mapping[str, float]], *, sort: bool = False, reversed: bool = False) -> Table:     ... @dataclass class Stream:     converter: data_to_table | None     def

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-11 Thread Facundo Batista
El lun, 11 de oct. de 2021 a la(s) 06:50, Simon Cross (hodgestar+python...@gmail.com) escribió: > Multiprocessing sort of added support for this via multiprocessing.Array -- > see >

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-11 Thread Facundo Batista
El dom, 10 de oct. de 2021 a la(s) 15:20, Gregory P. Smith (g...@krypto.org) escribió: >> Is that because my particular case is very uncommon? Or maybe we *do* >> want this but we don't have it yet? Or do we already have a better way >> of doing this? >> >> Thanks! >> >> [0]

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Sam Gross
I've updated the linked gists with the results from interpreters compiled with PGO, so the numbers have slightly changed. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread gohanpra
1. Barry Warsaw > This means that any PEP which proposes syntactic changes to support typing > features must also address the implications of those syntax changes for the > general Python language. PEP 646 (Variadic Generics) is a good example of > this. Early on we recognized that the

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Sam Gross
On Mon, Oct 11, 2021 at 7:04 AM Antoine Pitrou wrote: > It's crude, but you can take a look at `ccbench` in the Tools directory. > Thanks, I wasn't familiar with this. The ccbench results look pretty good: about 18.1x speed-up on "pi calculation" and 19.8x speed-up on "regular expression" with

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread Guido van Rossum
On Mon, Oct 11, 2021 at 1:52 AM Łukasz Langa wrote: > > On 7 Oct 2021, at 18:41, S Pradeep Kumar wrote: > > Note that we considered and rejected using a full def-signature syntax > like > > (record: PurchaseRecord, permissions: List[AuthPermission], /) -> > FormattedItem > > because

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread Jim J. Jewett
It isn't clear to me that reusing the inspect.Signature object was even considered. If it was rejected as too complicated for most signatures, please put that in the PEP. My own opinion, admittedly not as a typing fan, is that if you need enough details to want more than Callable, then you

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Thomas Grainger
I have a PR to remove this FAQ entry: https://github.com/python/cpython/pull/28886 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread jack . jansen
If I can make a wild suggestion: why not create a little language for type specifications? If you look at other programming languages you’ll see that the “type definition sub-language” is often completely different from the “execution sub-language”, with only some symbols in common and used in

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Pablo Galindo Salgado
As far as I understand we should get a smaller improvement on single thread because some of the optimizations listed in this work are partially or totally implemented. This is excluding any non linear behaviour between the different optimizations of course, and assuming that both versions yield

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Sam Gross
On Mon, Oct 11, 2021 at 12:58 PM Thomas Grainger wrote: > Is D1.update(D2) still atomic with this implementation? > https://docs.python.org/3.11/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe > No. For example, another thread reading from the dict concurrently may observe

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Abdur-Rahmaan Janhangeer
When you mean "an order of magnitude less overhead than the current CPython implementation" do you mean compared with the main branch? We recently implemented already almost everything is listed in this paragraph: https://github.com/python/cpython/pull/27077 We also pack some extra similar

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Sam Gross
> > I’m unclear what is actually retried. You use this note throughout the > document, so I think it would help to clarify exactly what is retried and > why that solves the particular problem. I’m confused because, is it the > refcount increment that’s retried or the entire sequence of steps

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Ronald Oussoren via Python-Dev
> On 11 Oct 2021, at 18:58, Thomas Grainger wrote: > > Is D1.update(D2) still atomic with this implementation? > https://docs.python.org/3.11/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe > >

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread Patrick Reader
On 11/10/2021 16:45, Steven Troxler wrote: > https://docs.google.com/document/d/1oCRBAAPs3efTYoIRSUwWLtOr1AUVqc8OCEYn4vKYjGI/edit?usp=sharing I think ParamSpecs should not have any special syntax. `**P` looks way too much like kwargs, and any other syntax would be totally new precedent. I'd

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread Patrick Reader
On 11/10/2021 17:03, Steven Troxler wrote: > Personally, I think we should both adopt function-as-a-type and shorthand > syntax, and reject both the XOR and hybrid syntax because function-as-a-type > is convenient for expressing the same things (named, default-value, and > variadic args). +1

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Sam Gross
On Fri, Oct 8, 2021 at 11:35 AM Chris Jerdonek wrote: > Is it also slower even when running with PYTHONGIL=1? If it could be made > the same speed for single-threaded code when running in GIL-enabled mode, > that might be an easier intermediate target while still adding value. > Running with

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread Steven Troxler
> Is this also why re-using an actual callable at a type was rejected? The idea to use a function as a type (which Łukasz proposed at the PyCon typing summit) is still under consideration. Our thought was that it would probably be a separate PEP from a syntax change. Personally, I think we

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Thomas Grainger
Is D1.update(D2) still atomic with this implementation? https://docs.python.org/3.11/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe On Mon, 11 Oct 2021, 17:54 Sam Gross, wrote: > On Fri, Oct 8, 2021 at 12:04 PM Nathaniel Smith wrote: > >> I notice the fb.com address -- is

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread Steven Troxler
We drew up a draft syntax to clarify the three styles that have been discussed in typing-sig: - shorthand syntax, which supports exactly what Callable currently supports - XOR syntax, which would allow a def-like style in addition to shorthand - hybrid syntax, which extends shorthand to support

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Sam Gross
On Fri, Oct 8, 2021 at 12:04 PM Nathaniel Smith wrote: > I notice the fb.com address -- is this a personal project or something > facebook is working on? what's the relationship to Cinder, if any? > It is a Facebook project, at least in the important sense that I work on it as an employee at

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-11 Thread Simon Cross
On Sun, Oct 10, 2021 at 4:23 PM Facundo Batista wrote: > > struct.pack_into(format, buffer, offset, v1, v2, ...) I've encountered this wart with pack and pack_into too. The current interface makes sense when if v1, v2 are a small number of items from a data record, but it becomes a bit

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-11 Thread Antoine Pitrou
On Sun, 10 Oct 2021 11:19:44 -0300 Facundo Batista wrote: > Hello everyone! > > I need to pack a long list of numbers into shared memory, so I thought > about using `struct.pack_into`. > > Its signature is > > struct.pack_into(format, buffer, offset, v1, v2, ...) > > I have a long list of

[Python-Dev] Re: Python multithreading without the GIL

2021-10-11 Thread Antoine Pitrou
On Thu, 7 Oct 2021 15:52:56 -0400 Sam Gross wrote: > Hi, > > I've been working on changes to CPython to allow it to run without the > global interpreter lock. I'd like to share a working proof-of-concept that > can run without the GIL. The proof-of-concept involves substantial changes > to

[Python-Dev] Re: Packing a long list of numbers into memory

2021-10-11 Thread Serhiy Storchaka
11.10.21 01:35, MRAB пише: > Maybe what's needed is to add, say, '*' to the format string to indicate > that multiple values should come from an iterable, e.g.: > >     struct.pack_into(f'{len(nums)}*Q', buf, 0, nums) > > in this case len(nums) from the nums argument. See

[Python-Dev] Re: RFC on Callable Type Syntax

2021-10-11 Thread Łukasz Langa
> On 7 Oct 2021, at 18:41, S Pradeep Kumar wrote: > > Note that we considered and rejected using a full def-signature syntax like > > (record: PurchaseRecord, permissions: List[AuthPermission], /) -> > FormattedItem > > because it would be more verbose for common cases and could lead

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

2021-10-11 Thread Serhiy Storchaka
10.10.21 23:38, Guido van Rossum пише: > What's exasperating to me about this whole discussion is that nobody has > shown any reason why this kind of code would be occurring in user code. > STORE_FAST x LOAD_FAST x seems that it must come from a user writing > >     x = x No, it comes from