[Python-Dev] Re: Speeding up CPython

[Gregory P. Smith]

On Wed, Oct 21, 2020 at 4:04 PM Greg Ewing 
wrote:

> A concern I have about this is what effect it will have on the
> complexity of CPython's implementation.
>
> CPython is currently very simple and straightforward. Some parts
> are not quite as simple as they used to be, but on the whole it's
> fairly easy to understand, and I consider this to be one of its
> strengths.
>
> I worry that adding four layers of clever speedup tricks will
> completely destroy this simplicity, leaving us with something
> that can no longer be maintained or contributed to by
> ordinary mortals.
>

I have never considered ceval.c to be simple and straightforward.  Nor our
parser(s).  Or our optimizers.  Or the regular expression implementation.
Or the subprocess internals.  (we may differ on lists of what isn't simple
and straightforward, but i guarantee you we've all got something in mind)

Making this not a major concern for me.

We'd decide if there is something we find to be dark magic that seems
challenging to maintain during the review phases and decide what if
anything needs to be done about that to ameliorate such issues.

-gps


> --
> Greg
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6XBJ2OA46KNMJ6FFI3B6RFYRTTD4HYOI/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QMIKJ3EE73U3J6TM4RGHMTHGB6N4WZGV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Greg Ewing]

A concern I have about this is what effect it will have on the
complexity of CPython's implementation.

CPython is currently very simple and straightforward. Some parts
are not quite as simple as they used to be, but on the whole it's
fairly easy to understand, and I consider this to be one of its
strengths.

I worry that adding four layers of clever speedup tricks will
completely destroy this simplicity, leaving us with something
that can no longer be maintained or contributed to by
ordinary mortals.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6XBJ2OA46KNMJ6FFI3B6RFYRTTD4HYOI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Christian Heimes]

On 21/10/2020 20.55, Larry Hastings wrote:
> On 10/21/20 5:58 AM, Petr Viktorin wrote:
>> At the risk of going off topic: That's for GCC. As far as I know, MSVC
>> uses something like __declspec( thread ).
>> What are the options for generic C99 compilers, other than staying slow?
> 
> 
> As a practical matter: does CPython even support "generic C99
> compilers"?  AFAIK we support three specific compilers: GCC, Clang, and
> MSVC.
> 
> (Maybe we also support icc?  I think mostly because it supports GCC
> language extensions.)

We don't prohibit users to use exotic compilers. Some users maintain
Python on platforms like AIX and Solaris with closed source compilers.

In my opinion it would fine to focus on X86_64 and GCC first. That will
cover the majority of servers and consumer PCs. Clang and GCC have a
similar feature set and extensions, so clang should be doable with
manageable amount of effort, too. After X86_64 I'd consider AArch64
(ARM64) and MSVC next.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N4OKG5Y4TY2VMOJRTTVMLJ3U2LDK3DJO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Larry Hastings]

On 10/21/20 5:58 AM, Petr Viktorin wrote:
At the risk of going off topic: That's for GCC. As far as I know, MSVC 
uses something like __declspec( thread ).

What are the options for generic C99 compilers, other than staying slow?



As a practical matter: does CPython even support "generic C99 
compilers"?  AFAIK we support three specific compilers: GCC, Clang, and 
MSVC.


(Maybe we also support icc?  I think mostly because it supports GCC 
language extensions.)



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FLFDFSZCNVRKRFYBJ72NAN2YOHFD2GD6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PyPy performance stats (was Re: Speeding up CPython)

[Antoine Pitrou]

On Wed, 21 Oct 2020 22:48:59 +1100
Chris Angelico  wrote:
> 
> Ah, cool, got it - thanks! My reading of this is that a good few of
> the benchmarks are coming out firmly in PyPy's favour, but a number of
> them are still in CPython's favour (including "telco" where PyPy is
> staggeringly worse for some reason).

The "telco" benchmark benefits heavily from the C decimal module written
and ex-maintained by Stefan Krah.

> for instance,
> SQLAlchemy seems to be slower, but if you're using SQLAlchemy, you're
> probably more concerned about database performance than the CPU cost
> inside Python.

I certainly wouldn't say that.  The ORM's overhead can be quite
important, and SQLAlchemy has dedicated C extensions to mitigate it.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I7WVMX4JO2AKS2LYTBVY7S7MLCNPXKL3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 11: Drop support for AIX releases without dlopen

[Brett Cannon]

On Tue, Oct 20, 2020 at 11:10 AM Kevin Adler 
wrote:

> Brett Cannon wrote:
> > On Mon, Oct 19, 2020 at 4:43 PM Kevin Adler kad...@linux.vnet.ibm.com
> > wrote:
> > > Should this list be updated to mention that AIX 5.3 and below are no
> > > longer supported?
> > > Only if we are going to rip out all AIX-related code.
>
> Ok. Looking through the history, I think I found an equivalent change to
> model after: when support for FreeBSD 9 and older was removed.
>
> https://github.com/python/cpython/commit/13ff24582c99dfb439b1af7295b401415e7eb05b


Yes, but that's because we have historically considered FreeBSD supported
(there's a buildbot and core devs have been willing to keep it working).


>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LQLPA3MCD4PKZZ7HHNH7EW4ZYKUD3IWA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7UE2J6KIKC2BAJWIWIIAMWUSF222BK5N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Ronald Oussoren via Python-Dev]

> On 21 Oct 2020, at 14:39, Larry Hastings  wrote:
> 
> On 10/21/20 4:04 AM, Antoine Pitrou wrote:
>> (apart from small fixes relating to borrowed references, and
>> that's mostly to make PyPy's life easier).
> 
> Speaking as the Gilectomy guy: borrowed references are evil.  The definition 
> of the valid lifetime of a borrowed reference doesn't exist, because they are 
> a hack (baked into the API!) that we mostly "get away with" just because of 
> the GIL.  If I still had wishes left on my monkey's paw I'd wish them away*.
> 
> 
Even with the GIL borrowed references are problematic. There are a lot of cases 
where using a borrowed reference after calling an API that might run Python 
code might invalidate the borrowed reference. In general the only safe thing to 
do with a borrowed reference is to turn it into a strong reference as soon as 
possible.

Ronald
—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/
> 
> /arry
> 
> * Unfortunately, I used my last wish back in February, wishing I could spend 
> more time at home.
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/PRIVTI2RFGEGVNQRGUCHRRY5WBJNZKJS/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PV3DSRZVS4JLYTGU5Z6JPBKDZT4X4QY5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Steve Holden]

On Wed, Oct 21, 2020 at 8:37 AM Paul Moore  wrote:

> On Wed, 21 Oct 2020 at 08:14, Christian Heimes 
> wrote:
> >
> > On 21/10/2020 00.14, Steven D'Aprano wrote:
> > > On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote:
> > >
> > >> What I don't see is where the money's coming from. It's fine to ask,
> > >> but will anyone come up with that sort of funding?
> > >
> > > I don't think Mark is asking for you or I to fund the exercise. He's
> > > asking for the PSF to fund it.
> >
> > No, he is not. Mark is asking the PSF to organize a fund raiser and keep
> > half the money.
>
> Right. I'd misinterpreted the fact that the PSF was to get half the
> money as meaning they weren't doing the fundraising. My
> misunderstanding, thanks for the clarification.
>
> Paul
>

In 2004 a single company paid me to organise the "Need for Speed" sprint,
held in Iceland. A lot was achieved, particularly in string searching and
the re module, though I can't honestly say how much impact it had on
"overall performance". The work we did with pybench that week definitely
moved Python benchmarking along some.

Sixteen years later, the PSF's income may be down due to external factors,
but its connectivity at a high level with Python users has improved
immeasurably. Need for Speed cost ~$300,000 in inflation-adjusted dollars.
If one relatively small trading house could fund at that level it seems
likely the PSF could fund this suggested a project quite separately from
its existing revenues as long as the development community was behind it
and prepared to help with materials for the "sell."
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/36A6PRABPZPI6OSNNC2ZACJVHNBT7DUD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Petr Viktorin]

On 10/21/20 1:40 PM, Mark Shannon wrote:

Hi Petr,

On 21/10/2020 11:49 am, Petr Viktorin wrote:
Let me explain an impression I'm getting. It is *just one aspect* of 
my opinion, one that doesn't make sense to me. Please tell me where it 
is wrong.



In the C API, there's a somewhat controversial refactoring going on, 
which involves passing around tstate arguments. I'm not saying [the 
first discussion] was perfect, and there are still issues, but, 
however flawed the "do-ocracy" process is, it is the best way we found 
to move forward. No one who can/wants to do the work has a better 
solution.


Later, Mark says there is an even better way – or at least, a less 
intrusive one! In [the second discussion], he hints at it vaguely 
(from that limited info I have, it involves switching to C11 and/or 
using compiler-specific extensions -- not an easy change to do). But 
frustratingly, Mark doesn't reveal any actual details, and a lot of 
the complaints are about churn and merge conflicts.
And now, there's news -- the better solution won't be revealed unless 
the PSF pays for it!


There's no secret. C thread locals are well documented.
I even provided a code example last time we discussed it.

You reminded me of it yesterday ;)
https://godbolt.org/z/dpSo-Q


At the risk of going off topic: That's for GCC. As far as I know, MSVC 
uses something like __declspec( thread ).

What are the options for generic C99 compilers, other than staying slow?

The "even faster" solution I mentioned yesterday, is as I stated 
yesterday to use an aligned stack.

If you wanted more info, you could have asked :)

First, you ensure that the stack is in a 2**N aligned block.
Assuming that the C stack grows down from the top, then the threadstate 
struct goes at the bottom. It's probably a good idea to put a guard page 
between the C stack and the threadstate struct.


The struct's address can then be found by masking off the bottom N bits 
from the stack pointer.
This approach uses 0 registers and cost 1 ALU instruction. Can't get 
cheaper than that :)


It's not portable and probably a pain to implement, but it is fast.

But it doesn't matter how it's implemented. The implementation is hidden 
behind `PyThreadState_GET()`, it can be changed to use a thread local,

or to some fancy aligned stack, without the rest of the codebase changing.


Not portable and hard to implement is a pain for maintenance – 
especially porting CPython to new compilers/platforms/situations.


The alternative is changing the codebase, which (it seems from the 
discussions) would give us comparable performance, everywhere, and the 
result can be maintained by many more people.



That's a very bad situation to be in for having discussions: 
basically, either we disregard Mark and go with the not-ideal 
solution, or virtually all work on changing the C API and internal 
structures is blocked.


The existence of multiple interpreters should be orthogonal to speeding 
up those interpreters, provided the separation is clean and well designed.

But it should be clean and well designed anyway, IMO.


+1

I sense a similar thing happening here: 
https://github.com/ericsnowcurrently/multi-core-python/issues/69 -- 


The title of that issue is 'Clarify what is a "sub-interpreter" and what 
is an "interpreter"'?


there's a vague proposal to do things very differently, but I find it 


This?
https://github.com/ericsnowcurrently/multi-core-python/issues/69#issuecomment-712837899 


I'll continue there.

hard to find anything actionable. I would like to change my plans to 
align with Mark's fork, or to better explain some of the 
non-performance reasons for recent/planned changes. But I can't, 
because details are behind a paywall.


Let's make this very clear.
My objections to the way multiple interpreters is being implemented has 
very little to do speeding up the interpreter and entirely to do with 
long term maintenance and ultimate success of the project.


Obviously, I would like it if multiple interpreters didn't slowdown 
CPython.

But that has always been the case.



Thank you for clearing my doubts!


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W5JF77HVMIJ3Q5RSL3R2TOJGZ4JEWJRS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Larry Hastings]

On 10/21/20 4:04 AM, Antoine Pitrou wrote:

(apart from small fixes relating to borrowed references, and
that's mostly to make PyPy's life easier).



Speaking as the Gilectomy guy: borrowed references are evil.  The 
definition of the valid lifetime of a borrowed reference doesn't exist, 
because they are a hack (baked into the API!) that we mostly "get away 
with" just because of the GIL.  If I still had wishes left on my 
monkey's paw I'd wish them away*.



//arry/

* Unfortunately, I used my last wish back in February, wishing I could 
spend more time at home.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PRIVTI2RFGEGVNQRGUCHRRY5WBJNZKJS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 640: Unused variable syntax.

[Larry Hastings]

On 10/20/20 10:45 PM, Paul Sokolovsky wrote:

One problem with this PEP, which I didn't see mentioned in the other
replies, is that it tries to grab "?" character, which is already
sought-for by another pending PEP: "PEP 505 -- None-aware operators",
https://www.python.org/dev/peps/pep-0505/ .



I don't think PEP 505 is much of a concern.  It's from 2015, and it's 
marked "Deferred".  I suspect if it was going to happen it would have 
happened by now.



//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z6MUYOAUEMJE6BHBOKN4XJIAL2IZY6CE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Macro for logging

[Antoine Pitrou]

On Wed, 21 Oct 2020 14:19:37 +0200
Marco Sulla  wrote:
> If not already present, do you think it's useful to add a macro that does
> something like
> 
> # ifdef Py_DEBUG
> fprintf(stderr, "%s\n", message);
> # endif

In general, you want to do this on a per-component basis, so each C
source file tends to redefine its own macros if it needs to.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZG4OEHT4B2XCQUQAR5U4MDU2TOV2QO5M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Macro for logging

[Marco Sulla]

If not already present, do you think it's useful to add a macro that does
something like

# ifdef Py_DEBUG
fprintf(stderr, "%s\n", message);
# endif

?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6W6YO6JSJZOGWYWNWB2ARUS4LSLY3C7Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Simon Cross]

On Wed, Oct 21, 2020 at 4:28 AM Terry Reedy  wrote:
> I don't think the two projects are mutually exclusive.

100% agreed. I would even go as far as to say that HPy and other
proposals to improve Python are mutually beneficial. HPy aims to
remove dependencies between C extensions and Python internals, so that
Python internals can evolve more easily. Someone else still needs to
do the evolving.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LRNDTBSPIM26ITQPQJER3UPQPLKSO5SS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PyPy performance stats (was Re: Speeding up CPython)

[Chris Angelico]

On Wed, Oct 21, 2020 at 10:38 PM Matti Picus  wrote:
>
> On 10/21/20 20:42:02 +1100 Chris Angelico wrote:
>
> > When I go looking for PyPy performance stats, everything seems to be
> > Python 2.7. Is there anywhere that compares PyPy3 to CPython 3.6 (or
> > whichever specific version)? Or maybe it's right there on
> > https://speed.pypy.org/  and I just can't see it - that's definitely
> > possible:)
> >
> > ChrisA
>
>
> They are not on the front page. You can find them, but it requires
> digging around in the Comparison page[0].
>
> - Executables: use only "cpython 3.7.6" and any of the "pypy3-jit-64"
>
> - Chart type: normal bars
>
> - Normalization: cpython3.7.6 or pypy3-jit-64, whichever way makes more
> sense to you
>
> - horizontal.

Ah, cool, got it - thanks! My reading of this is that a good few of
the benchmarks are coming out firmly in PyPy's favour, but a number of
them are still in CPython's favour (including "telco" where PyPy is
staggeringly worse for some reason). But a number of the "PyPy is
worse" benchmarks are ones that seem less significant - for instance,
SQLAlchemy seems to be slower, but if you're using SQLAlchemy, you're
probably more concerned about database performance than the CPU cost
inside Python.

> For some reason the "permalink" feature does not let me share that
> configuration.

Yeah, there seem to be some issues with that page. The first time I
loaded it, the chart was entirely broken - I was looking for the "show
chart" button, but it was just that it had errored out for some
reason. Still, at least the data's there.

> I guess we could switch to emphasizing python3 on the front page, help
> in updating and reconfiguring Codespeed [1] would be awesome.

If only I had infinite time in the day, I'd love to dive in and help
out... It definitely would be worth at least having a quick summary
and link somewhere.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IP7DXROXIYAV7RXL2JSILON7CBKZYELJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Mark Shannon]

Hi Petr,

On 21/10/2020 11:49 am, Petr Viktorin wrote:
Let me explain an impression I'm getting. It is *just one aspect* of my 
opinion, one that doesn't make sense to me. Please tell me where it is 
wrong.



In the C API, there's a somewhat controversial refactoring going on, 
which involves passing around tstate arguments. I'm not saying [the 
first discussion] was perfect, and there are still issues, but, however 
flawed the "do-ocracy" process is, it is the best way we found to move 
forward. No one who can/wants to do the work has a better solution.


Later, Mark says there is an even better way – or at least, a less 
intrusive one! In [the second discussion], he hints at it vaguely (from 
that limited info I have, it involves switching to C11 and/or using 
compiler-specific extensions -- not an easy change to do). But 
frustratingly, Mark doesn't reveal any actual details, and a lot of the 
complaints are about churn and merge conflicts.
And now, there's news -- the better solution won't be revealed unless 
the PSF pays for it!


There's no secret. C thread locals are well documented.
I even provided a code example last time we discussed it.

You reminded me of it yesterday ;)
https://godbolt.org/z/dpSo-Q

The "even faster" solution I mentioned yesterday, is as I stated 
yesterday to use an aligned stack.

If you wanted more info, you could have asked :)

First, you ensure that the stack is in a 2**N aligned block.
Assuming that the C stack grows down from the top, then the threadstate 
struct goes at the bottom. It's probably a good idea to put a guard page 
between the C stack and the threadstate struct.


The struct's address can then be found by masking off the bottom N bits 
from the stack pointer.
This approach uses 0 registers and cost 1 ALU instruction. Can't get 
cheaper than that :)


It's not portable and probably a pain to implement, but it is fast.

But it doesn't matter how it's implemented. The implementation is hidden 
behind `PyThreadState_GET()`, it can be changed to use a thread local,

or to some fancy aligned stack, without the rest of the codebase changing.



That's a very bad situation to be in for having discussions: basically, 
either we disregard Mark and go with the not-ideal solution, or 
virtually all work on changing the C API and internal structures is 
blocked.


The existence of multiple interpreters should be orthogonal to speeding 
up those interpreters, provided the separation is clean and well designed.

But it should be clean and well designed anyway, IMO.



I sense a similar thing happening here: 
https://github.com/ericsnowcurrently/multi-core-python/issues/69 -- 


The title of that issue is 'Clarify what is a "sub-interpreter" and what 
is an "interpreter"'?


there's a vague proposal to do things very differently, but I find it 


This?
https://github.com/ericsnowcurrently/multi-core-python/issues/69#issuecomment-712837899

hard to find anything actionable. I would like to change my plans to 
align with Mark's fork, or to better explain some of the non-performance 
reasons for recent/planned changes. But I can't, because details are 
behind a paywall.


Let's make this very clear.
My objections to the way multiple interpreters is being implemented has 
very little to do speeding up the interpreter and entirely to do with 
long term maintenance and ultimate success of the project.


Obviously, I would like it if multiple interpreters didn't slowdown CPython.
But that has always been the case.

Cheers,
Mark.




[the first discussion]: 
https://mail.python.org/archives/list/python-dev@python.org/thread/PQBGECVGVYFTVDLBYURLCXA3T7IPEHHO/#Q4IPXMQIM5YRLZLHADUGSUT4ZLXQ6MYY 



[the second discussion]: 
https://mail.python.org/archives/list/python-dev@python.org/thread/KGBXVVJQZJEEZD7KDS5G3GLBGZ6XNJJX/#WOKAUQYDJDVRA7SJRJDEAHXTRXSVPNMO 




On 10/20/20 2:53 PM, Mark Shannon wrote:

Hi everyone,

CPython is slow. We all know that, yet little is done to fix it.

I'd like to change that.
I have a plan to speed up CPython by a factor of five over the next 
few years. But it needs funding.


I am aware that there have been several promised speed ups in the past 
that have failed. You might wonder why this is different.


Here are three reasons:
1. I already have working code for the first stage.
2. I'm not promising a silver bullet. I recognize that this is a 
substantial amount of work and needs funding.
3. I have extensive experience in VM implementation, not to mention a 
PhD in the subject.


My ideas for possible funding, as well as the actual plan of 
development, can be found here:


https://github.com/markshannon/faster-cpython

I'd love to hear your thoughts on this.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 

[Python-Dev] Re: Speeding up CPython

[Antoine Pitrou]

On Wed, 21 Oct 2020 12:49:54 +0200
Petr Viktorin  wrote:
> 
> Later, Mark says there is an even better way – or at least, a less 
> intrusive one! In [the second discussion], he hints at it vaguely (from 
> that limited info I have, it involves switching to C11 and/or using 
> compiler-specific extensions -- not an easy change to do). But 
> frustratingly, Mark doesn't reveal any actual details, and a lot of the 
> complaints are about churn and merge conflicts.
> And now, there's news -- the better solution won't be revealed unless 
> the PSF pays for it!
> 
> That's a very bad situation to be in for having discussions: basically, 
> either we disregard Mark and go with the not-ideal solution, or 
> virtually all work on changing the C API and internal structures is blocked.

The disagreement is basically on the promises of the "not-ideal
solution".  Victor claims it will help improve performance.  People
like Mark and I are skeptical that the C API is really an important
concern (apart from small fixes relating to borrowed references, and
that's mostly to make PyPy's life easier).

Personally, I trust that Mark's proposed plan is workable.  That
doesn't mean it *will* work (that depends a lot on being able to
maintain motivation and integrate changes at a good pace - which may
be a challenge given the technical conservativeness in the CPython
community), but it's technically sound.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SDGDVGCRGJOMXTQLXXJQ5IHM3L3WPYAU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Petr Viktorin]

Let me explain an impression I'm getting. It is *just one aspect* of my 
opinion, one that doesn't make sense to me. Please tell me where it is 
wrong.



In the C API, there's a somewhat controversial refactoring going on, 
which involves passing around tstate arguments. I'm not saying [the 
first discussion] was perfect, and there are still issues, but, however 
flawed the "do-ocracy" process is, it is the best way we found to move 
forward. No one who can/wants to do the work has a better solution.


Later, Mark says there is an even better way – or at least, a less 
intrusive one! In [the second discussion], he hints at it vaguely (from 
that limited info I have, it involves switching to C11 and/or using 
compiler-specific extensions -- not an easy change to do). But 
frustratingly, Mark doesn't reveal any actual details, and a lot of the 
complaints are about churn and merge conflicts.
And now, there's news -- the better solution won't be revealed unless 
the PSF pays for it!


That's a very bad situation to be in for having discussions: basically, 
either we disregard Mark and go with the not-ideal solution, or 
virtually all work on changing the C API and internal structures is blocked.


I sense a similar thing happening here: 
https://github.com/ericsnowcurrently/multi-core-python/issues/69 -- 
there's a vague proposal to do things very differently, but I find it 
hard to find anything actionable. I would like to change my plans to 
align with Mark's fork, or to better explain some of the non-performance 
reasons for recent/planned changes. But I can't, because details are 
behind a paywall.



[the first discussion]: 
https://mail.python.org/archives/list/python-dev@python.org/thread/PQBGECVGVYFTVDLBYURLCXA3T7IPEHHO/#Q4IPXMQIM5YRLZLHADUGSUT4ZLXQ6MYY


[the second discussion]: 
https://mail.python.org/archives/list/python-dev@python.org/thread/KGBXVVJQZJEEZD7KDS5G3GLBGZ6XNJJX/#WOKAUQYDJDVRA7SJRJDEAHXTRXSVPNMO



On 10/20/20 2:53 PM, Mark Shannon wrote:

Hi everyone,

CPython is slow. We all know that, yet little is done to fix it.

I'd like to change that.
I have a plan to speed up CPython by a factor of five over the next few 
years. But it needs funding.


I am aware that there have been several promised speed ups in the past 
that have failed. You might wonder why this is different.


Here are three reasons:
1. I already have working code for the first stage.
2. I'm not promising a silver bullet. I recognize that this is a 
substantial amount of work and needs funding.
3. I have extensive experience in VM implementation, not to mention a 
PhD in the subject.


My ideas for possible funding, as well as the actual plan of 
development, can be found here:


https://github.com/markshannon/faster-cpython

I'd love to hear your thoughts on this.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RDXLCH22T2EZDRCBM6ZYYIUTBWQVVVWH/ 


Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7DKURFZ3JEZTKCUAUDCPR527FUBYMY7N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Christian Heimes]

On 21/10/2020 11.37, Steven D'Aprano wrote:
> On Wed, Oct 21, 2020 at 09:06:58AM +0200, Christian Heimes wrote:
>> On 21/10/2020 00.14, Steven D'Aprano wrote:
>>> On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote:
>>>
 What I don't see is where the money's coming from. It's fine to ask,
 but will anyone come up with that sort of funding?
>>>
>>> I don't think Mark is asking for you or I to fund the exercise. He's 
>>> asking for the PSF to fund it.
>>
>> No, he is not. Mark is asking the PSF to organize a fund raiser and keep
>> half the money.
> 
> I think that's inaccurate. The funding.md document doesn't mention "fund 
> raiser", it doesn't specify how the PSF is to collect the funds, and I 
> wouldn't expect it to.

You are spreading FUD. Stop speculating and give Mark the benefit of a
doubt. This kind of negative attitude is a main cause of contributor
burnout.

If you are unsure about Mark's intentions and goals then please open an
issue on his repository.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NKJN6BPR737NENM4F4AGV6DZ7FTPPMBF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Chris Angelico]

On Wed, Oct 21, 2020 at 8:23 PM Matti Picus  wrote:
> Just to set the record straight, PyPy has been available on conda-forge
> [0] since March, and has seen close to 70,000 downloads [1] from that
> channel alone, in addition to the downloads from
> https://downloads.python.org/pypy and the other channels where it is
> available. This is far from CPython's wild popularity, but people are
> successfully using it with the scientific python stack. It is true there
> is more work to be done, that does not mean it is useless.
>

When I go looking for PyPy performance stats, everything seems to be
Python 2.7. Is there anywhere that compares PyPy3 to CPython 3.6 (or
whichever specific version)? Or maybe it's right there on
https://speed.pypy.org/ and I just can't see it - that's definitely
possible :)

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V2P5LDQWLMRYP24Q7O7QE3S4YDLUQ224/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Steven D'Aprano]

On Wed, Oct 21, 2020 at 09:06:58AM +0200, Christian Heimes wrote:
> On 21/10/2020 00.14, Steven D'Aprano wrote:
> > On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote:
> > 
> >> What I don't see is where the money's coming from. It's fine to ask,
> >> but will anyone come up with that sort of funding?
> > 
> > I don't think Mark is asking for you or I to fund the exercise. He's 
> > asking for the PSF to fund it.
> 
> No, he is not. Mark is asking the PSF to organize a fund raiser and keep
> half the money.

I think that's inaccurate. The funding.md document doesn't mention "fund 
raiser", it doesn't specify how the PSF is to collect the funds, and I 
wouldn't expect it to.

The PSF has various income streams, one of which is donations, and how 
they collect the money for this proposal is up to them, not Mark. If the 
PSF decide that the best way to make the money is to have a bake sale, 
that's their prerogative. The *how* is not really relevant or up to us 
(except to the degree that some of us may be members of the PSF).

And they won't be *keeping* half the money, they will be spending it on 
on-going maintenance. (Or at least that is Mark's expectation.) So the 
intent is for the money to be spent at some point, and not for general 
expenses, but specifically on this project.

-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U4S7JVWAD2FM4ZYPCJ774GODM2OGOSB2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Matti Picus]

On Wed, Oct 21, 2020 at 3:38 AM Steven D'Aprano st...@pearwood.info 
 wrote:


> some insulting FUD that is not worth repeating, and an apology


Just to set the record straight, PyPy has been available on conda-forge 
[0] since March, and has seen close to 70,000 downloads [1] from that 
channel alone, in addition to the downloads from 
https://downloads.python.org/pypy and the other channels where it is 
available. This is far from CPython's wild popularity, but people are 
successfully using it with the scientific python stack. It is true there 
is more work to be done, that does not mean it is useless.



It is in CPython's interest to have a viable alternative interpreter for 
many reasons. The code of conduct [3] should guide discourse when 
relating to all people and projects in the Python ecosystem, not just 
internally to CPython.


Matti


[0] https://conda-forge.org/blog/posts/2020-03-10-pypy/

[1] https://anaconda.org/conda-forge/pypy3.6

[2] https://www.python.org/psf/conduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MPU2RYONWC4KHXEJ5D4VHHNSICIDY6LD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Christian Heimes]

On 21/10/2020 09.35, Paul Moore wrote:
> On Wed, 21 Oct 2020 at 08:14, Christian Heimes  wrote:
>>
>> On 21/10/2020 00.14, Steven D'Aprano wrote:
>>> On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote:
>>>
 What I don't see is where the money's coming from. It's fine to ask,
 but will anyone come up with that sort of funding?
>>>
>>> I don't think Mark is asking for you or I to fund the exercise. He's
>>> asking for the PSF to fund it.
>>
>> No, he is not. Mark is asking the PSF to organize a fund raiser and keep
>> half the money.
> 
> Right. I'd misinterpreted the fact that the PSF was to get half the
> money as meaning they weren't doing the fundraising. My
> misunderstanding, thanks for the clarification.

You are welcome! IMHO you got it right in your initial posting.

I'm irritated that Steven is spreading FUD although Mark's documents
clearly states his intention [1]:

The PSF seems to be the obvious organization to coordinate funding.

Mark is asking the PSF to organize a fund raiser and to act as a
trustee. This is the most logical and reasonable approach. We don't want
to have another requests incident, do we? For stage 2-4 Mark is also
willing to put his reputation and financial security in jeopardy *and*
give the PSF half of the income in exchange for little risk.

Christian

[1] https://github.com/markshannon/faster-cpython/blob/master/funding.md
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5JEPPD4ZFHSPLMBOFF4F444HG72M3E2M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Antoine Pitrou]

On Tue, 20 Oct 2020 16:10:27 +0100
Mark Shannon  wrote:
> Hi Antoine,
> 
> On 20/10/2020 2:32 pm, Antoine Pitrou wrote:
> > On Tue, 20 Oct 2020 13:53:34 +0100
> > Mark Shannon  wrote:  
> >> Hi everyone,
> >>
> >> CPython is slow. We all know that, yet little is done to fix it.
> >>
> >> I'd like to change that.
> >> I have a plan to speed up CPython by a factor of five over the next few
> >> years. But it needs funding.
> >>
> >> I am aware that there have been several promised speed ups in the past
> >> that have failed. You might wonder why this is different.
> >>
> >> Here are three reasons:
> >> 1. I already have working code for the first stage.
> >> 2. I'm not promising a silver bullet. I recognize that this is a
> >> substantial amount of work and needs funding.
> >> 3. I have extensive experience in VM implementation, not to mention a
> >> PhD in the subject.
> >>
> >> My ideas for possible funding, as well as the actual plan of
> >> development, can be found here:
> >>
> >> https://github.com/markshannon/faster-cpython  
> > 
> > Do you plan to do all this in C, or would you switch to C++ (or
> > something else)?  
> 
> All C, no C++. I promise :)

Interesting, I was mostly expecting/suggesting the opposite.  Once you
pass a certain level of complexity, C is really a burden.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H7665MH2KQ7W5NEZW2BMS3AD4EP2NFGK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Paul Moore]

On Wed, 21 Oct 2020 at 08:14, Christian Heimes  wrote:
>
> On 21/10/2020 00.14, Steven D'Aprano wrote:
> > On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote:
> >
> >> What I don't see is where the money's coming from. It's fine to ask,
> >> but will anyone come up with that sort of funding?
> >
> > I don't think Mark is asking for you or I to fund the exercise. He's
> > asking for the PSF to fund it.
>
> No, he is not. Mark is asking the PSF to organize a fund raiser and keep
> half the money.

Right. I'd misinterpreted the fact that the PSF was to get half the
money as meaning they weren't doing the fundraising. My
misunderstanding, thanks for the clarification.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N6PBKMDETBSZURV6KOKCARHRJBZCMVDW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Christian Heimes]

On 21/10/2020 00.14, Steven D'Aprano wrote:
> On Tue, Oct 20, 2020 at 06:04:37PM +0100, Paul Moore wrote:
> 
>> What I don't see is where the money's coming from. It's fine to ask,
>> but will anyone come up with that sort of funding?
> 
> I don't think Mark is asking for you or I to fund the exercise. He's 
> asking for the PSF to fund it.

No, he is not. Mark is asking the PSF to organize a fund raiser and keep
half the money.

Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TOKFADEIOASTU3KMAIMYA226TIJ6IM3T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Stefan Ring]

On Wed, Oct 21, 2020 at 3:51 AM Gregory P. Smith  wrote:
>
> meta: i've written too many words and edited so often i can't see my own 
> typos and misedits anymore.  i'll stop now. :)

Haha! Very interesting background, thank you for writing down all of this!
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CFC4XSS5AR2MBEDSQTNTL4GKH63RKVRZ/
Code of Conduct: http://python.org/psf/codeofconduct/