On 11Apr2020 0025, Antoine Pitrou wrote:
On Fri, 10 Apr 2020 23:33:28 +0100
Steve Dower <steve.do...@python.org> wrote:
On 10Apr2020 2055, Antoine Pitrou wrote:
On Fri, 10 Apr 2020 19:20:00 +0200
Victor Stinner <vstin...@python.org> wrote:
Note: Cython and cffi should be preferred to write new C extensions.
This PEP is about existing C extensions which cannot be rewritten with
Cython.
Using Cython does not make the C API irrelevant. In some
applications, the C API has to be low-level enough for performance.
Whether the application is written in Cython or not.
It does to the code author.
The point here is that we want authors who insist on coding against the
C API to be aware that they have fewer compatibility guarantees [...]
Yeah, you missed the point of my comment here. Cython *does* call into
the C API, and it's quite insistent on performance optimizations too.
Saying "just use Cython" doesn't make the C API unimportant - it just
hides it from your own sight.
It centralises the change. I have no problem giving Cython access to
things that we discourage every developer from using, provided they
remain responsive to change and use the special access responsibly (e.g.
by not touching reserved fields at all).
We could do a better job of helping them here.
**Backward compatibility:** backward incompatible on purpose. Break the
limited C API and the stable ABI, with the assumption that `Most C
extensions don't rely directly on CPython internals`_ and so will remain
compatible.
The problem here is not only compatibility but potential performance
regressions in C extensions.
I don't think we've ever guaranteed performance between releases.
Correctness, sure, but not performance.
That's a rather weird argument. Just because you don't guarantee
performance doesn't mean it's ok to introduce performance regressions.
It's especially a weird argument to make when discussing a PEP where
most of the arguments are distant promises of improved performance.
If you've guaranteed compatibility but not performance, it means you can
make changes that prioritise compatibility over performance.
If you promise to keep everything the same, you can never change
anything. Arguing that everything is an implied contract between major
version releases is the weird argument.
Fork and "Copy-on-Read" problem
...............................
Solve the "Copy on read" problem with fork: store reference counter
outside ``PyObject``.
Nowadays it is strongly recommended to use multiprocessing with the
"forkserver" start method:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
With "forkserver", the forked process is extremely lightweight and
there are little savings to be made in the child.
Unfortunately, a recommendation that only applies to a minority of
Python users. Oh well.
Which "minority" are you talking about? Neither of us has numbers, but
I'm quite sure that the population of Python users calling into
multiprocessing (or a third-party library relying on multiprocessing,
such as Dask) is much larger than the population of Python users
calling fork() directly and relying on copy-on-write for optimization
purposes.
But if you have a different experience to share, please do so.
Neither Windows not macOS support fork (macOS only recently).
Break that down however you like, but by number of *developers* (as
opposed to number of machines), and factoring in those who care about
cross-platform compatibility, fork is not a viable thing to rely on.
Separating refcounts theoretically improves cache locality, specifically
the case where cache invalidation impacts multiple CPUs (and even the
case where a single thread moves between CPUs).
I'm a bit curious why it would improve, rather than degrade, cache
locality. If you take the typical example of the eval loop, an object
is incref'ed and decref'ed just about the same time that it gets used.
Two CPUs can read the contents of a string from their own cache. As soon
as one touches the refcount, the cache line containing both the refcount
and the string data in the other CPU is invalidated, and now it has to
wait for synchronisation before reading the data.
If the refcounts are in a separate cache line, this synchronization
doesn't have to happen.
I'll also note that the PEP proposes to remove APIs which return
borrowed references... yet increasing the number of cases where
accessing an object implies updating its refcount.
Yeah, I'm more okay with keeping borrowed references in some cases, but
it does make things more complicated. Apparently some developers get it
wrong consistently enough that we have to fix it? (ALL developers get it
wrong during development ;) )
and this code
that they're using doesn't have any system dependencies that differ in
debug builds (spoiler: they do).
Are you talking about Windows? On non-Windows systems, I don't think
there are "system dependencies that differ in debug builds".
Of course I'm talking about Windows. I'm about the only person here who
does, and I'm having to represent at least half of our overall userbase
(look up my PyCon 2019 talk for the charts).
Just because I'm the minority in this group doesn't mean I'm holding a
minority opinion.
Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/RE7KWRD4FQ3S53KLJQV7LVYIYF5LAOTX/
Code of Conduct: http://python.org/psf/codeofconduct/
