Le mer. 21 nov. 2018 à 12:11, Antoine Pitrou a écrit :
> You mean the same API can compile to two different things depending on
> a configuration?
Yes, my current plan is to keep #include but have an opt-in
define to switch to the new C API.
> I expect it to be error-prone. For example, let's
On Tue, 20 Nov 2018 23:17:05 +0100
Victor Stinner wrote:
> Le mar. 20 nov. 2018 à 23:08, Stefan Krah a écrit :
> > Intuitively, it should probably not be part of a limited API, but I never
> > quite understood the purpose of this API, because I regularly need any
> > function that I can get my
On 11/20/2018 10:33 PM, Nathaniel Smith wrote:
On Tue, Nov 20, 2018 at 6:05 PM Glenn Linderman wrote:
On 11/20/2018 2:17 PM, Victor Stinner wrote:
IMHO performance and hiding implementation details are exclusive. You
should either use the C API with impl. details for best performances,
or use
On Tue, Nov 20, 2018 at 6:05 PM Glenn Linderman wrote:
>
> On 11/20/2018 2:17 PM, Victor Stinner wrote:
>> IMHO performance and hiding implementation details are exclusive. You
>> should either use the C API with impl. details for best performances,
>> or use a "limited" C API for best
On 11/20/2018 2:17 PM, Victor Stinner wrote:
Le mar. 20 nov. 2018 à 23:08, Stefan Krah a écrit :
Intuitively, it should probably not be part of a limited API, but I never
quite understood the purpose of this API, because I regularly need any
function that I can get my hands on.
(...)
Reading
Le mar. 20 nov. 2018 à 23:08, Stefan Krah a écrit :
> Intuitively, it should probably not be part of a limited API, but I never
> quite understood the purpose of this API, because I regularly need any
> function that I can get my hands on.
> (...)
> Reading typed strings directly into an array
On Mon, Nov 19, 2018 at 04:08:07PM +0100, Victor Stinner wrote:
> Le lun. 19 nov. 2018 à 13:18, Stefan Krah a écrit :
> > In practice people desperately *have* to use whatever is there, including
> > functions with underscores that are not even officially in the C-API.
> >
> > I have to use
On Tue, Nov 20, 2018 at 1:34 AM Petr Viktorin wrote:
>
> On 11/19/18 12:14 PM, Victor Stinner wrote:
> > To design a new C API, I see 3 options:
> >
> > (1) add more functions to the existing Py_LIMITED_API
> > (2) "fork" the current public C API: remove functions and hide as much
> >
On 11/19/18 12:14 PM, Victor Stinner wrote:
To design a new C API, I see 3 options:
(1) add more functions to the existing Py_LIMITED_API
(2) "fork" the current public C API: remove functions and hide as much
implementation details as possible
(3) write a new C API from scratch, based on the
On Mon., Nov. 19, 2018, 14:04 Neil Schemenauer On 2018-11-19, Antoine Pitrou wrote:
> > There are important use cases for the C API where it is desired to have
> > fast type-specific access to Python objects such as tuples, ints,
> > strings, etc. This is relied upon by modules such as _json and
On 2018-11-19, Antoine Pitrou wrote:
> There are important use cases for the C API where it is desired to have
> fast type-specific access to Python objects such as tuples, ints,
> strings, etc. This is relied upon by modules such as _json and _pickle,
> and third-party extensions as well.
Thank
On 2018-11-19, Victor Stinner wrote:
> Moreover, I failed to find anyone who can explain me how the C API
> is used in the wild, which functions are important or not, what is
> the C API, etc.
One idea is to download a large sample of extension modules from
PyPI and then analyze them with some
Hi Stefan,
Le lun. 19 nov. 2018 à 13:18, Stefan Krah a écrit :
> In practice people desperately *have* to use whatever is there, including
> functions with underscores that are not even officially in the C-API.
>
> I have to use _PyFloat_Pack* in order to be compatible with CPython,
Oh, I never
Victor Stinner wrote:
> Moreover, I failed to find anyone who can explain me how the C API is used
> in the wild, which functions are important or not, what is the C API, etc.
In practice people desperately *have* to use whatever is there, including
functions with underscores that are not even
To design a new C API, I see 3 options:
(1) add more functions to the existing Py_LIMITED_API
(2) "fork" the current public C API: remove functions and hide as much
implementation details as possible
(3) write a new C API from scratch, based on the current C API.
Something like #define
On Mon, 19 Nov 2018 11:53:42 +0100
Antoine Pitrou wrote:
> On Mon, 19 Nov 2018 11:28:46 +0100
> Victor Stinner wrote:
> > I would expect that the most common source of speed up of a C
> > extension is the removal of the cost of bytecode evaluation (ceval.c
> > loop).
>
> Well, I don't. All
On 19.11.2018 11:53, Antoine Pitrou wrote:
> On Mon, 19 Nov 2018 11:28:46 +0100
> Victor Stinner wrote:
>> Python internals rely on internals to implement further optimizations,
>> than modifying an "immutable" tuple, bytes or str object, because you
>> can do that at the C level. But I'm not
On Mon, 19 Nov 2018 11:28:46 +0100
Victor Stinner wrote:
> I would expect that the most common source of speed up of a C
> extension is the removal of the cost of bytecode evaluation (ceval.c
> loop).
Well, I don't. All previous experiments showed that simply compiling
Python code to C code
Le lun. 19 nov. 2018 à 10:48, Antoine Pitrou a écrit :
> If the C API only provides Python-level semantics, then it will
> roughly have the speed of pure Python (modulo bytecode execution).
>
> There are important use cases for the C API where it is desired to have
> fast type-specific access to
On Sun, 18 Nov 2018 16:53:19 +0100
Stefan Behnel wrote:
>
> So, in Cython, we use macros wherever possible, and often avoid generic
> protocols in favour of type specialisations. We sometimes keep local copies
> of C-API helper functions, because inlining them allows the C compiler to
> strip
On Fri, 16 Nov 2018 09:46:36 -0800
Brett Cannon wrote:
>
> I think part of the challenge here (and I believe it has been brought up
> elsewhere) is no one knows what kind of API is necessary for some faster VM
> other than PyPy. To me, the only C API that would could potentially start
> working
On Fri, 16 Nov 2018 at 10:11, Paul Moore wrote:
> On Fri, 16 Nov 2018 at 17:49, Brett Cannon wrote:
> > And Just to be clear, I totally support coming up with a totally
> stripped-down C API as I have outlined above as that shouldn't be
> controversial for any VM that wants to have a C-level
Neil Schemenauer schrieb am 17.11.18 um 00:10:
> I think making PyObject an opaque pointer would help.
... well, as long as type checks are still as fast as with "ob_type", and
visible to the C compiler so that it can eliminate redundant ones, I
wouldn't mind. :)
> - Borrowed references are a
On 2018-11-16, Nathaniel Smith wrote:
> [..] it seems like you should investigate (a) whether you can make
> Py_LIMITED_API *be* that API, instead of having two different
> ifdefs
That might be a good idea. One problem is that we might like to
make backwards incompatible changes to
On Fri, Nov 16, 2018 at 3:12 PM Neil Schemenauer wrote:
> Also, the extension module should not take a big performance hit.
> So, you can't change all APIs that were macros into non-inlined
> functions. People are not going to accept that and rightly so.
> However, it could be that we introduce
On 2018-11-16, Brett Cannon wrote:
> I think part of the challenge here (and I believe it has been
> brought up elsewhere) is no one knows what kind of API is
> necessary for some faster VM other than PyPy.
I think we have some pretty good ideas as to what are the
problematic parts of the current
On Fri, 16 Nov 2018 at 17:49, Brett Cannon wrote:
> And Just to be clear, I totally support coming up with a totally
> stripped-down C API as I have outlined above as that shouldn't be
> controversial for any VM that wants to have a C-level API.
If a stripped down API like this is intended as
Brett:
> But otherwise I think we are making assumptions here. For me, unless we are
> trying to trim the C API down to just what is syntactically supported in
> Python and in such a way that it hides all C-level details I feel like we are
> guessing at what's best for other VMs, both today and
On Wed, 14 Nov 2018 at 16:09, Gregory P. Smith wrote:
> It seems like the discussion so far is:
>>
>> Victor: "I know people when people hear 'new API' they get scared and
>> think we're going to do a Python-3-like breaking transition, but don't
>> worry, we're never going to do that."
>>
>
> It seems like the discussion so far is:
>
> Victor: "I know people when people hear 'new API' they get scared and
> think we're going to do a Python-3-like breaking transition, but don't
> worry, we're never going to do that."
> Nathaniel: "But then what does the new API add?"
> Greg: "It lets
Le mer. 14 nov. 2018 à 17:28, Paul Moore a écrit :
> OK, got it. Thanks for taking the time to clarify and respond to my
> concerns. Much appreciated.
I'm my fault. I am failing to explain my plan proplerly. It seems like
I had to update my website to better explain :-)
Victor
On Wed, 14 Nov 2018 at 16:00, Victor Stinner wrote:
>
> In short, you don't have to modify your C extensions and they will
> continue to work as before on Python 3.8.
[...]
> I hope that "later" we will get a faster CPython using new optimizations there>, only compatible with C extensions
In short, you don't have to modify your C extensions and they will
continue to work as before on Python 3.8.
I only propose to add a new C API, don't touch the existing one in any
way. Introducing backward incompatible changes to the existing C API
is out of my plan.
/usr/bin/python3.8 will
On Wed, 14 Nov 2018 at 14:39, Paul Moore wrote:
> If it is the case that there's no need for any 3rd party code to
> change in order to continue working with 3.8+, then I apologise for
> the interruption.
This is where being able to edit posts, a la Discourse would be useful :-)
It occurs to
On Wed, 14 Nov 2018 at 14:28, Victor Stinner wrote:
> > assuming the experiment is successful, forced (as opposed to opt-in)
> > migration to the new API would be handled in a gradual,
>
> No, the current C API will remain available. No one is forced to do
> anything. That's not part of my plan.
Le mer. 14 nov. 2018 à 14:36, Paul Moore a écrit :
> PS What percentage does "top 5" translate to? In terms of both
> downloads and actual numbers of extensions? With only 5, it would be
> very easy (I suspect) to get only scientific packages, and (for
> example) miss out totally on database
On Tue, 13 Nov 2018 at 21:02, Victor Stinner wrote:
> My plan is to select something like the top five most popular C
> extensions based on PyPI download statistics. I cannot test
> everything, I have to put practical limits.
You should probably also consider embedding applications - these have
On Dienstag, 13. November 2018 21:59:14 CET Victor Stinner wrote:
> Le mar. 13 nov. 2018 à 20:32, André Malo a écrit :
> > As long as they are recompiled. However, they will lose a lot of
> > performance. Both these points have been mentioned somewhere, I'm
> > certain, but it cannot be stressed
On Wed, 14 Nov 2018 11:48:15 +0100
Victor Stinner wrote:
> Le mer. 14 nov. 2018 à 11:24, Antoine Pitrou a écrit :
> > For example in PyArrow we use PySequence_Fast_GET_ITEM() (*)
>
> Maybe PyArrow is a kind of C extension which should have one
> implementation for the new C API (PyPy) and one
Le mer. 14 nov. 2018 à 11:24, Antoine Pitrou a écrit :
> For example in PyArrow we use PySequence_Fast_GET_ITEM() (*)
Maybe PyArrow is a kind of C extension which should have one
implementation for the new C API (PyPy) and one implementation for the
current C API (CPython)?
Cython can be used
On Wed, 14 Nov 2018 11:03:49 +0100
Victor Stinner wrote:
>
> Oh, I should stop to promote my "CPython fork" idea.
>
> There is already an existing VM which is way faster than CPython but
> its performances are limited by the current C API. The VM is called...
> PyPy!
>
> The bet is that
Le mer. 14 nov. 2018 à 03:24, Nathaniel Smith a écrit :
> So I think what you're saying is that your goal is to get a
> new/better/shinier VM, and the plan to accomplish that is:
>
> 1. Define a new C API.
> 2. Migrate projects to the new C API.
> 3. Build a new VM that gets benefits from only
On Sun, Nov 11, 2018 at 3:19 PM, Victor Stinner wrote:
> I'm not sure yet how far we should go towards a perfect API which
> doesn't leak everything. We have to move slowly, and make sure that we
> don't break major C extensions. We need to write tools to fully
> automate the conversion. If it's
On Mon, Nov 12, 2018 at 10:46 PM, Gregory P. Smith wrote:
>
> On Fri, Nov 9, 2018 at 5:50 PM Nathaniel Smith wrote:
>>
>> On Fri, Nov 9, 2018 at 4:30 PM, Victor Stinner
>> wrote:
>> > Ah, important points. I don't want to touch the current C API nor make
>> > it less efficient. And
Le mar. 13 nov. 2018 à 20:32, André Malo a écrit :
> As long as they are recompiled. However, they will lose a lot of performance.
> Both these points have been mentioned somewhere, I'm certain, but it cannot be
> stressed enough, IMHO.
Somewhere is here:
Victor Stinner wrote:
> Replacing macros with functions has little impact on backward
> compatibility. Most C extensions should still work if macros become
> functions.
As long as they are recompiled. However, they will lose a lot of performance.
Both these points have been mentioned somewhere,
Victor Stinner wrote:
> Replacing macros with functions has little impact on backward
> compatibility. Most C extensions should still work if macros become
> functions.
As long as they are recompiled. However, they will lose a lot of performance.
Both these points have been mentioned somewhere,
Le mar. 13 nov. 2018 à 08:13, Gregory P. Smith a écrit :
> When things have only ever been macros (Py_INCREF, etc) the name can be
> reused if there has never been a function of that name in an old C API. But
> beware of reuse for anything where the semantics change to avoid
>
On Sun, Nov 11, 2018 at 3:19 PM Victor Stinner wrote:
> Le sam. 10 nov. 2018 à 04:02, Nathaniel Smith a écrit :
> > So is it fair to say that your plan is that CPython will always use
> > the current ("old") API internally, and the "new" API will be
> > essentially an abstraction layer, that's
On Fri, Nov 9, 2018 at 5:50 PM Nathaniel Smith wrote:
> On Fri, Nov 9, 2018 at 4:30 PM, Victor Stinner
> wrote:
> > Ah, important points. I don't want to touch the current C API nor make
> > it less efficient. And compatibility in both directions (current C API
> > <=> new C API) is very
Le sam. 10 nov. 2018 à 04:02, Nathaniel Smith a écrit :
> So is it fair to say that your plan is that CPython will always use
> the current ("old") API internally, and the "new" API will be
> essentially an abstraction layer, that's designed to let people write
> C extensions that target the old
On 2018-11-09, Dino Viehland wrote:
> Rather than adding yet another pre-processor directive for this I would
> suggest just adding a new header file that only has the new stable API.
> For example it could just be "py.h" or "pyapi.h". It would have all of the
> definitions for the stable API.
I
>
> That's exactly why I dislike "New", it's like adding "Ex" or "2" to a
> function name :-)
>
> Well, before bikeshedding on the C define name, I would prefer to see
> if the overall idea of trying to push code for the new C API in the
> master branch is a good idea, or if it's too early and the
On Fri, Nov 9, 2018 at 6:03 PM, Victor Stinner wrote:
> Le sam. 10 nov. 2018 à 02:50, Nathaniel Smith a écrit :
>> Doesn't this mean that you're just making the C API larger and more
>> complicated, rather than simplifying it? You cite some benefits
>> (tagged pointers, changing the layout of
Sometimes, code is easier to understand than a long explanation, so
here is a very simple example of modified function for the new C API:
https://bugs.python.org/issue35206
https://github.com/python/cpython/pull/10443/files
PyTuple_GET_ITEM() becomes a function call and the function
Le sam. 10 nov. 2018 à 02:50, Nathaniel Smith a écrit :
> Doesn't this mean that you're just making the C API larger and more
> complicated, rather than simplifying it? You cite some benefits
> (tagged pointers, changing the layout of PyObject, making PyPy's life
> easier), but I don't see how
On Fri, Nov 9, 2018 at 4:30 PM, Victor Stinner wrote:
> Ah, important points. I don't want to touch the current C API nor make
> it less efficient. And compatibility in both directions (current C API
> <=> new C API) is very important for me. There is no such plan as
> "Python 4" which would
Le sam. 10 nov. 2018 à 01:49, Michael Selik a écrit :
>> It uses an opt-in option (Py_NEWCAPI define -- I'm not sure about the
>> name) to get the new API. The current C API is unchanged.
>
> While one can hope that this will be the only time the C API will be revised,
> it may be better to
To hide all implementation details, I propose to stop using macros and
use function calls instead. For example, replace:
#define PyTuple_GET_ITEM(op, i) \
(((PyTupleObject *)(op))->ob_item[i])
with:
# define PyTuple_GET_ITEM(op, i) PyTuple_GetItem(op, i)
With this change, C extensions using
On Fri, Nov 9, 2018 at 4:33 PM Victor Stinner wrote:
> It uses an opt-in option (Py_NEWCAPI define -- I'm not sure about the
> name) to get the new API. The current C API is unchanged.
>
While one can hope that this will be the only time the C API will be
revised, it may be better to number it
Hi,
The current C API of Python is both a strength and a weakness of the
Python ecosystem as a whole. It's a strength because it allows to
quickly reuse a huge number of existing libraries by writing a glue
for them. It made numpy possible and this project is a big sucess!
It's a weakness because
61 matches
Mail list logo
