Re: [Python-Dev] Let's change to C API!
On Tue, 31 Jul 2018 02:29:42 +0200 Victor Stinner wrote: > >> Last year, I gave a talk at the Language Summit (during Pycon) to > >> explain that CPython should become 2x faster to remain competitive. > >> IMHO all attempts to optimize Python (CPython forks) have failed > >> because they have been blocked by the C API which implies strict > >> constraints. > > > > Well, that's your opinion, but did you prove it? > > I discussed with many Python developers who agree with me that the C API > blocked CPython forks. For example, I heard that Pyston was very fast and > very promising before starting to support the C API. What exactly in the C API made it slow or non-promising? > The C API requires that your implementations make almost all the same > design choices that CPython made 25 years ago (C structures, memory > allocators, reference couting, specific GC implementation, GIL, etc.). Yes, but those choices are not necessarily bad. > Multiple PyPy developers told me that cpyext remains a blocker issue to use > PyPy. Probably, but we're talking about speeding up CPython here, right? If we're talking about making more C extensions PyPy-compatible, that's a different discussion, and one where I think Stefan is right that we should push people towards Cython and alternatives, rather than direct use of the C API (which people often fail to use correctly, in my experience). But the C API is still useful for specialized uses, *including* for development tools such as Cython. > Oh, by the way, I will not promise anything about any potential performance > gain. When I write "2x faster", I mean that our current approach for > optimization failed to make Python 2x faster over the last 10 years. Python > 3 is more or less as fast, or a little bit faster, than Python 2. But > Python 2 must not be used as an example of performance. People hesitate > between Go, Javascript and Python. And Python is not the winner in term of > performance. I agree about the overall diagnosis. I just disagree that changing the C API will open up easy optimization opportunities. Actually I'd like to see a list of optimizations that you think are held up by the C API. > I have to confess that helping Larry is part of my overall plan. Which is why I'd like to see Larry chime in here. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
On 2018-07-31 08:58, Antoine Pitrou wrote: I think Stefan is right that we should push people towards Cython and alternatives, rather than direct use of the C API (which people often fail to use correctly, in my experience). I know this probably isn't the correct place to bring it up, but I'm sure that CPython itself could benefit from using Cython. For example, most of the C extensions in Modules/ could be written in Cython. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
First of all, I'm sorry to I forgot change my mail title. (I though about reserving one more slot for Cython for further Cython-to-Cython call optimization, but I rejected my idea because I'm not sure it really help Cython.) On Mon, Jul 30, 2018 at 11:55 PM Jeroen Demeyer wrote: > > On 2018-07-30 15:35, INADA Naoki wrote: > > As repeatedly said, PEP 580 is very complicated protocol > > when just implementing callable object. > > Can you be more concrete what you find complicated? Maybe I can improve > the PEP to explain it more. Also, I'm open to suggestions to make it > less complicated. When thinking from extension writer's point of view, almost all of PEP 580 is complicated comparing PEP 576. Remember they don't need custom method/function type. So PEP 576/580 are needed only when implementing callable object, like itemgetter or lru_cache in stdlib. * We continue to use PyMethodDef and METH_* when writing tp_methods. They should learn PyCCallDef and CCALL_* flags in addition to PyMethodDef and METH_*. * In PEP 576, just put function pointer to type slot. On the other hand, when implementing callable object with PEP 580, (1) Put PyCCallDef somewhere, (2) Put CCallRoot in instance, (3) put offset of (2) to tp_ccall. * Difference between cc_parent and cc_self are unclear too. I think PEP 580 is understandable only for people who tried to implement method objects. It's complete rewrite of PyCFunction and method_descriptor. But extension author can write extension without knowing implementation of them. > > > It is optimized for implementing custom method object, although > > almost only Cython want the custom method type. > > For the record, Numba also seems interested in the PEP: > https://groups.google.com/a/continuum.io/d/msg/numba-users/2G6k2R92MIM/P-cFKW7xAgAJ > OK, Numba developer interested in: * Supporting FASTCALL for Dispatcher type: PEP 576 is more simple for it as I described above. * Direct C function calling (skip PyObject calling abstraction). While it's not part of PEP 580, it's strong motivation for PEP 580. I want to see PoC of direct C calling. And I think PoC can be implemented without waiting PEP 580. * Cython can have specialization for CyFunction, like it have for CFunction. (Note that Cython doesn't utilize LOAD_METHOD / CALL_METHOD for CFunction too. So lacking support for LOAD_METHOD / CALL_METHOD is not a big problem for now.) * Cython can implement own C signature and embed it in CyFunction. After that, we (including Numba, Cython, and PyPy developers) can discuss how portable C signature can be embedded in PyCCallDef. > > I'm not sure adding such complicated protocol almost only for Cython. > > If CyFunction can be implemented behind PEP 576, it may be better. > > I recall my post > https://mail.python.org/pipermail/python-dev/2018-July/154238.html > explaining the main difference between PEP 576 and PEP 580. I wrote my mail after reading the mail, of course. But it was unclear without reading PEP and implementation carefully. For example, "hook which part" seems meta-discussion to me before reading your implementation. I think only way to understand PEP 580 is reading implementation and imagine how Cython and Numba use it. > > I would like to stress that PEP 580 was designed for maximum > performance, both today and for future extensions (such as calling with > native C types). > I don't know what the word *stress* mean here. (Sorry, I'm not good at English enough for such hard discussion). But I want to see PoC of real benefit of PEP 580, as I said above. > > * PEP 576 and 580 are not strictly mutually exclusive; PEP 576 may be > >accepted in addition to PEP 580 > > I don't think that this is a good idea: you will mostly end up with the > disadvantages of both approaches. > Hm, My point was providing easy and simple way to support FASTCALL in callable object like functools.partial or functools.lru_cache. But it should be discussed after PEP 580. -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
On Tue, 31 Jul 2018 09:27:03 +0200 Jeroen Demeyer wrote: > On 2018-07-31 08:58, Antoine Pitrou wrote: > > I think Stefan is right that we > > should push people towards Cython and alternatives, rather than direct > > use of the C API (which people often fail to use correctly, in my > > experience). > > I know this probably isn't the correct place to bring it up, but I'm > sure that CPython itself could benefit from using Cython. For example, > most of the C extensions in Modules/ could be written in Cython. We don't depend on any third-party Python modules. Adding a Cython dependency for CPython development would be a tough sell. Also, a C extension can be built-in (linked statically into the interpreter), which I think would be hard to do with Cython. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On 2018-07-31 09:36, INADA Naoki wrote: I want to see PoC of direct C calling. To be honest, there is no implementation plan for this yet. I know that several people want this feature, so it makes sense to think about it. For me personally, the main open problem is how to deal with arguments which may be passed both as Python object or as native C type. For example, when doing a function call like f(1,2,3), it may happen that the first argument is really a Python object (so it should be passed as Python int) but that the other two arguments are C integers. And I think PoC can be implemented without waiting PEP 580. For one particular class (say CyFunction), yes. But this feature would be particularly useful for calling between different kinds of C code, for example between Numba and CPython built-ins, or between Pythran and Cython, ... That is why I think it should be implemented as an extension of PEP 580. Anyway, this is a different subject that we should not mix in the discussion about PEP 580 (that is also why I am replying to this specific point separately). Jeroen. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On 2018-07-31 09:36, INADA Naoki wrote: I think PEP 580 is understandable only for people who tried to implement method objects. Is this really a problem? Do we expect that all Python developers can understand all PEPs, especially on a technical subject like this? To give a different example, I would say that PEP 567 is also quite technical and not understandable by people who don't care about about context variables. If PEP 580 is accepted, we can make it very clear in the documentation that this is only meant for implementing fast function/method classes and that ordinary "extension writers" can safely skip that part. For example, you write They should learn PyCCallDef and CCALL_* flags in addition to PyMethodDef and METH_*. but that's not true: they can easily NOT learn those flags, just like they do NOT need to learn about context variables if they don't need them. I would like to stress that PEP 580 was designed for maximum performance, both today and for future extensions (such as calling with native C types). I don't know what the word *stress* mean here. (Sorry, I'm not good at English enough for such hard discussion). But I want to see PoC of real benefit of PEP 580, as I said above. "to stress" = to draw attention to, to make it clear that So, PEP 580 is meant to keep all existing optimizations for functions/methods. It can also be extended in the future (for example, to support direct C calling) by just adding extra flags and structure fields to PyCCallDef. Hm, My point was providing easy and simple way to support FASTCALL in callable object like functools.partial or functools.lru_cache. That can be done easily with only PEP 580. Jeroen. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On Tue, Jul 31, 2018 at 5:46 PM Jeroen Demeyer wrote: > > On 2018-07-31 09:36, INADA Naoki wrote: > > I want to see PoC of direct C calling. > > To be honest, there is no implementation plan for this yet. I know that > several people want this feature, so it makes sense to think about it. Yes. It will be the strong benefit of PEP 580. > > For me personally, the main open problem is how to deal with arguments > which may be passed both as Python object or as native C type. For > example, when doing a function call like f(1,2,3), it may happen that > the first argument is really a Python object (so it should be passed as > Python int) but that the other two arguments are C integers. I don't think it's the main point... At first time, we can just skip direct calling path and use normal Python call. > > > And I think PoC can be implemented without waiting PEP 580. > > For one particular class (say CyFunction), yes. But this feature would > be particularly useful for calling between different kinds of C code, > for example between Numba and CPython built-ins, or between Pythran and > Cython, ... > > That is why I think it should be implemented as an extension of PEP 580. Of course, finally it can be implemented based on PEP 580. But I said "Proof of Concept". Optimize only Cython-to-Cython case is enough for "Proof of Concept". PoC makes it easy to discuss concrete API design for cross implementation call. > > Anyway, this is a different subject that we should not mix in the > discussion about PEP 580 (that is also why I am replying to this > specific point separately). > For me, this is the most important benefit of PEP 580. I can't split it from PEP 580. And I didn't say PoC is needed *before* accepting or discussing PEP 580. I just meant PoC can be implemented without waiting PEP 580 accepted, and I want see it early. Any PEP won't be accepted in few month, because we don't have flow to accept PEPs for now. And I think Victor and Serhiy are key person about these PEPs, but I don't know when they have enough time to review / discuss / make decision about these PEPs. So it's worthless that waiting PEP accepted before start PoC. -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On 2018-07-31 11:12, INADA Naoki wrote: For me, this is the most important benefit of PEP 580. I can't split it from PEP 580. I want PEP 580 to stand by itself. And you say that it is already complicated enough, so we should not mix native C calling into it. PEP 580 is written to allow future extensions like that, but it should be reviewed without those future extensions. Jeroen. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On Tue, Jul 31, 2018 at 6:14 PM Jeroen Demeyer wrote: > > On 2018-07-31 09:36, INADA Naoki wrote: > > I think PEP 580 is understandable only for people who tried to implement > > method objects. > > Is this really a problem? Do we expect that all Python developers can > understand all PEPs, especially on a technical subject like this? > > To give a different example, I would say that PEP 567 is also quite > technical and not understandable by people who don't care about about > context variables. > Please don't "straw man" discussion! I said "So PEP 576/580 are needed only when implementing callable object". I showed example "functools.partial or functools.lru_cache". Using FASTCALL for callable object, instead of tp_call is one of important goal of PEP 576 and 580. So, cleary, ordinal extension author should be able to understand and implement PEP 576 and 580. > If PEP 580 is accepted, we can make it very clear in the documentation > that this is only meant for implementing fast function/method classes > and that ordinary "extension writers" can safely skip that part. For > example, you write > > > They should learn PyCCallDef and CCALL_* flags in addition > > to PyMethodDef and METH_*. > > but that's not true: they can easily NOT learn those flags, just like > they do NOT need to learn about context variables if they don't need them. > Surely, they should understand they must use CCALL_* flags instead of METH_* flags when implementing fast-callable object. > >> I would like to stress that PEP 580 was designed for maximum > >> performance, both today and for future extensions (such as calling with > >> native C types). > >> > > I don't know what the word *stress* mean here. (Sorry, I'm not good at > > English > > enough for such hard discussion). > > But I want to see PoC of real benefit of PEP 580, as I said above. > > "to stress" = to draw attention to, to make it clear that > OK, I'm very drawn attention to that already. > > Hm, My point was providing easy and simple way to support FASTCALL > > in callable object like functools.partial or functools.lru_cache. > > That can be done easily with only PEP 580. > After spent several days to read PEP 580 and your implementation, I think I can implement it. I think it's not easy, but it's not impossible too. But I think we should admit the fact it's complicated than PEP 576. Although I don't think it's important than some benefits of PEP 580. NOTE that I didn't say it's a big problem. I just explained because you said "Can you be more concrete what you find complicated?" -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On Tue, 31 Jul 2018 19:10:48 +0900 INADA Naoki wrote: > > I said "So PEP 576/580 are needed only when implementing callable object". > I showed example "functools.partial or functools.lru_cache". > > Using FASTCALL for callable object, instead of tp_call is one of important > goal > of PEP 576 and 580. > > So, cleary, ordinal extension author should be able to understand and > implement > PEP 576 and 580. No, Jeroen is right here. Ordinary extension authors do not need to understand and implement PEP 576 and 580. First, it's rare for extension types to be callable. Second, even if an extension type is callable, implementing PEP 576 or 580 is only useful if tp_call overhead is really important. In any case, just using tp_call will be as good as it was before. There is no regression, hence it is incorrect to say that we place an additional burden on the shoulders of extension authors. Yes, they can learn an additional PEP to increase tp_call performance; but they don't have to. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On Tue, Jul 31, 2018 at 7:27 PM Antoine Pitrou wrote: > > On Tue, 31 Jul 2018 19:10:48 +0900 > INADA Naoki wrote: > > > > I said "So PEP 576/580 are needed only when implementing callable object". > > I showed example "functools.partial or functools.lru_cache". > > > > Using FASTCALL for callable object, instead of tp_call is one of important > > goal > > of PEP 576 and 580. > > > > So, cleary, ordinal extension author should be able to understand and > > implement > > PEP 576 and 580. > > No, Jeroen is right here. Ordinary extension authors do not need to > understand and implement PEP 576 and 580. Ah, my wording "ordinary" was wrong. But Jeroen was wrong too. He said "PEP 567 is also quite technical and not understandable by people who don't care about context variables." People can skip PEP 567 easily when they don't care about contextvars. On the other hand, *if we accept PEP 576 over PEP 580", C extension author will use it easily. So when comparing PEP 576 and 580, understandability for C extension author is not out of the scope. Note that I didn't claim how important it is. I just compare design of PEP 576 and PEP 580. > > First, it's rare for extension types to be callable. > Second, even if an extension type is callable, implementing PEP 576 or > 580 is only useful if tp_call overhead is really important. > > In any case, just using tp_call will be as good as it was before. > There is no regression, hence it is incorrect to say that we place an > additional burden on the shoulders of extension authors. Yes, they > can learn an additional PEP to increase tp_call performance; but they > don't have to. I agree here. I have grepped tp_call usage and about to send mail explaining "Why FASTCALL for tp_call is not so important". > > Regards > > Antoine. > -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
2018-07-31 8:58 GMT+02:00 Antoine Pitrou : > What exactly in the C API made it slow or non-promising? > >> The C API requires that your implementations make almost all the same >> design choices that CPython made 25 years ago (C structures, memory >> allocators, reference couting, specific GC implementation, GIL, etc.). > > Yes, but those choices are not necessarily bad. I understood that PyPy succeeded to become at least 2x faster than CPython by stopping to use reference counting internally. >> Multiple PyPy developers told me that cpyext remains a blocker issue to use >> PyPy. > > Probably, but we're talking about speeding up CPython here, right? My project has different goals. I would prefer to not make any promise about speed. So speed is not my first motivation, or at least not the only one :-) I also want to make the debug build usable. I also want to allow OS vendors to provide multiple Python versions per OS release: *reduce* the maintenance burden, obviously it will still mean more work. It's a tradeoff depending on the lifetime of your OS and the pressure of customers to get the newest Python :-) FYI Red Hat already provide recent development tools on top of RHEL (and Centos and Fedora) because customers are asking for that. We don't work for free :-) I also want to see more alternatives implementations of Python! I would like to see RustPython succeed! See the latest version of https://pythoncapi.readthedocs.io/ for the full rationale. > If we're talking about making more C extensions PyPy-compatible, that's > a different discussion, For pratical reasons, IMHO it makes sense to put everything in the same "new C API" bag. Obviously, I propose to make many changes, and some of them can be more difficult to implement. My proposal contains many open questions and is made of multiple milestones, with a strong requirement on backward compatibility. > and one where I think Stefan is right that we > should push people towards Cython and alternatives, rather than direct > use of the C API (which people often fail to use correctly, in my > experience). Don't get me wrong: my intent is not to replace Cython. Even if PyPy is pushing hard cffi, many C extensions still use the C API. Maybe if the C API becomes more annoying and require developers to adapt their old code base for the "new C API", some of them will reconsider to use Cython, cffi or something else :-D But backward compatibility is a big part of my plan, and in fact, I expect that porting most C extensions to the new C API will be between "free" and "cheap". Obviously, it depends on how much changes we put in the "new C API" :-) I would like to work incrementally. > But the C API is still useful for specialized uses, *including* for > development tools such as Cython. It seems like http://pythoncapi.readthedocs.io/ didn't explain well my intent. I updated my doc to make it very clear that the "old C API" remains available *on purpose*. The main question is if you will be able to use Cython with the "old C API" on a new "experimental runtime", or if Cython will be stuck at the "regular runtime". https://pythoncapi.readthedocs.io/runtimes.html It's just that for the long term (end of my roadmap), you will have to opt-in for the old C API. > I agree about the overall diagnosis. I just disagree that changing the > C API will open up easy optimization opportunities. Ok, please help me to rephrase the documentation to not make any promise :-) Currently, I wrote: """ Optimization ideas Once the new C API will succeed to hide implementation details, it becomes possible to experiment radical changes in CPython to implement new optimizations. See Experimental runtime. """ https://pythoncapi.readthedocs.io/optimization_ideas.html In my early plan, I wrote "faster runtime". I replaced it with "experimental runtime" :-) Do you think that it's wrong to promise that a smaller C API without implementation details will allow to more easily *experiment* optimizations? > Actually I'd like to see a list of optimizations that you think are > held up by the C API. Hum, let me use the "Tagged pointers" example. Most C functions use "PyObject*" as an opaque C type. Good. But technically, since we give access to fields of C structures, like PyObject.ob_refcnt or PyListObject.ob_item, C extensions currently dereference directly pointers. I'm not convinced that tagged pointers will make CPython way faster. I'm just saying that the C API prevents you to even experiment such change to measure the impact on performance. https://pythoncapi.readthedocs.io/optimization_ideas.html#tagged-pointers-doable For the "Copy-on-Write" idea, the issue is that many macros access directly fields of C structures and so at the machine code, the ABI uses a fixed offset in memory to read data, whereas my plan is to allow each runtime to use a different memory layout, like putting Py_GC elsewhere (or even remove it!!!) and/or put ob_refcnt elsewhere. https
Re: [Python-Dev] Let's change to C API!
2018-07-31 9:27 GMT+02:00 Jeroen Demeyer :
> On 2018-07-31 08:58, Antoine Pitrou wrote:
>>
>> I think Stefan is right that we
>> should push people towards Cython and alternatives, rather than direct
>> use of the C API (which people often fail to use correctly, in my
>> experience).
>
>
> I know this probably isn't the correct place to bring it up, but I'm sure
> that CPython itself could benefit from using Cython. For example, most of
> the C extensions in Modules/ could be written in Cython.
CPython build system has very little dependencies. We even include
vendored copies of third party libraries to make the build even
simpler:
http://pythondev.readthedocs.io/cpython.html#vendored-external-libraries
We try to make CPython build as simple as possible. I'm quite sure
that Cython rely on the stdlib. Would depending on Cython open a
chicken-and-egg issue?
I would be nice to be able to use something to "generate" C
extensions, maybe even from pure Python code. But someone has to work
on a full solution to implement that.
The statu co is that CPython uses C extensions calling directly the C API.
Some people complained that CPython doesn't use its own stable ABI for
its own stable ABI. I concur that it's an issue. Because of that,
nobody noticed that we broke the stable ABI (we did it, multiple
times...). Hum, maybe I should explain that my plan is also try to use
the "new C API" for some C extensions of the stdlib. I'm not sure if
we can do it for all C extensions, since performance matters, and
sometimes we really need to access private fields ("implementation
details").
Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
On Tue, 31 Jul 2018 12:51:23 +0200 Victor Stinner wrote: > 2018-07-31 8:58 GMT+02:00 Antoine Pitrou : > > What exactly in the C API made it slow or non-promising? > > > >> The C API requires that your implementations make almost all the same > >> design choices that CPython made 25 years ago (C structures, memory > >> allocators, reference couting, specific GC implementation, GIL, etc.). > > > > Yes, but those choices are not necessarily bad. > > I understood that PyPy succeeded to become at least 2x faster than > CPython by stopping to use reference counting internally. "I understood that"... where did you get it from? :-) > I also want to make the debug build usable. So I think that we should ask what the ABI differences between debug and non-debug builds are. AFAIK, the two main ones are Py_TRACE_REFS and Py_REF_DEBUG. Are there any others? Honestly, I don't think Py_TRACE_REFS is useful. I don't remember any bug being discovered thanks to it. Py_REF_DEBUG is much more useful. The main ABI issue with Py_REF_DEBUG is not object structure (it doesn't change object structure), it's when a non-debug extension steals a reference (or calls a reference-stealing C API function), because then increments and decrements are unbalanced. > I also want to allow OS vendors to provide multiple Python versions > per OS release: *reduce* the maintenance burden, obviously it will > still mean more work. It's a tradeoff depending on the lifetime of > your OS and the pressure of customers to get the newest Python :-) FYI > Red Hat already provide recent development tools on top of RHEL (and > Centos and Fedora) because customers are asking for that. We don't > work for free :-) OS vendors seem to be doing a fine job AFAICT. And if I want a recent Python I just download Miniconda/Anaconda. > I also want to see more alternatives implementations of Python! I > would like to see RustPython succeed! As long as RustPython gets 10 commits a year, it has no chance of being a functional Python implementation, let alone a successful one. AFAICS it's just a toy project. > > and one where I think Stefan is right that we > > should push people towards Cython and alternatives, rather than direct > > use of the C API (which people often fail to use correctly, in my > > experience). > > Don't get me wrong: my intent is not to replace Cython. Even if PyPy > is pushing hard cffi, many C extensions still use the C API. cffi is a ctypes replacement. It's nice when you want to bind with foreign C code, not if you want tight interaction with CPython objects. > Maybe if the C API becomes more annoying and require developers to > adapt their old code base for the "new C API", some of them will > reconsider to use Cython, cffi or something else :-D I think you don't realize that the C API is *already* annoying. People started with it mostly because there wasn't a better alternative at the time. You don't need to make it more annoying than it already is ;-) Replacing existing C extensions with something else is entirely a developer time/effort problem, not an attractivity problem. And I'm not sure that porting a C extension to a new C API is more reasonable than porting to Cython entirely. > Do you think that it's wrong to promise that a smaller C API without > implementation details will allow to more easily *experiment* > optimizations? I don't think it's wrong. Though as long as CPython itself uses the internal C API, you'll still have a *lot* of code to change before you can even launch a functional interpreter and standard library... It's just that I disagree that removing the C API will make CPython 2x faster. Actually, important modern optimizations for dynamic languages (such as inlining, type specialization, inline caches, object unboxing) don't seem to depend on the C API at all. > >> I have to confess that helping Larry is part of my overall plan. > > > > Which is why I'd like to see Larry chime in here. > > I already talked a little bit with Larry about my plan, but he wasn't > sure that my plan is enough to be able to stop reference counting > internally and move to a different garbage collector. I'm only sure > that it's possible to keep using reference counting for the C API, > since there are solutions for that (ex: maintain a hash table > PyObject* => reference count). Theoretically possible, but the cost of reference counting will go through the roof if you start using a hash table. > Honestly, right now, I'm only convinvced of two things: > > * Larry implementation is very complex and so I doubt that he is going > to succeed. I'm talking about solutions to maintain optimize reference > counting in multithreaded applications. Like his idea of "logs" of > reference counters. Well, you know, *any* solution is going to be very complex. Switching to a full GC for a runtime (CPython) which can allocate hundreds of thousands of objects per second will require a lot of optimization work
Re: [Python-Dev] Let's change to C API!
On 2018-07-31 12:56, Victor Stinner wrote: We try to make CPython build as simple as possible. I'm quite sure that Cython rely on the stdlib. It does rely on modules like "re" and "functools". Would depending on Cython open a chicken-and-egg issue? Yes, that's a problem but it's not unsolvable. For example, we could use the PEP 399 pure Python modules for running Cython. Or we could keep certain "core" C modules (which are used by Cython) implemented directly in C. Note that Cython is not all-or-nothing: it is easy to mix pure Python modules, Cython modules and pure C modules. You can also combine pure C code and Cython code in the same module. Anyway, I know that this is probably not going to happen, but I just wanted to bring it up in case people would find it a great idea. But maybe not many CPython core developers actually know and use Cython? I would be nice to be able to use something to "generate" C extensions, maybe even from pure Python code. Cython has a "pure Python mode" which does exactly that. There are several ways to include typing information, to ensure that a module remains Python-compatible but can be compiled by Cython in an optimized way. Jeroen. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On 2018-07-31 12:10, INADA Naoki wrote: Surely, they should understand they must use CCALL_* flags instead of METH_* flags when implementing fast-callable object. Yes indeed. But implementing a fast-callable object is quite specialized, not something that ordinary extension writers would care about. And if they don't care about performance, tp_call remains supported. More generally: with PEP 580, everything from the documented C API remains supported. So people can write extensions exactly as before. They only need to care about PEP 580 if they want to use the new features that PEP 580 adds (or if they used undocumented internals). Jeroen. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
Antoine: would you mind to subscribe to the capi-sig mailing list? As expected, they are many interesting points discussed here, but I would like to move all C API discussions to capi-sig. I only continue on python-dev since you started here (and ignored my request to start discussing my idea on capi-sig :-)). 2018-07-31 13:55 GMT+02:00 Antoine Pitrou : >> I understood that PyPy succeeded to become at least 2x faster than >> CPython by stopping to use reference counting internally. > > "I understood that"... where did you get it from? :-) I'm quite sure that PyPy developers told me that, but I don't recall who nor when. I don't think that PyPy became 5x faster just because of a single change. But I understand that to be able to implement some optimizations, you first have to remove constraints caused by a design choice like reference counting. For example, PyPy uses different memory allocators depending on the scope and the lifetime of an object. I'm not sure that you can implement such optimization if you are stuck with reference counting. > So I think that we should ask what the ABI differences between debug > and non-debug builds are. Debug build is one use case. Another use case for OS vendors is to compile a C extension once (ex: on Python 3.6) and use it on multiple Python versions (3.7, 3.8, etc.). > AFAIK, the two main ones are Py_TRACE_REFS and Py_REF_DEBUG. Are there > any others? No idea. > Honestly, I don't think Py_TRACE_REFS is useful. I don't remember > any bug being discovered thanks to it. Py_REF_DEBUG is much more > useful. The main ABI issue with Py_REF_DEBUG is not object structure > (it doesn't change object structure), it's when a non-debug extension > steals a reference (or calls a reference-stealing C API function), > because then increments and decrements are unbalanced. About Py_REF_DEBUG:_Py_RefTotal counter is updated at each INCREF/DECREF. _Py_RefTotal is a popular feature of debug build, and I'm not sure how we can update it without replacing Py_INCREF/DECREF macros with function calls. I'm ok to remove/deprecate Py_TRACE_REFS feature if nobody uses it. > OS vendors seem to be doing a fine job AFAICT. And if I want a recent > Python I just download Miniconda/Anaconda. Is it used in production to deploy services? Or is it more used by developers? I never used Anaconda. > cffi is a ctypes replacement. It's nice when you want to bind with > foreign C code, not if you want tight interaction with CPython objects. I have been told that cffi is a different way to do the same thing. Instead of writing C code with the C API glue, only write C code, and then write a cffi binding for it. But I never used Cython nor cffi, so I'm not sure which one is the most appropriate depending on the use case. > I think you don't realize that the C API is *already* annoying. People > started with it mostly because there wasn't a better alternative at the > time. You don't need to make it more annoying than it already is ;-) > > Replacing existing C extensions with something else is entirely a > developer time/effort problem, not an attractivity problem. And I'm > not sure that porting a C extension to a new C API is more reasonable > than porting to Cython entirely. Do you think that it's doable to port numpy to Cython? It's made of 255K lines of C code. A major "rewrite" of such large code base is very difficult since people want to push new things in parallel. Or is it maybe possible to do it incrementally? > It's just that I disagree that removing the C API will make CPython 2x > faster. How can we make CPython 2x faster? Why everybody, except of PyPy, failed to do that? Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
2018-07-31 14:01 GMT+02:00 Jeroen Demeyer : > Anyway, I know that this is probably not going to happen, but I just wanted > to bring it up in case people would find it a great idea. But maybe not many > CPython core developers actually know and use Cython? I know that Yury wants to use Cython for "C extensions" of CPython. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On 2018-07-31 12:10, INADA Naoki wrote: After spent several days to read PEP 580 and your implementation, I think I can implement it. I think it's not easy, but it's not impossible too. The signature of "extended_call_ptr" in PEP 576 is almost the same as the signature of a CCALL_FUNCARG|CCALL_FASTCALL|CCALL_KEYWORDS function in PEP 580 (the only difference is a "self" argument which can be ignored if you don't need it). So, if you can implement it using PEP 576, it's not a big step to implement it using PEP 580. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
On 2018-07-31 15:34, Victor Stinner wrote: But I never used Cython nor cffi, so I'm not sure which one is the most appropriate depending on the use case. Cython is a build-time tool, while cffi is a run-time tool. But Cython does a lot more than just FFI. It is a Python->C compiler which can be used for FFI but also for many other things. A major "rewrite" of such large code base is very difficult since people want to push new things in parallel. Or is it maybe possible to do it incrementally? Yes, that's not a problem: you can easily mix pure Python code, Cython code and C code. I think that this kind of mixing is an important part in Cython's philosophy: for stuff where you don't care about performance: use Python. For most stuff where you do care: use Cython. For very specialized code which cannot easily be translated to Cython: use C. Jeroen. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On Tue, Jul 31, 2018 at 9:55 PM Jeroen Demeyer wrote: > > On 2018-07-31 12:10, INADA Naoki wrote: > > Surely, they should understand they must use CCALL_* flags instead of > > METH_* flags when implementing fast-callable object. > > Yes indeed. But implementing a fast-callable object is quite > specialized, not something that ordinary extension writers would care > about. And if they don't care about performance, tp_call remains supported. > > More generally: with PEP 580, everything from the documented C API > remains supported. So people can write extensions exactly as before. > They only need to care about PEP 580 if they want to use the new > features that PEP 580 adds (or if they used undocumented internals). > > > Jeroen. 100% agree. I never claimed fastcall support for callable object is important. >From my first mail in this thread, I prefer PEP 580, although I was not 100% sure PEP 580 is better. My point (why I'm not 100% sure PEP 580 is better) was simplicity / straightforwardness. I just explained it as "understandability for extension author". Method types (PyCFunction / CyFunction / method_descripter, etc) are callable type. So "make simple and fast callable protocol, then implement complicated method types behind the protocol". seems much straightforward than "Add complicated protocol for method types, then callable can be implemented like method type". If there are no drawback in PEP 576, CCall can be used behind it like: .tp_ccalloffset = offsetof(...), .tp_extcall = PyCCall_ExtCall, // filled by PyType_Read() But PEP 576 is not perfect: * FASTCALL | KEYWORDS is not stable yet, and signature of ext_call is not flexible enough. * LOAD_METHOD support is not straightforward * I want something like CCall anyway, especially for "direct call with C type" optimization in the future. That's why I prefer PEP 580. When I wrote first mail in this thread in last week, I'm about 70% sure. Since I can't find performance critical "tp_call" usage (*), I'm 90% sure for now. (*) Actually speaking, I think tp_new is called more often than tp_call. -- INADA Naoki ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
On 2018-07-31 10:07, Jeroen Demeyer wrote: On 2018-07-31 09:36, INADA Naoki wrote: [snip] I would like to stress that PEP 580 was designed for maximum performance, both today and for future extensions (such as calling with native C types). I don't know what the word *stress* mean here. (Sorry, I'm not good at English enough for such hard discussion). But I want to see PoC of real benefit of PEP 580, as I said above. "to stress" = to draw attention to, to make it clear that In this context, I'd define it as: "to stress" = to emphasize ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
On Tue, 31 Jul 2018 15:34:05 +0200 Victor Stinner wrote: > Antoine: would you mind to subscribe to the capi-sig mailing list? As > expected, they are many interesting points discussed here, but I would > like to move all C API discussions to capi-sig. I only continue on > python-dev since you started here (and ignored my request to start > discussing my idea on capi-sig :-)). Well, I responded to your e-mail discussion thread. I see more messages in this thread here than on capi-sig. ;-) > For example, PyPy uses different memory allocators depending on the > scope and the lifetime of an object. I'm not sure that you can > implement such optimization if you are stuck with reference counting. But what does reference counting have to do with memory allocators exactly? > > OS vendors seem to be doing a fine job AFAICT. And if I want a recent > > Python I just download Miniconda/Anaconda. > > Is it used in production to deploy services? Or is it more used by > developers? I never used Anaconda. I don't know, but there's no hard reason why you couldn't use it to deploy services (though some people may prefer Docker or other technologies). > > I think you don't realize that the C API is *already* annoying. People > > started with it mostly because there wasn't a better alternative at the > > time. You don't need to make it more annoying than it already is ;-) > > > > Replacing existing C extensions with something else is entirely a > > developer time/effort problem, not an attractivity problem. And I'm > > not sure that porting a C extension to a new C API is more reasonable > > than porting to Cython entirely. > > Do you think that it's doable to port numpy to Cython? It's made of > 255K lines of C code. Numpy is a bit special as it exposes its own C API, so porting it entirely to Cython would be difficult (how do you expose a C macro in Cython?). Also, internally it has a lot of macro-generated code for specialized loop implementations (metaprogramming in C :-)). I suppose some bits could be (re)written in Cython. Actually, the numpy.random module is already a Cython module. > > It's just that I disagree that removing the C API will make CPython 2x > > faster. > > How can we make CPython 2x faster? Why everybody, except of PyPy, > failed to do that? Because PyPy spent years working full time on a JIT compiler. It's also written in (a dialect of) Python, which helps a lot with experimenting and building abstractions, compared to C or even C++. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
On Tue, Jul 31, 2018 at 7:35 AM Victor Stinner wrote: > 2018-07-31 14:01 GMT+02:00 Jeroen Demeyer : > > Anyway, I know that this is probably not going to happen, but I just wanted > > to bring it up in case people would find it a great idea. But maybe not many > > CPython core developers actually know and use Cython? > > I know that Yury wants to use Cython for "C extensions" of CPython. FWIW, Brett, Barry, and I were discussing the (small) possibility of adding a minimal tool based on Cython to the repo. I've opened a thread about this on capi-sig. Replies there, please. -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
I replied on capi-sig. 2018-07-31 18:03 GMT+02:00 Antoine Pitrou : > On Tue, 31 Jul 2018 15:34:05 +0200 > Victor Stinner wrote: >> Antoine: would you mind to subscribe to the capi-sig mailing list? As >> expected, they are many interesting points discussed here, but I would >> like to move all C API discussions to capi-sig. I only continue on >> python-dev since you started here (and ignored my request to start >> discussing my idea on capi-sig :-)). > > Well, I responded to your e-mail discussion thread. I see more > messages in this thread here than on capi-sig. ;-) > >> For example, PyPy uses different memory allocators depending on the >> scope and the lifetime of an object. I'm not sure that you can >> implement such optimization if you are stuck with reference counting. > > But what does reference counting have to do with memory allocators > exactly? > >> > OS vendors seem to be doing a fine job AFAICT. And if I want a recent >> > Python I just download Miniconda/Anaconda. >> >> Is it used in production to deploy services? Or is it more used by >> developers? I never used Anaconda. > > I don't know, but there's no hard reason why you couldn't use it to > deploy services (though some people may prefer Docker or other > technologies). > >> > I think you don't realize that the C API is *already* annoying. People >> > started with it mostly because there wasn't a better alternative at the >> > time. You don't need to make it more annoying than it already is ;-) >> > >> > Replacing existing C extensions with something else is entirely a >> > developer time/effort problem, not an attractivity problem. And I'm >> > not sure that porting a C extension to a new C API is more reasonable >> > than porting to Cython entirely. >> >> Do you think that it's doable to port numpy to Cython? It's made of >> 255K lines of C code. > > Numpy is a bit special as it exposes its own C API, so porting it > entirely to Cython would be difficult (how do you expose a C macro in > Cython?). Also, internally it has a lot of macro-generated code for > specialized loop implementations (metaprogramming in C :-)). > > I suppose some bits could be (re)written in Cython. Actually, the > numpy.random module is already a Cython module. > >> > It's just that I disagree that removing the C API will make CPython 2x >> > faster. >> >> How can we make CPython 2x faster? Why everybody, except of PyPy, >> failed to do that? > > Because PyPy spent years working full time on a JIT compiler. It's also > written in (a dialect of) Python, which helps a lot with experimenting > and building abstractions, compared to C or even C++. > > Regards > > Antoine. > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update on Cygwin support (was: Clarifying Cygwin support in CPython)
On Mon, Jul 30, 2018 at 5:26 PM Nick Coghlan wrote: > > On 26 July 2018 at 02:13, Erik Bray wrote: > > I think a new approach that might be more practical for actually > > getting this platform re-supported, is to go ahead and add a CI build, > > and just skip all known failing test modules. This is what I've done > > in a new PR to add a Cygwin build on AppVeyor: > > > > https://github.com/python/cpython/pull/8463 > > > > This is far from ideal of course, and should not mean the platform is > > "supported". But now I and others can go through and fix the > > remaining test failures, re-enable those modules in the CI > > configuration, and actually obtain some meaningful results, which will > > hopefully encourage the core committers to accept fixes for the > > platform. > > I believe the PEP 538 & 540 locale handling tests are amongst those > that are still a bit sketchy (or outright broken?) on Cygwin, and I > think having an advisory CI bot would definitely help with that. > (Cygwin/MinGW are an interesting hybrid that really highlight the fact > that neither "POSIX implies not Windows" nor "Windows implies the > Win32 API" are entirely valid assumptions) Yes, I believe those tests are still a little broken, though the improvements you last made to them should be helpful in getting it passing. I haven't looked at it in a few months. Indeed, it makes for some interesting broken assumptions. Another example I've encountered recently is because Cygwin uses the posixpath module, all handling of Windows-style paths is broken. This is fine, because in general a developer should *not* be using Windows paths on Cygwin; POSIX paths only. However, the fact remains that Cygwin does (mostly) transparently support Windows paths at the system level, so some things work. But if a user runs a script that happens to be written in Python, but passes Windows paths to it, say, as command-line arguments, it may or may not work. If the path is passed directly to open(), no problem. But if it goes through os.path.abspath for example things blow up. I'm undecided as to whether this is something that developers writing applications that support Cygwin need to handle, or if this is something that could work better on the Python end as well. I lean toward the former, but I also wonder if there isn't more that could be done in the stdlib to improve this issue as well. In the meantime I wrote pycygwin [1] to help with these sorts of issues in my own software. > So your suggested approach seems like a plausible way forward to me. > > The main potentially viable alternative I see would be to set up the > *buildbot* first, and then devote the custom builder branch to the > task of Cygwin testing for a while: > https://devguide.python.org/buildbots/#custom-builders > > However, I think the overall UX of that would be worse than going down > the advisory CI path (especially since it wouldn't really help with > the aspect of parallel development introducing new Cygwin failures). Exactly. And at least for starters I might have to push to buildbot-custom, because without a few minimal fixes in place CPython currently does not build successfully at all on Cygwin, which makes the buildbot a little unhelpful. But Zach is already in touch with me about getting a buildbot worker set up anyways. I agree it's still good to have, and will be more and more useful as I get the requisite fixes merged... Thanks, E [1] http://pycygwin.readthedocs.io/en/latest/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
Well, I tried to subscribe to capi-sig, but I didn't get a confirmation e-mail. Regards Antoine. On Tue, 31 Jul 2018 18:25:25 +0200 Victor Stinner wrote: > I replied on capi-sig. > > 2018-07-31 18:03 GMT+02:00 Antoine Pitrou : > > On Tue, 31 Jul 2018 15:34:05 +0200 > > Victor Stinner wrote: > >> Antoine: would you mind to subscribe to the capi-sig mailing list? As > >> expected, they are many interesting points discussed here, but I would > >> like to move all C API discussions to capi-sig. I only continue on > >> python-dev since you started here (and ignored my request to start > >> discussing my idea on capi-sig :-)). > > > > Well, I responded to your e-mail discussion thread. I see more > > messages in this thread here than on capi-sig. ;-) > > > >> For example, PyPy uses different memory allocators depending on the > >> scope and the lifetime of an object. I'm not sure that you can > >> implement such optimization if you are stuck with reference counting. > > > > But what does reference counting have to do with memory allocators > > exactly? > > > >> > OS vendors seem to be doing a fine job AFAICT. And if I want a recent > >> > Python I just download Miniconda/Anaconda. > >> > >> Is it used in production to deploy services? Or is it more used by > >> developers? I never used Anaconda. > > > > I don't know, but there's no hard reason why you couldn't use it to > > deploy services (though some people may prefer Docker or other > > technologies). > > > >> > I think you don't realize that the C API is *already* annoying. People > >> > started with it mostly because there wasn't a better alternative at the > >> > time. You don't need to make it more annoying than it already is ;-) > >> > > >> > Replacing existing C extensions with something else is entirely a > >> > developer time/effort problem, not an attractivity problem. And I'm > >> > not sure that porting a C extension to a new C API is more reasonable > >> > than porting to Cython entirely. > >> > >> Do you think that it's doable to port numpy to Cython? It's made of > >> 255K lines of C code. > > > > Numpy is a bit special as it exposes its own C API, so porting it > > entirely to Cython would be difficult (how do you expose a C macro in > > Cython?). Also, internally it has a lot of macro-generated code for > > specialized loop implementations (metaprogramming in C :-)). > > > > I suppose some bits could be (re)written in Cython. Actually, the > > numpy.random module is already a Cython module. > > > >> > It's just that I disagree that removing the C API will make CPython 2x > >> > faster. > >> > >> How can we make CPython 2x faster? Why everybody, except of PyPy, > >> failed to do that? > > > > Because PyPy spent years working full time on a JIT compiler. It's also > > written in (a dialect of) Python, which helps a lot with experimenting > > and building abstractions, compared to C or even C++. > > > > Regards > > > > Antoine. > > ___ > > Python-Dev mailing list > > [email protected] > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
On Tue, 31 Jul 2018 at 10:32 Antoine Pitrou wrote: > > Well, I tried to subscribe to capi-sig, but I didn't get a > confirmation e-mail. > I subscribed yesterday without issue. I would email postmaster to try and find out what happened. -Brett > > Regards > > Antoine. > > > On Tue, 31 Jul 2018 18:25:25 +0200 > Victor Stinner wrote: > > I replied on capi-sig. > > > > 2018-07-31 18:03 GMT+02:00 Antoine Pitrou : > > > On Tue, 31 Jul 2018 15:34:05 +0200 > > > Victor Stinner wrote: > > >> Antoine: would you mind to subscribe to the capi-sig mailing list? As > > >> expected, they are many interesting points discussed here, but I would > > >> like to move all C API discussions to capi-sig. I only continue on > > >> python-dev since you started here (and ignored my request to start > > >> discussing my idea on capi-sig :-)). > > > > > > Well, I responded to your e-mail discussion thread. I see more > > > messages in this thread here than on capi-sig. ;-) > > > > > >> For example, PyPy uses different memory allocators depending on the > > >> scope and the lifetime of an object. I'm not sure that you can > > >> implement such optimization if you are stuck with reference > counting. > > > > > > But what does reference counting have to do with memory allocators > > > exactly? > > > > > >> > OS vendors seem to be doing a fine job AFAICT. And if I want a > recent > > >> > Python I just download Miniconda/Anaconda. > > >> > > >> Is it used in production to deploy services? Or is it more used by > > >> developers? I never used Anaconda. > > > > > > I don't know, but there's no hard reason why you couldn't use it to > > > deploy services (though some people may prefer Docker or other > > > technologies). > > > > > >> > I think you don't realize that the C API is *already* annoying. > People > > >> > started with it mostly because there wasn't a better alternative at > the > > >> > time. You don't need to make it more annoying than it already is > ;-) > > >> > > > >> > Replacing existing C extensions with something else is entirely a > > >> > developer time/effort problem, not an attractivity problem. And I'm > > >> > not sure that porting a C extension to a new C API is more > reasonable > > >> > than porting to Cython entirely. > > >> > > >> Do you think that it's doable to port numpy to Cython? It's made of > > >> 255K lines of C code. > > > > > > Numpy is a bit special as it exposes its own C API, so porting it > > > entirely to Cython would be difficult (how do you expose a C macro in > > > Cython?). Also, internally it has a lot of macro-generated code for > > > specialized loop implementations (metaprogramming in C :-)). > > > > > > I suppose some bits could be (re)written in Cython. Actually, the > > > numpy.random module is already a Cython module. > > > > > >> > It's just that I disagree that removing the C API will make CPython > 2x > > >> > faster. > > >> > > >> How can we make CPython 2x faster? Why everybody, except of PyPy, > > >> failed to do that? > > > > > > Because PyPy spent years working full time on a JIT compiler. It's > also > > > written in (a dialect of) Python, which helps a lot with experimenting > > > and building abstractions, compared to C or even C++. > > > > > > Regards > > > > > > Antoine. > > > ___ > > > Python-Dev mailing list > > > [email protected] > > > https://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
Antoine Pitrou schrieb am 31.07.2018 um 09:45: > On Tue, 31 Jul 2018 09:27:03 +0200 > Jeroen Demeyer wrote: >> On 2018-07-31 08:58, Antoine Pitrou wrote: >>> I think Stefan is right that we >>> should push people towards Cython and alternatives, rather than direct >>> use of the C API (which people often fail to use correctly, in my >>> experience). >> >> I know this probably isn't the correct place to bring it up, but I'm >> sure that CPython itself could benefit from using Cython. For example, >> most of the C extensions in Modules/ could be written in Cython. > > We don't depend on any third-party Python modules. Adding a Cython > dependency for CPython development would be a tough sell. I don't really want to get into that discussion (it's more about processes than arguments), but let me note that the CPython development already has a couple of dependencies, such as github and its bots, or tools like argument clinic (admittedly included), make and a C compiler (not included), and a text editor. It's not like it's free of tools that help in writing and maintaining the code. That's pretty much the level at which I also see Cython. It's more complex than argument clinic, but it otherwise serves a similar need. > Also, a C extension can be built-in (linked statically into the > interpreter), which I think would be hard to do with Cython. Someone recently contributed a feature of hiding the pyinit function for the embedding case, so people do these things already. This could use the normal inittab mechanism, for example. What I think you might be referring to is that Cython modules require the CPython runtime to be initialised to a certain extent, so you couldn't implement "sys" in Cython, for example. But Jeroen is right, Cython should be a viable option for (most of?) the extension modules in the stdlib. Whether the CPython core devs would accept it in their workflow or not is a totally different question. Stefan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
Jeroen Demeyer schrieb am 31.07.2018 um 14:01: > On 2018-07-31 12:56, Victor Stinner wrote: >> I would be nice to be able to use something to "generate" C >> extensions, maybe even from pure Python code. > > Cython has a "pure Python mode" which does exactly that. There are several > ways to include typing information, to ensure that a module remains > Python-compatible but can be compiled by Cython in an optimized way. FWIW, modules like difflib can easily be speed up by factors when compiling and optimising them with Cython, without giving up the Python syntax compatibility. I just gave a Cython talk at EuroPython last week where I used difflib as one of my examples. Stefan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] New _Py_InitializeFromConfig() function (PEP 432)
Hi, I finished my work on the _PyCoreConfig structure: it's a C structure in Include/pystate.h which has many fields used to configure Python initialization. In Python 3.6 and older, these parameters were scatted around the code, and it was hard to get an exhaustive list of it. This work is linked to the Nick Coghlan's PEP 432 "Restructuring the CPython startup sequence": https://www.python.org/dev/peps/pep-0432/ Right now, the new API is still private. Nick Coghlan splitted the initialization in two parts: "core" and "main". I'm not sure that this split is needed. We should see what to do, but it would be nice to make the _PyCoreConfig API public! IMHO it's way better than the old way to configuration Python initialization. -- It is now possible to only use _PyCoreConfig to initialize Python: it overrides old ways to configure Python like environment variables (ex: PYTHONPATH), global configuration variables (ex: Py_BytesWarningFlag) and C functions (ex: Py_SetProgramName()). I added tests to test_embed on the different ways to configure Python initialization: * environment variables (ex: PYTHONPATH) * global configuration variables (ex: Py_BytesWarningFlag) and C functions (ex: Py_SetProgramName()) * _PyCoreConfig I found and fixed many issues when writing these tests :-) Reading the current configuration, _PyCoreConfig_Read(), no longer changes the configuration. Now the code to read the configuration and the code to apply the configuration is properly separated. The work is not fully complete, there are a few remaining corner cases and some parameters (ex: Py_FrozenFlag) which cannot be set by _PyCoreConfig yet. My latest issue used to work on this API: https://bugs.python.org/issue34170 I had to refactor a lot of code to implement all of that. -- The problem is that Python 3.7 got the half-baked implementation, and it caused issues: * Calling Py_Main() after Py_Initialize() fails with a fatal error on Python 3.7.0 https://bugs.python.org/issue34008 * PYTHONOPTIMIZE environment variable is ignored by Py_Initialize() https://bugs.python.org/issue34247 I fixed the first issue, I'm now working on the second one to see how it can be fixed. Other option would be to backport the code from master to the 3.7 branch, since the code in master has a way better design. But it requires to backport a lot of changes. I'm not sure yet what is the best option. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accessing mailing list archives
Would it be possible to normalize by the number of mailing list members and also by "active" members? The latter would be tricky to define. On Mon, Jul 30, 2018 at 3:29 PM Victor Stinner wrote: > Hi Bob, > > I wrote a basic script to compute the number of emails per PEP. It > requires to download gzipped mbox files from the web page of archives per > month, then ungzip them: > > https://github.com/vstinner/misc/blob/master/python/parse_mailman_mbox_peps.py > > Results: > https://mail.python.org/pipermail/python-committers/2018-April/005310.html > > Victor > > Le lundi 30 juillet 2018, Bob Purvy a écrit : > > hi all, > > I've been trying to figure out how to access the archives > programmatically. I'm sure this is easy once you know, but googling various > things hasn't worked. What I want to do is graph the number of messages > about PEP 572 by time. (or has someone already done that?) > > I installed GNU Mailman, and downloaded the gzip'ed archives for a > number of months and unzipped them, and I suspect that there's some way to > get them all into a single database, but it hasn't jumped out at me. If I > count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines > in one of those text files, I get slightly different numbers for each. > > Alternatively, they're maybe already in a database, and I just need API > access to do the querying? Can someone help me out? > > Bob ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mike%40selik.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accessing mailing list archives
Feel free to modify the script to make your own statistics ;-) Victor 2018-08-01 0:57 GMT+02:00 Michael Selik : > Would it be possible to normalize by the number of mailing list members and > also by "active" members? The latter would be tricky to define. > > On Mon, Jul 30, 2018 at 3:29 PM Victor Stinner wrote: >> >> Hi Bob, >> >> I wrote a basic script to compute the number of emails per PEP. It >> requires to download gzipped mbox files from the web page of archives per >> month, then ungzip them: >> >> https://github.com/vstinner/misc/blob/master/python/parse_mailman_mbox_peps.py >> >> Results: >> https://mail.python.org/pipermail/python-committers/2018-April/005310.html >> >> Victor >> >> Le lundi 30 juillet 2018, Bob Purvy a écrit : >> > hi all, >> > I've been trying to figure out how to access the archives >> > programmatically. I'm sure this is easy once you know, but googling various >> > things hasn't worked. What I want to do is graph the number of messages >> > about PEP 572 by time. (or has someone already done that?) >> > I installed GNU Mailman, and downloaded the gzip'ed archives for a >> > number of months and unzipped them, and I suspect that there's some way to >> > get them all into a single database, but it hasn't jumped out at me. If I >> > count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines >> > in one of those text files, I get slightly different numbers for each. >> > Alternatively, they're maybe already in a database, and I just need API >> > access to do the querying? Can someone help me out? >> > Bob ___ >> Python-Dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/mike%40selik.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accessing mailing list archives
On 30Jul2018 13:40, Bob Purvy wrote: I've been trying to figure out how to access the archives programmatically. I'm sure this is easy once you know, but googling various things hasn't worked. What I want to do is graph the number of messages about PEP 572 by time. (or has someone already done that?) I installed GNU Mailman, and downloaded the gzip'ed archives for a number of months and unzipped them, and I suspect that there's some way to get them all into a single database, but it hasn't jumped out at me. If I count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines in one of those text files, I get slightly different numbers for each. Alternatively, they're maybe *already* in a database, and I just need API access to do the querying? Can someone help me out? Like Victor, I download mailing list archives. Between pulling them in and also subscribing, ideally I get a complete history in my "python" mail folder. Likewise for other lists. The mailman archives are UNIX mbox files, compressed, with a bit of header munging (to make address harvesting harder). You can concatenate them and uncompress and reverse the munging like this: cat *.gz | gunzip | fix-mail-dates --mbox | un-at- where fix-mail-dates is here: https://bitbucket.org/cameron_simpson/css/src/tip/bin/fix-mail-dates and un-at- is here: https://bitbucket.org/cameron_simpson/css/src/tip/bin/un-at- and the output is a nice UNIX mbox file. You can load that into most mail readers or parse it with Python's email modules (in the stdlib). It should be easy enough to scan such a thing and count header contents etc. Ignore the "From " line content, prefer the "From:" header. (Separate messages on "From " of course, just don't grab email addresses from it.) Cheers, Cameron Simpson ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Confused on git commit tree about Lib/datetime.py
I found a interesting issue when checking the Lib/datetime.py implementation in python3 This patch is introduced by cf86e368ebd17e10f68306ebad314eea31daaa1e [0]. But if you check the github page[0], or using git tag --contains, you will find v2.7.x includes this commit too. $ git tag --contains cf86e368ebd17e10f68306ebad314eea31daaa1e 3.2 v2.7.10 v2.7.10rc1 v2.7.11 v2.7.11rc1 ... whereas, if you check the v2.7.x code base, nothing is found $ git log v2.7.4 -- Lib/datetime.py I guess it maybe a git tool bug, or the commit tree is messed up. Is there any guys could explain this situation? [0] https://github.com/python/cpython/commit/cf86e368ebd17e10f68306ebad314eea31daaa1e -- Regards, Jeffrey Zhang Blog: http://xcodest.me ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Confused on git commit tree about Lib/datetime.py
On Wed, Aug 1, 2018 at 1:16 PM, Jeffrey Zhang wrote: > I found a interesting issue when checking the Lib/datetime.py implementation > in python3 > > This patch is introduced by cf86e368ebd17e10f68306ebad314eea31daaa1e [0]. > But if you > check the github page[0], or using git tag --contains, you will find v2.7.x > includes this commit too. > > $ git tag --contains cf86e368ebd17e10f68306ebad314eea31daaa1e > 3.2 > v2.7.10 > v2.7.10rc1 > v2.7.11 > v2.7.11rc1 > ... > > whereas, if you check the v2.7.x code base, nothing is found > > $ git log v2.7.4 -- Lib/datetime.py > > > I guess it maybe a git tool bug, or the commit tree is messed up. Is there > any guys could explain this > situation? I suppose you could say that the commit tree is "messed up", in a sense, but it's not truly messed up, just a little odd. It's a consequence of the way merges have been done in the CPython repository. Nothing is actually broken, except for the ability to track down a commit the way you're doing. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
