[Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-05 Thread Jeroen Demeyer
Hello all, I have updated PEP 575 in response to some posts on this mailing list and to some discussions in person with the core Cython developers. See https://www.python.org/dev/peps/pep-0575/ The main differences with respect to the previous version are: * "builtin_function" was renamed to

[Python-Dev] PEP 575 (Unifying function/method classes) update

2018-04-27 Thread Jeroen Demeyer
Hello all, I have updated PEP 575 and its reference implementation. See https://www.python.org/dev/peps/pep-0575/ The main differences with respect to the previous version are: * METH_PASS_FUNCTION now passes the function *in addition* to self (previously, it was passed *instead* of self). *

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-08 Thread Jeroen Demeyer
On 2018-05-06 09:35, Nick Coghlan wrote: Thanks for this update Jeroen! If it doesn't come up otherwise, I'll try to claim one of the lightning talk slots at the Language Summit to discuss this with folks in person :) Sounds great! I'd love to hear what people think. As an example of how the

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-15 Thread Jeroen Demeyer
On 2018-05-14 22:38, Petr Viktorin wrote: Why are these flags added? I made a minor edit to the PEP to remove those flags: https://github.com/python/peps/pull/649 ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] PEP: 576 Title: Rationalize Built-in function classes

2018-05-20 Thread Jeroen Demeyer
On 2018-05-19 11:15, mark wrote: PEP 576 aims to fulfill the same goals as PEP 575 (this is a copy of my comments on GitHub before this PEP was official) **Performance** Most importantly, changing bound methods of extension types from builtin_function_or_method to bound_method will yield a

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-20 Thread Jeroen Demeyer
On 2018-05-19 15:29, Nick Coghlan wrote: That's not how code reviews work, as their complexity is governed by the number of lines changed (added/removed/modified), not just the number of lines that are left at the end. Of course, you are right. I didn't mean literally that only the end result

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-17 Thread Jeroen Demeyer
On 2018-05-16 17:31, Petr Viktorin wrote: Less disruptive changes tend to have a better backwards compatibility story. A less intertwined change makes it easier to revert just a single part, in case that becomes necessary. I'll just repeat what I said in a different post on this thread: we can

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-15 Thread Jeroen Demeyer
On 2018-05-14 22:38, Petr Viktorin wrote: Why are these flags added? They aren't free – the space of available flags is not infinite. If something (Cython?) needs eight of them, it would be nice to mention the use case, at least as an example. What should Python do with a m_methods entry that

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-15 Thread Jeroen Demeyer
On 2018-05-14 19:56, Petr Viktorin wrote: It does quite a lot of things, and the changes are all intertwined, which will make it hard to get reviewed and accepted. The problem is that many things *are* already intertwined currently. You cannot deal with functions without involving methods for

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-15 Thread Jeroen Demeyer
On 2018-05-15 18:36, Petr Viktorin wrote: What is your ultimate use case? (I'll just answer this one question now and reply to the more technical comments in another thread) My ultimate use case is being able to implement functions and methods which are (A) equally fast as the existing

Re: [Python-Dev] PEP: 576 Title: Rationalize Built-in function classes

2018-05-22 Thread Jeroen Demeyer
For the record: the only reason that I replied on GitHub was because the proposal was not yet posted (as far as I know) to any mailing list. Typically, a post is made to a mailing list more or less at the same time as creating the PEP. In this case, there was a delay of a few days, maybe also

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-16 Thread Jeroen Demeyer
On 2018-05-16 17:31, Petr Viktorin wrote: The larger a change is, the harder it is to understand I already disagree here... I'm afraid that you are still confusing the largeness of the *change* with the complexity of the *result* after the change was implemented. A change that *removes*

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-15 Thread Jeroen Demeyer
On 2018-05-15 18:36, Petr Viktorin wrote: Naturally, large-scale changes have less of a chance there. Does it really matter that much how large the change is? I think you are focusing too much on the change instead of the end result. As I said in my previous post, I could certainly make

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-18 Thread Jeroen Demeyer
On 2018-06-18 15:09, Victor Stinner wrote: 2) we implemented a lot of other optimizations which made calls faster without having to touch tp_call nor tp_fastcall. And that's a problem because these optimizations typically only work for specific classes. My PEP wants to replace those by

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-17 Thread Jeroen Demeyer
Hello, I have been working on a slightly different PEP to use a new type slot tp_ccalloffset instead the base_function base class. You can see the work in progress here: https://github.com/jdemeyer/PEP-ccall By creating a new protocol that each class can implement, there is a full

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-17 Thread Jeroen Demeyer
On 2018-06-17 14:50, Ronald Oussoren wrote: This looks interesting. Why did you add a tp_ccalloffset slot to the type with the actual information in instances instead of storing the information in a slot? Think of built-in functions. Every built-in function is a different callable and calls

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-18 Thread Jeroen Demeyer
On 2018-06-18 03:34, INADA Naoki wrote: Victor had tried to add `tp_fastcall` slot, but he suspended his effort because it's benefit is not enough for it's complexity. https://bugs.python.org/issue29259 I has a quick look at that patch and it's really orthogonal to what I'm proposing. I'm

Re: [Python-Dev] Stable ABI

2018-06-01 Thread Jeroen Demeyer
On 2018-06-01 17:18, Nathaniel Smith wrote: Unfortunately, very few people use the stable ABI currently, so it's easy for things like this to get missed. So there are no tests for the stable ABI in Python? ___ Python-Dev mailing list

[Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
Hello, On https://github.com/python/cpython/pull/7909 I encountered friction for a PR which I expected to be uncontroversial: it just moves some code without changing any functionality. So basically my question is: is there some CPython policy *against* refactoring code to make it easier to

Re: [Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
On 2018-06-26 13:54, Ivan Pozdeev via Python-Dev wrote: This is exactly what that the YAGNI principle is about, and Inada was right to point to it. Until you have an immediate practical need for something, you don't really know the shape and form for it that you will be the most comfortable

Re: [Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
On 2018-06-26 13:54, INADA Naoki wrote: Real need is important than my preference. If it is needed PEP 580, I'm OK. Of course it's not needed. I never claimed that it was. I think it's *nice to have* right now and slightly more *nice to have* when changes/additions are made to call.c in the

Re: [Python-Dev] About [].append == [].append

2018-06-23 Thread Jeroen Demeyer
On 2018-06-23 03:50, Steven D'Aprano wrote: I think it is more important that builtin methods and Python methods behave the same. +1 This inconsistency is the *real* problem here. It's one little extra complication to merging those classes (which was proposed in PEP 575, 576 and 579).

[Python-Dev] PEP 580 (C call protocol) draft implementation

2018-06-22 Thread Jeroen Demeyer
Hello all, I have a first draft implementation of PEP 580 (introducing the C call protocol): https://github.com/jdemeyer/cpython/tree/pep580 Almost all tests pass, only test_gdb and test_pydoc fail for me. I still have to fix those. Jeroen. ___

[Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

2018-06-20 Thread Jeroen Demeyer
Hello, Let me present PEP 579 and PEP 580. PEP 579 is an informational meta-PEP, listing some of the issues with functions/methods implemented in C. The idea is to create several PEPs each fix some part of the issues mentioned in PEP 579. PEP 580 is a standards track PEP to introduce a new

Re: [Python-Dev] C-level calling

2018-06-20 Thread Jeroen Demeyer
On 2018-06-20 08:00, Stefan Behnel wrote: Just to add another bit of background on top of the current discussion, there is an idea around, especially in the scipy/big-data community, (and I'm not giving any guarantees here that it will lead to a PEP + implementation, as it depends on people's

Re: [Python-Dev] PEP 576

2018-06-26 Thread Jeroen Demeyer
On 2018-06-26 21:43, Mark Shannon wrote: https://github.com/markshannon/pep-576 Your version of PEP 576 looks very different from the "official" PEP 576 at https://www.python.org/dev/peps/pep-0576/ So can you please make a pull request to https://github.com/python/peps/pulls Also feel free

Re: [Python-Dev] PEP 576

2018-06-27 Thread Jeroen Demeyer
On 2018-06-26 21:43, Mark Shannon wrote: https://github.com/markshannon/pep-576 This actually looks close to Victor Stinner's bpo-29259. But instead of storing the function pointer in the class, you're storing it in the instance. One concern that I have is that this might lead to code

Re: [Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
On 2018-06-27 00:02, Guido van Rossum wrote: And TBH a desire to refactor a lot of code is often a sign of a relatively new contributor who hasn't learned their way around the code yet, so they tend to want to make the code follow their understanding rather than letting their understanding

Re: [Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
On 2018-06-26 13:54, INADA Naoki wrote: ​No, YAGNI is posted by someone and they removed their comment. Sorry for that, I misunderstood the email that GitHub sent me. ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
On 2018-06-26 13:11, Ivan Pozdeev via Python-Dev wrote: AFAICS, your PR is not a strict improvement What does "strict improvement" even mean? Many changes are not strict improvements, but still useful to have. Inada pointed me to YAGNI

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

2018-04-26 Thread Jeroen Demeyer
- In Python code, __objclass__ should be the defining class, not the module. Indeed. My idea would be to add an accessor __parent__ returning the m_parent field (whatever it is) and then implement __objclass__ as something like: @property def __objclass__(self): parent = getattr(self,

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-30 Thread Jeroen Demeyer
On 2018-04-30 15:38, Mark Shannon wrote: While a unified *interface* makes sense, a unified class hierarchy and implementation, IMO, do not. The main reason for the common base class is performance: in the bytecode interpreter, when we call an object, CPython currently has a special case for

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-05-03 Thread Jeroen Demeyer
On 2018-05-03 11:30, Victor Stinner wrote: Please don't queue backward incompatible changes for Python 4.0. You should use the regular deprecation process. I don't really see how that can be done here. As Stefan said The problem is that this change does not really fit into the deprecation

[Python-Dev] Comparing PEP 576 and PEP 580

2018-07-03 Thread Jeroen Demeyer
Hello all, in order to make reviewing PEP 576/580 easier and possibly take some ideas from one PEP to the other, let me state the one fundamental difference between these PEPs. There are many details in both PEPs that can still change, so I'm focusing on what I think is the big structural

Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

2018-06-21 Thread Jeroen Demeyer
On 2018-06-21 11:22, Victor Stinner wrote: https://www.python.org/dev/peps/pep-0580/#the-c-call-protocol CCALL_VARARGS: cc_func(PyObject *self, PyObject *args) If we add a new calling convention This is not a *new* calling convention, it's the *existing* calling convention for METH_VARARGS.

Re: [Python-Dev] About [].append == [].append

2018-06-21 Thread Jeroen Demeyer
On 2018-06-21 13:33, Ivan Pozdeev via Python-Dev wrote: First, tell us what problem you're solving. There is no specific problem I want to solve here. I just noticed an inconsistency and I wondered if it would be OK to change the implementation of comparisons of builtin_function_or_method

[Python-Dev] About [].append == [].append

2018-06-21 Thread Jeroen Demeyer
Currently, we have: >>> [].append == [].append False However, with a Python class: >>> class List(list): ... def append(self, x): super().append(x) >>> List().append == List().append True In the former case, __self__ is compared using "is" and in the latter case, it is compared using

Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

2018-06-20 Thread Jeroen Demeyer
On 2018-06-20 16:42, Antoine Pitrou wrote: I'm wondering what amount of code and debugging is needed for, say, Cython or Numba to implement that protocol as a caller, without going through the C API's indirections (for performance). The goal is to have a really fast C API without a lot of

Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

2018-06-20 Thread Jeroen Demeyer
On 2018-06-20 16:09, Antoine Pitrou wrote: But there seems to be some complication on top of that: - PyCCall_FastCall() accepts several types for the keywords, even a dict; That is actually a *simplification* instead of a *complication*. Currently, there is a huge amount of code

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-18 Thread Jeroen Demeyer
On 2018-06-18 16:55, INADA Naoki wrote: Speeding up most python function and some bultin functions was very significant. But I doubt making some 3rd party call 20% faster can make real applications significant faster. These two sentences are almost contradictory. I find it strange to claim

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-19 Thread Jeroen Demeyer
On 2018-06-18 15:09, Victor Stinner wrote: There are multiple issues with tp_fastcall: Personally, I think that you are exaggerating these issues. Below, I'm writing the word FASTCALL to refer to tp_fastcall in your patch as well as my C call protocol in the PEP-in-progress. * ABI issue:

Re: [Python-Dev] Error message for wrong number of arguments

2018-08-03 Thread Jeroen Demeyer
Actually, I just realized that it's not really possible to fix the error messages for built-in methods. The problem is that Argument Clinic does not know whether a function or method is being handled. For example, there is no indication at all that this is a method (note that the name

Re: [Python-Dev] Error message for wrong number of arguments

2018-08-03 Thread Jeroen Demeyer
Actually, scratch that, I posted too soon. There is also a block /*[clinic input] class list "PyListObject *" "_Type" [clinic start generated code]*/ So it could work. ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] [PEP 576/580] Reserve one type slot for Cython

2018-07-30 Thread Jeroen Demeyer
On 2018-07-30 15:35, INADA Naoki wrote: As repeatedly said, PEP 580 is very complicated protocol when just implementing callable object. Can you be more concrete what you find complicated? Maybe I can improve the PEP to explain it more. Also, I'm open to suggestions to make it less

[Python-Dev] Error message for wrong number of arguments

2018-07-30 Thread Jeroen Demeyer
Hello, I noticed an inconsistency in the error messages for the number of arguments to a method call. For Python methods, the "self" argument is counted. For built-in methods, the "self" argument is *not* counted: >>> class mylist(list): ... def append(self, val): super().append(val)

Re: [Python-Dev] PEP 576/579/580 benchmark: mistune

2018-07-30 Thread Jeroen Demeyer
On 2018-07-30 13:11, INADA Naoki wrote: Like previous SageMath bench, this is caused by Cython's specialization; __Pyx_PyObject_CallOneArg. It specializing calling PyFunction and PyCFunction, but it doesn't specialized for calling CyFunction. Yes, I saw that too. But this is exactly what

Re: [Python-Dev] Error message for wrong number of arguments

2018-07-30 Thread Jeroen Demeyer
On 2018-07-30 20:22, Chris Barker wrote: is it possible for the interpreter to know when this error is generated that this is a bound method expecting a "self", rather than an arbitrary function with n parameters? That would be quite hard. The error message is generated by the underlying

Re: [Python-Dev] Let's change to C API!

2018-07-29 Thread Jeroen Demeyer
My first impression is that making things faster and hiding implementation details in the ABI are contrary goals. I agree with hiding implementation details in the API but not in the ABI. For example, you mention that you want to make Py_INCREF() a function call instead of a macro. But since

[Python-Dev] PEP 576/579/580 benchmark: mistune

2018-07-27 Thread Jeroen Demeyer
Hello all, since my latest benchmark for PEP 580 [1] involved SageMath, which is quite a big project, I instead propose a much simpler benchmark involving mistune. mistune [2] is a Markdown parser implemented in the Python language. It optionally allows Cython compilation. It doesn't use

Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580

2018-07-31 Thread Jeroen Demeyer
On 2018-07-31 12:10, INADA Naoki wrote: Surely, they should understand they must use CCALL_* flags instead of METH_* flags when implementing fast-callable object. Yes indeed. But implementing a fast-callable object is quite specialized, not something that ordinary extension writers would care

Re: [Python-Dev] Let's change to C API!

2018-07-31 Thread Jeroen Demeyer
On 2018-07-31 15:34, Victor Stinner wrote: But I never used Cython nor cffi, so I'm not sure which one is the most appropriate depending on the use case. Cython is a build-time tool, while cffi is a run-time tool. But Cython does a lot more than just FFI. It is a Python->C compiler which can

Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580

2018-07-31 Thread Jeroen Demeyer
On 2018-07-31 12:10, INADA Naoki wrote: After spent several days to read PEP 580 and your implementation, I think I can implement it. I think it's not easy, but it's not impossible too. The signature of "extended_call_ptr" in PEP 576 is almost the same as the signature of a

Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580

2018-08-01 Thread Jeroen Demeyer
On 2018-07-31 11:12, INADA Naoki wrote: Any PEP won't be accepted in few month, because we don't have flow to accept PEPs for now. Is that certain? I haven't been following the process discussions, so I'm just asking the question. For example, given that you are already looking at PEP 580,

Re: [Python-Dev] Let's change to C API!

2018-07-31 Thread Jeroen Demeyer
On 2018-07-31 08:58, Antoine Pitrou wrote: I think Stefan is right that we should push people towards Cython and alternatives, rather than direct use of the C API (which people often fail to use correctly, in my experience). I know this probably isn't the correct place to bring it up, but I'm

Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580

2018-07-31 Thread Jeroen Demeyer
On 2018-07-31 09:36, INADA Naoki wrote: I want to see PoC of direct C calling. To be honest, there is no implementation plan for this yet. I know that several people want this feature, so it makes sense to think about it. For me personally, the main open problem is how to deal with

Re: [Python-Dev] Error message for wrong number of arguments

2018-08-03 Thread Jeroen Demeyer
On 2018-07-30 17:28, Nick Coghlan wrote: I would, and I think it would make sense for the PEP to cite improving consistency (and reducing code duplication?) in that regard as an advantage of the PEP. I'm not sure to which PEP you are referring (PEP 580 or a new PEP?). After thinking a bit

Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580

2018-07-31 Thread Jeroen Demeyer
On 2018-07-31 09:36, INADA Naoki wrote: I think PEP 580 is understandable only for people who tried to implement method objects. Is this really a problem? Do we expect that all Python developers can understand all PEPs, especially on a technical subject like this? To give a different

Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580

2018-07-31 Thread Jeroen Demeyer
On 2018-07-31 11:12, INADA Naoki wrote: For me, this is the most important benefit of PEP 580. I can't split it from PEP 580. I want PEP 580 to stand by itself. And you say that it is already complicated enough, so we should not mix native C calling into it. PEP 580 is written to allow

Re: [Python-Dev] Can we split PEP 489 (extension module init) ?

2018-08-11 Thread Jeroen Demeyer
> Would this be better than a flag + raising an error on init? Exactly. PEP 489 only says "Extensions using the new initialization scheme are expected to support subinterpreters". What's wrong with raising an exception when the module is initialized the second time? Jeroen.

Re: [Python-Dev] Let's change to C API!

2018-08-23 Thread Jeroen Demeyer
On 2018-08-23 07:36, Antoine Pitrou wrote: - the bootstrap problem (Cython self-compiles with CPython) This is not a big problem: we can make sure that all stdlib dependencies of Cython either have PEP 399 pure Python implementations or we keep them in pure C. - the dependency /

[Python-Dev] PEP 580 (C call protocol) minor update

2018-07-16 Thread Jeroen Demeyer
I made some minor updates to PEP 580 (PEP editors: please merge https://github.com/python/peps/pull/741) and its reference implementation: - Added a new introductory section explaining the basic idea. - The C protocol no longer deals with __name__; a __name__ attribute is required but the

[Python-Dev] Specification of C function profiling?

2018-07-16 Thread Jeroen Demeyer
Hello, it seems to me that there is no clear specification for the sys.setprofile() event c_call: the docs say "A C function is about to be called" but it's not clear what that means exactly, in particular when that C function is an unbound method like list.append. I also noticed that

[Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-21 Thread Jeroen Demeyer
Hello, I finally managed to get some real-life benchmarks for why we need a faster C calling protocol (see PEPs 576, 579, 580). I focused on the Cython compilation of SageMath. By default, a function in Cython is an instance of builtin_function_or_method (analogously, method_descriptor for

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-21 Thread Jeroen Demeyer
On 2018-07-21 19:07, INADA Naoki wrote: Good job. But I already +1 for adding support for extension callable type. Do you think this benchmark can be optimized more in future optimization which is possible by PEP 580, but not 576? I should clarify that the benchmark did not involve an

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-21 Thread Jeroen Demeyer
On 2018-07-21 19:55, Guido van Rossum wrote: I don’t think we can safely assume Python 3.7 has the same performance, actually. A lot has changed. I'm not denying that some things have changed. Rather, I claim that those changes wouldn't invalidate the benchmarks. I am comparing calls

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-22 Thread Jeroen Demeyer
On 2018-07-22 08:27, INADA Naoki wrote: It's interesting... But I failed to build sage. What went wrong? It's build step is too different from normal Python package. That's true because Sage considers itself a distribution rather than a package. But it's possible to pick the distribution

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-22 Thread Jeroen Demeyer
On 2018-07-22 01:14, Guido van Rossum wrote: Jeroen was asked to provide benchmarks but only provided them for Python 2. The reasoning that not much has changed that could affect the benchmarks feels a bit optimistic, that’s all. The micro-benchmarks showed a clear difference on Python 3.8

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-22 Thread Jeroen Demeyer
On 2018-07-22 14:52, Stefan Behnel wrote: Someone has to maintain the *existing* code base and help newcomers to get into it and understand it. This is an important point that people seem to be overlooking. The complexity and maintenance burden of PEP 580 should be compared to the

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-22 Thread Jeroen Demeyer
On 2018-07-22 18:14, Guido van Rossum wrote: Sorry I don't have better news. I don't consider that particularly bad news (but not good news either). I feel like the status on PEP 580 isn't anywhere near accepted anyway. I just hope that Python development won't stall completely. Even if no

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-22 Thread Jeroen Demeyer
On 2018-07-22 22:32, Antoine Pitrou wrote: Two things: - the issue26110 changes are not very large, it's just two additional opcodes and a bit of support code I didn't mean to compare PEP 580 to that specific issue, it was just an example. Even if issue26110 is small, the total complexity

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-23 Thread Jeroen Demeyer
I did exactly the same benchmark again with Python 3.7 and the results are similar. I'm copying and editing the original post for completeness: I finally managed to get some real-life benchmarks for why we need a faster C calling protocol (see PEPs 576, 579, 580). I focused on the Cython

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-23 Thread Jeroen Demeyer
On 2018-07-23 01:54, Ivan Pozdeev via Python-Dev wrote: All the material to discuss that we have in this thread is a single test result that's impossible to reproduce and impossible to run in Py3. I just posted that it can be reproduced on Python 3.7:

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-23 Thread Jeroen Demeyer
On 2018-07-23 00:28, Guido van Rossum wrote: So does your implementation of the PEP result in a net increase or decrease of the total lines of code? 12 files changed, 918 insertions(+), 704 deletions(-) That's a net increase, so there is no obvious win here. Still, I have various excuses for

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-23 Thread Jeroen Demeyer
On 2018-07-23 12:13, Antoine Pitrou wrote: IMHO the main point of discussion should be judging the solution you are proposing Yes please. I would very much welcome a discussion about the actual content of the PEP instead of meta-discussions like how complicated it is.

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-05 Thread Jeroen Demeyer
On 2018-07-05 05:41, INADA Naoki wrote: And stabilizing calling convention is prerequirements of designing new calling APIs. I don't see why. I made my PEP with the assumption that the METH_FASTCALL calling convention won't change. As far as I know, nobody advocated for changing it. But even

[Python-Dev] On the METH_FASTCALL calling convention

2018-07-05 Thread Jeroen Demeyer
Hello all, As discussed in some other threads ([1], [2]), we should discuss the METH_FASTCALL calling convention. For passing only positional arguments, a C array of Python objects is used, which is as fast as it can get. When the Python interpreter calls a function, it builds that C array

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-05 Thread Jeroen Demeyer
On 2018-07-05 13:32, INADA Naoki wrote: Core devs interested in this area is limited resource. I know and unfortunately there is nothing that I can do about that. It would be a pity that PEP 580 (or a variant like PEP 576) is not accepted simply because no core developer cares enough. As

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-05 Thread Jeroen Demeyer
On 2018-07-05 14:20, INADA Naoki wrote: What you can do is "divide and conquer". Split PEP in small topics we can focus. The PEP is already small and focused, I really did my best to make it as minimal as possible. I don't see a meaningful way to split it up even further.

Re: [Python-Dev] On the METH_FASTCALL calling convention

2018-07-06 Thread Jeroen Demeyer
On 2018-07-06 06:07, INADA Naoki wrote: Maybe, one way to improve METH_FASTCALL | METH_KEYWORDS can be this. kwds can be either tuple or dict. But that would be just pushing the complexity down to the callee. I'd rather have a simpler protocol at the expense of a slightly more complex

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-06 Thread Jeroen Demeyer
On 2018-07-05 21:57, Guido van Rossum wrote: Would it be possible to get outside experts to help? I don't understand what you mean: to help with what? Designing the PEP? Discussing the PEP? Accepting the PEP? Lobbying Python core devs? The Cython developers (in particular Stefan Behnel)

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-06 Thread Jeroen Demeyer
On 2018-07-05 14:20, INADA Naoki wrote: like you ignored my advice about creating realistic benchmark for calling 3rd party callable before talking about performance... I didn't really want to ignore that, I just didn't know what to do. As far as I can tell, the official Python benchmark

Re: [Python-Dev] Micro-benchmarks for PEP 580

2018-07-10 Thread Jeroen Demeyer
OK, I tried with --duplicate 200 and you can see the results at https://gist.github.com/jdemeyer/f0d63be8f30dc34cc989cd11d43df248 In short, the timings with and without PEP 580 are roughly the same (which is to be expected). Interestingly, a small but significant improvement can be seen when

Re: [Python-Dev] Micro-benchmarks for PEP 580

2018-07-10 Thread Jeroen Demeyer
On 2018-07-10 14:59, INADA Naoki wrote: Currently, we create temporary long object for passing argument. If there is protocol for exposeing format used by PyArg_Parse*, we can bypass temporal Python object and call myfunc_impl directly. Indeed, I already mentioned this at

[Python-Dev] Micro-benchmarks for function calls (PEP 576/579/580)

2018-07-09 Thread Jeroen Demeyer
Here is an initial version of a micro-benchmark for C function calling: https://github.com/jdemeyer/callbench I don't have results yet, since I'm struggling to find the right options to "perf timeit" to get a stable result. If somebody knows how to do this, help is welcome. Jeroen.

Re: [Python-Dev] Micro-benchmarks for PEP 580

2018-07-11 Thread Jeroen Demeyer
On 2018-07-11 00:48, Victor Stinner wrote: About your benchmark results: "FASTCALL unbound method(obj, 1, two=2): Mean +- std dev: 42.6 ns +- 29.6 ns" That's a very big standard deviation :-( Yes, I know. My CPU was overheating and was slowed down. But that seemed to have happened for a

Re: [Python-Dev] Micro-benchmarks for PEP 580

2018-07-11 Thread Jeroen Demeyer
On 2018-07-11 10:27, Antoine Pitrou wrote: I agree PEP 580 is extremely complicated and it's not obvious what the maintenance burden will be in the long term. But the status quo is also very complicated! If somebody would write a PEP describing the existing implementation of

Re: [Python-Dev] Micro-benchmarks for PEP 580

2018-07-11 Thread Jeroen Demeyer
On 2018-07-11 10:50, Victor Stinner wrote: As you wrote, the cost of function costs is unlikely the bottleneck of application. With that idea, METH_FASTCALL is not needed either. I still find it very strange that nobody seems to question all the crazy existing optimizations for function

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-06 Thread Jeroen Demeyer
On 2018-07-06 23:12, Guido van Rossum wrote: It's your PEP. And you seem to be struggling with something. But I can't tell quite what it is you're struggling with. To be perfectly honest (no hard feelings though!): what I'm struggling with is getting feedback (either positive or negative)

Re: [Python-Dev] PEP 575, 576, 579 and 580

2018-07-07 Thread Jeroen Demeyer
On 2018-07-07 15:38, Mark Shannon wrote: Hi, We seem to have a plethora of PEPs where we really ought to have one (or none?). - PEP 575 has been withdrawn. - PEP 579 is an informational PEP with the bigger picture; it does contain some of the requirements that you want to discuss here. - PEP

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-07 Thread Jeroen Demeyer
On 2018-07-07 14:54, Mark Shannon wrote: There is a minimal implementation and has been for a while. There is a link at the bottom of the PEP. Yes, I saw that but the implementation does not correspond to the PEP. In particular, this sentence from the PEP has not been implemented: When

Re: [Python-Dev] On the METH_FASTCALL calling convention

2018-07-08 Thread Jeroen Demeyer
On 2018-07-07 10:55, Serhiy Storchaka wrote: The first part of handling arguments can be made outside of the C function, by the calling API. Sure, it could be done but I don't see the advantage. I don't think you will gain performance because you are just moving code from one place to

Re: [Python-Dev] PEP 575, 576, 579 and 580

2018-07-09 Thread Jeroen Demeyer
On 2018-07-08 23:13, Mark Shannon wrote: I've added you suggestion, and everyone else's, to this github repo: https://github.com/markshannon/extended-calling-convention Feel free to comment on github, submit PRs or just email me directly if you have anything else you want to add. Do you agree

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-04 Thread Jeroen Demeyer
On 2018-07-04 03:31, INADA Naoki wrote: I think both PEPs are relying on FASTCALL calling convention, and can't be accepted until FASTCALL is stable & public. First of all, the fact that FASTCALL has not been made public should not prevent from discussing those PEPs and even making a

Re: [Python-Dev] Are undocumented functions part of the stable ABI?

2018-04-04 Thread Jeroen Demeyer
On 2018-04-04 17:56, Guido van Rossum wrote: It would be helpful if you explained the context of your request. The context is PEP 575. I guess my question is mostly about PyCFunction_Check(). I will not be able to keep it 100% backwards compatible simply because the goal of that PEP is

[Python-Dev] Are undocumented functions part of the stable ABI?

2018-04-04 Thread Jeroen Demeyer
Hello, I have a question about PEP 384: can undocumented functions ever be considered as part of the stable ABI? With undocumented, I mean not appearing in the "Python/C API Reference Manual". Whatever the answer to this question is, it would be good to make it explicit in PEP 384. I am

Re: [Python-Dev] Are undocumented functions part of the stable ABI?

2018-04-11 Thread Jeroen Demeyer
On 2018-04-10 13:49, Nick Coghlan wrote: If it's only a semantic level change in the way the macro gets expanded, then whether or not it needs an ABI version guard gets judged on a case-by-case basis, and in this particular case, my view would be that developers should be able to write

[Python-Dev] PEP 575: Unifying function/method classes

2018-04-12 Thread Jeroen Demeyer
Dear Python developers, I would like to request a review of PEP 575, which is about changing the classes used for built-in functions and Python functions and methods. The text of the PEP can be found at https://www.python.org/dev/peps/pep-0575/ No substantial changes to the contents of the

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-14 Thread Jeroen Demeyer
On 2018-04-13 21:30, Raymond Hettinger wrote: It would be nice to have a section that specifically discusses the implications with respect to other existing function-like tooling: classmethod, staticmethod, partial, itemgetter, attrgetter, methodgetter, etc. My hope is that there are no

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-14 Thread Jeroen Demeyer
On 2018-04-13 15:23, Nick Coghlan wrote: There's also a section in the rationale which refers to METH_USRx flags, which I'm guessing from context are an idea you were considering proposing, but eventually dropped from the rest of the PEP. No, I actually still want to propose it. In my latest

Re: [Python-Dev] Are undocumented functions part of the stable ABI?

2018-04-09 Thread Jeroen Demeyer
On 2018-04-08 05:17, Nick Coghlan wrote: Changing macro definitions doesn't break the stable ABI, as long as the *old* macro expansions still do the right thing. To me, it looks like a bad idea to change macros. Imagine that the PyCFunction_Check macro changes in Python 3.8. Then an extension

  1   2   3   >