Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Nick Coghlan]

On 19 June 2018 at 16:12, INADA Naoki  wrote:
>
> On Tue, Jun 19, 2018 at 2:56 PM Jeroen Demeyer  wrote:
>>
>> On 2018-06-18 16:55, INADA Naoki wrote:
>> > Speeding up most python function and some bultin functions was very
>> > significant.
>> > But I doubt making some 3rd party call 20% faster can make real
>> > applications significant faster.
>>
>> These two sentences are almost contradictory. I find it strange to claim
>> that a given optimization was "very significant" in specific cases while
>> saying that the same optimization won't matter in other cases.
>
> It's not contradictory because there is basis:
>
>   In most real world Python application, number of calling Python methods or
>   bulitin functions are much more than other calls.
>
> For example, optimization for bulitin `tp_init` or `tp_new` by FASTCALL was
> rejected because it's implementation is complex and it's performance gain is
> not significant enough on macro benchmarks.
>
> And I doubt number of 3rd party calls are much more than calling builtin
> tp_init or tp_new.

I was going to ask a question here about JSON parsing
micro-benchmarks, but then I went back and re-read
https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html
and realised that the main problem discussed in that article was the
*memory* overhead of creating full Python object instances, not the
runtime cost of instantiating those objects.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Victor Stinner]

2018-06-19 13:58 GMT+02:00 Jeroen Demeyer :
> Personally, I think that you are exaggerating these issues.

I'm not trying to convince you to abandon the idea. I would be happy
to be able to use FASTCALL in more cases! I just tried to explain why
I chose to abandon my idea.

FASTCALL is cute on tiny microbenchmarks, but I'm not sure that having
spent almost one year on it was worth it :-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Ivan Levkivskyi]

On 19 June 2018 at 13:02, Nick Coghlan  wrote:

> On 19 June 2018 at 16:12, INADA Naoki  wrote:
> >
> > On Tue, Jun 19, 2018 at 2:56 PM Jeroen Demeyer 
> wrote:
> >>
> >> On 2018-06-18 16:55, INADA Naoki wrote:
> >> > Speeding up most python function and some bultin functions was very
> >> > significant.
> >> > But I doubt making some 3rd party call 20% faster can make real
> >> > applications significant faster.
> >>
> >> These two sentences are almost contradictory. I find it strange to claim
> >> that a given optimization was "very significant" in specific cases while
> >> saying that the same optimization won't matter in other cases.
> >
> >
> > It's not contradictory because there is basis:
> >
> >   In most real world Python application, number of calling Python
> methods or
> >   bulitin functions are much more than other calls.
> >
> > For example, optimization for bulitin `tp_init` or `tp_new` by FASTCALL
> was
> > rejected because it's implementation is complex and it's performance
> gain is
> > not significant enough on macro benchmarks.
> >
> > And I doubt number of 3rd party calls are much more than calling builtin
> > tp_init or tp_new.
>
> I don't think this assumption is correct, as scientific Python
> software spends a lot of time calling other components in the
> scientific Python stack, and bypassing the core language runtime
> entirely.
>
>
A recent Python survey by PSF/JetBrains shows that almost half of current
Python
users are using it for data science/ML/etc. For all these people most of
the time is spent
on calling C functions in extensions.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[INADA Naoki]

That's why I suggested to add new benchmark.

2018年6月19日(火) 22:22 Ivan Levkivskyi :

> On 19 June 2018 at 13:02, Nick Coghlan  wrote:
>
>> On 19 June 2018 at 16:12, INADA Naoki  wrote:
>> >
>> > On Tue, Jun 19, 2018 at 2:56 PM Jeroen Demeyer 
>> wrote:
>> >>
>> >> On 2018-06-18 16:55, INADA Naoki wrote:
>> >> > Speeding up most python function and some bultin functions was very
>> >> > significant.
>> >> > But I doubt making some 3rd party call 20% faster can make real
>> >> > applications significant faster.
>> >>
>> >> These two sentences are almost contradictory. I find it strange to
>> claim
>> >> that a given optimization was "very significant" in specific cases
>> while
>> >> saying that the same optimization won't matter in other cases.
>> >
>> >
>> > It's not contradictory because there is basis:
>> >
>> >   In most real world Python application, number of calling Python
>> methods or
>> >   bulitin functions are much more than other calls.
>> >
>> > For example, optimization for bulitin `tp_init` or `tp_new` by FASTCALL
>> was
>> > rejected because it's implementation is complex and it's performance
>> gain is
>> > not significant enough on macro benchmarks.
>> >
>> > And I doubt number of 3rd party calls are much more than calling builtin
>> > tp_init or tp_new.
>>
>> I don't think this assumption is correct, as scientific Python
>> software spends a lot of time calling other components in the
>> scientific Python stack, and bypassing the core language runtime
>> entirely.
>>
>>
> A recent Python survey by PSF/JetBrains shows that almost half of current
> Python
> users are using it for data science/ML/etc. For all these people most of
> the time is spent
> on calling C functions in extensions.
>
> --
> Ivan
>
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Nick Coghlan]

On 19 June 2018 at 16:12, INADA Naoki  wrote:
>
> On Tue, Jun 19, 2018 at 2:56 PM Jeroen Demeyer  wrote:
>>
>> On 2018-06-18 16:55, INADA Naoki wrote:
>> > Speeding up most python function and some bultin functions was very
>> > significant.
>> > But I doubt making some 3rd party call 20% faster can make real
>> > applications significant faster.
>>
>> These two sentences are almost contradictory. I find it strange to claim
>> that a given optimization was "very significant" in specific cases while
>> saying that the same optimization won't matter in other cases.
>
>
> It's not contradictory because there is basis:
>
>   In most real world Python application, number of calling Python methods or
>   bulitin functions are much more than other calls.
>
> For example, optimization for bulitin `tp_init` or `tp_new` by FASTCALL was
> rejected because it's implementation is complex and it's performance gain is
> not significant enough on macro benchmarks.
>
> And I doubt number of 3rd party calls are much more than calling builtin
> tp_init or tp_new.

I don't think this assumption is correct, as scientific Python
software spends a lot of time calling other components in the
scientific Python stack, and bypassing the core language runtime
entirely.

However, they're using the CPython C API's function calling
abstractions to do it, and those are currently expensive
(frustratingly so, when the caller, the callee, *and* the interpreter
implementation defining the call abstraction layer are all implemented
in C). Hence Jeroen's PEPs to make the FASTCALL API a generally
available one.

That's quite different from the situation with object constructors,
where a whole lot of applications will get to the point of having a
relatively stable working set of objects, and then see the rate of
object creation slow down markedly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-06-18 15:09, Victor Stinner wrote:

There are multiple issues with tp_fastcall:


Personally, I think that you are exaggerating these issues.

Below, I'm writing the word FASTCALL to refer to tp_fastcall in your 
patch as well as my C call protocol in the PEP-in-progress.



* ABI issue: it's possible to load a C extension using the old ABI,
without tp_fastcall: it's not possible to write type->tp_fastcall on
such type. This limitation causes different issues.


It's not hard to check for FASTCALL support and have a case distinction 
between using tp_call and FASTCALL.



* If tp_call is modified, tp_fastcall may be outdated.


I plan to support FASTCALL only for extension types. Those cannot be 
changed from Python.


If it turns out that FASTCALL might give significant benefits also for 
heap types, we can deal with those modifications: we already need to 
deal with such modifications anyway for existing slots like __call__.



* Many public functions of the C API still requires the tuple and dict
to pass positional and keyword arguments, so a compatibility layer is
required to types who only want to implement FASTCALL. Related issue:
what is something calls tp_call with (args: tuple, kwargs: dict)?
Crash or call a compatibility layer converting arguments to FASTCALL
calling convention?


You make it sound as if such a "compatibility layer" is a big issue. You 
just need one C API function to put in the tp_call slot which calls the 
object instead using FASTCALL.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Antoine Pitrou]

On Mon, 18 Jun 2018 19:49:28 +0200
Stefan Behnel  wrote:
> Victor Stinner schrieb am 18.06.2018 um 15:09:
> > I tried two options to add support for FASTCALL on calling an object:
> > add a flag in tp_flags and reuse tp_call, or add a new tp_fastcall
> > slot. I failed to implement correctly any of these two options.
> > 
> > There are multiple issues with tp_fastcall:
> > 
> > * ABI issue: it's possible to load a C extension using the old ABI,
> > without tp_fastcall: it's not possible to write type->tp_fastcall on
> > such type. This limitation causes different issues.  
> 
> Not a problem if we rededicate the unused (since Py3.0) "tp_print" slot for 
> it.

On the topic of the so-called old ABI (which doesn't really exist), I
would like to merge https://github.com/python/cpython/pull/4944

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[INADA Naoki]

On Tue, Jun 19, 2018 at 2:56 PM Jeroen Demeyer  wrote:

> On 2018-06-18 16:55, INADA Naoki wrote:
> > Speeding up most python function and some bultin functions was very
> > significant.
> > But I doubt making some 3rd party call 20% faster can make real
> > applications significant faster.
>
> These two sentences are almost contradictory. I find it strange to claim
> that a given optimization was "very significant" in specific cases while
> saying that the same optimization won't matter in other cases.
>

It's not contradictory because there is basis:

  In most real world Python application, number of calling Python methods or
  bulitin functions are much more than other calls.

For example, optimization for bulitin `tp_init` or `tp_new` by FASTCALL was
rejected because it's implementation is complex and it's performance gain is
not significant enough on macro benchmarks.

And I doubt number of 3rd party calls are much more than calling builtin
tp_init or tp_new.

Of course, current benchmark suite [1] doesn't cover all types of real
world Python
application.  You can create pull request which add benchmark for real world
application which depends on massive 3rd party calls.

[1] https://github.com/python/performance

Regards,
-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-06-18 16:55, INADA Naoki wrote:

Speeding up most python function and some bultin functions was very
significant.
But I doubt making some 3rd party call 20% faster can make real
applications significant faster.


These two sentences are almost contradictory. I find it strange to claim 
that a given optimization was "very significant" in specific cases while 
saying that the same optimization won't matter in other cases.


People *have* done benchmarks for actual code and this is causing actual 
slow-downs of around 20% in actual applications. That is the main reason 
why I am trying to push this PEP (or PEP 575 which solves the same 
problem in a different way).



Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Stefan Behnel]

Victor Stinner schrieb am 18.06.2018 um 15:09:
> I tried two options to add support for FASTCALL on calling an object:
> add a flag in tp_flags and reuse tp_call, or add a new tp_fastcall
> slot. I failed to implement correctly any of these two options.
> 
> There are multiple issues with tp_fastcall:
> 
> * ABI issue: it's possible to load a C extension using the old ABI,
> without tp_fastcall: it's not possible to write type->tp_fastcall on
> such type. This limitation causes different issues.

Not a problem if we rededicate the unused (since Py3.0) "tp_print" slot for it.

Even better, since the slot exists already in Py3.0+, tools like Cython,
NumPy (with its ufuncs etc.) or generic function dispatchers, basically
anything that benefits from fast calls, can enable support for it in all
CPython 3.x versions and benefit from faster calls among each other,
independent of the support in CPython. The explicit type flag opt-in that
the PEP proposes makes this completely safe.


> * If tp_call is modified, tp_fastcall may be outdated. Same if
> tp_fastcall is modified.

Slots are fixed at type creation and should never be modified afterwards.


> What happens on "del obj.__call__" or "del type.__call__"?

$ python3.7 -c 'del len.__call__'
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'builtin_function_or_method' object attribute '__call__' is
read-only

$ python3.7 -c 'del type.__call__'
Traceback (most recent call last):
  File "", line 1, in 
TypeError: can't set attributes of built-in/extension type 'type'

And a really lovely one:

$ python3.7 -c 'del (lambda:0).__call__'
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: __call__


> * Many public functions of the C API still requires the tuple and dict
> to pass positional and keyword arguments, so a compatibility layer is
> required to types who only want to implement FASTCALL.

Well, yes. It would require a trivial piece of code to map between the two.
Fine with me.


> Related issue:
> what is something calls tp_call with (args: tuple, kwargs: dict)?
> Crash or call a compatibility layer converting arguments to FASTCALL
> calling convention?

The latter, obviously. Also easy to implement, with the usual undefined
dict order caveat (although that's probably solved when running in Py3.6+).


> I abandoned my idea for two reasons:
> 
> 1) in the worst case, my changes caused a crash which is not accepted
> for an optimization.

This isn't really an optimisation. It's a generalisation of the call protocol.


> My first intent was to removed the
> property_descr_get() hack because its implementation is fragile and
> caused crashes.

Not sure which hack you mean.


> 2) we implemented a lot of other optimizations which made calls faster
> without having to touch tp_call nor tp_fastcall. The benefit of
> FASTCALL for tp_call/tp_fastcall was not really significant.

What Jeroen said. Cleaning up the implementation and generalising the call
protocol is going to open up a wonderfully bright future for CPython. :)

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Guido van Rossum]

Like Inada-san, I would like to see the implementation first.

On Mon, Jun 18, 2018, 07:33 Jeroen Demeyer  wrote:

> On 2018-06-18 15:09, Victor Stinner wrote:
> > 2) we implemented a lot of other optimizations which made calls faster
> > without having to touch tp_call nor tp_fastcall.
>
> And that's a problem because these optimizations typically only work for
> specific classes. My PEP wants to replace those by something more
> structural.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[INADA Naoki]

On Mon, Jun 18, 2018 at 11:33 PM Jeroen Demeyer  wrote:

> On 2018-06-18 15:09, Victor Stinner wrote:
> > 2) we implemented a lot of other optimizations which made calls faster
> > without having to touch tp_call nor tp_fastcall.
>
> And that's a problem because these optimizations typically only work for
> specific classes. My PEP wants to replace those by something more
> structural.
>

​And we need data how much it speedup some applications, not only
microbenchmarks.

Speeding up most python function and some bultin functions was very
significant.
But I doubt making some 3rd party call 20% faster can make real
applications significant faster.

-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-06-18 15:09, Victor Stinner wrote:

2) we implemented a lot of other optimizations which made calls faster
without having to touch tp_call nor tp_fastcall.


And that's a problem because these optimizations typically only work for 
specific classes. My PEP wants to replace those by something more 
structural.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Victor Stinner]

Hi,

I tried two options to add support for FASTCALL on calling an object:
add a flag in tp_flags and reuse tp_call, or add a new tp_fastcall
slot. I failed to implement correctly any of these two options.

There are multiple issues with tp_fastcall:

* ABI issue: it's possible to load a C extension using the old ABI,
without tp_fastcall: it's not possible to write type->tp_fastcall on
such type. This limitation causes different issues.
* If tp_call is modified, tp_fastcall may be outdated. Same if
tp_fastcall is modified. What happens on "del obj.__call__" or "del
type.__call__"?
* Many public functions of the C API still requires the tuple and dict
to pass positional and keyword arguments, so a compatibility layer is
required to types who only want to implement FASTCALL. Related issue:
what is something calls tp_call with (args: tuple, kwargs: dict)?
Crash or call a compatibility layer converting arguments to FASTCALL
calling convention?

Reusing tp_call for FASTCALL cause similar or worse issues.

I abandoned my idea for two reasons:

1) in the worst case, my changes caused a crash which is not accepted
for an optimization. My first intent was to removed the
property_descr_get() hack because its implementation is fragile and
caused crashes.
2) we implemented a lot of other optimizations which made calls faster
without having to touch tp_call nor tp_fastcall. The benefit of
FASTCALL for tp_call/tp_fastcall was not really significant.

Victor

2018-06-18 7:55 GMT+02:00 Jeroen Demeyer :
> On 2018-06-18 03:34, INADA Naoki wrote:
>>
>> Victor had tried to add `tp_fastcall` slot, but he suspended his effort
>> because
>> it's benefit is not enough for it's complexity.
>> https://bugs.python.org/issue29259
>
>
> I has a quick look at that patch and it's really orthogonal to what I'm
> proposing. I'm proposing to use the slot *instead* of existing fastcall
> optimizations. Victor's patch was about adding fastcall support to classes
> that didn't support it before.
>
>
>
> Jeroen.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[INADA Naoki]

I didn't meant comparing tp_fastcall and your PEP.

I just meant we need to compare complexity and benefit (performance),
and we need reference implementation for comparing.


On Mon, Jun 18, 2018 at 3:03 PM Jeroen Demeyer  wrote:

> On 2018-06-18 03:34, INADA Naoki wrote:
> > Victor had tried to add `tp_fastcall` slot, but he suspended his effort
> > because
> > it's benefit is not enough for it's complexity.
> > https://bugs.python.org/issue29259
>
> I has a quick look at that patch and it's really orthogonal to what I'm
> proposing. I'm proposing to use the slot *instead* of existing fastcall
> optimizations. Victor's patch was about adding fastcall support to
> classes that didn't support it before.
>
>
> Jeroen.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
>


-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-06-18 03:34, INADA Naoki wrote:

Victor had tried to add `tp_fastcall` slot, but he suspended his effort
because
it's benefit is not enough for it's complexity.
https://bugs.python.org/issue29259


I has a quick look at that patch and it's really orthogonal to what I'm 
proposing. I'm proposing to use the slot *instead* of existing fastcall 
optimizations. Victor's patch was about adding fastcall support to 
classes that didn't support it before.



Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[INADA Naoki]

Hi Jeroen.

It's interesting, but I think we need reference implementation to compare
it's benefit with it's complexity.

Victor had tried to add `tp_fastcall` slot, but he suspended his effort
because
it's benefit is not enough for it's complexity.
https://bugs.python.org/issue29259

I think if your idea can reduce complexity of current special cases without
any performance loss, it's nice.

On the other hand, if your idea increase complexity, I doubt it's benefit.

Increasing performance of all Python defined methods + most of builtin
methods
affects total application performance because it covers most calls.
But calling callable object other than them are relatively rare.  It may
not affect
real world performance of most applications.

So, until I can compare it's complexity and benefits, I can say only "it's
interesting."

Regards,

-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Ronald Oussoren]

> On 17 Jun 2018, at 16:31, Stefan Behnel  wrote:
> 
> Ronald Oussoren schrieb am 17.06.2018 um 14:50:
>> Why did you add a tp_ccalloffset slot to the type with the actual 
>> information in instances instead of storing the information in a slot? 
> 
> If the configuration of the callable was in the type, you would need a
> separate type for each kind of callable. That would quickly explode. Think
> of this as a generalised PyCFunction interface to arbitrary callables.
> There is a function pointer and some meta data, and both are specific to an
> instance.

That’s true for PyCFunction, but not necessarily as a general replacement for 
the tp_call slot.  I my code I’d basically use the same function pointer and 
metadata for all instances (that is, more like PyFunction than PyCFunction). 

> 
> Also, there are usually only a limited number of callables around, so
> memory doesn't matter. (And memory usage would be a striking reason to have
> something in a type rather than an instance.)

I was mostly surprised that something that seems to be a replacement for 
tp_call stores the interesting information in instances instead of the type 
itself. 

Ronald

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-06-17 14:50, Ronald Oussoren wrote:

This looks interesting. Why did you add a tp_ccalloffset slot to the type with 
the actual information in instances instead of storing the information in a 
slot?


Think of built-in functions. Every built-in function is a different 
callable and calls a different C function. So it must be stored in the 
instances.


However, the case where all instances share a PyCCallDef is also 
possible: all instances would then simply have the same PyCCallDef pointer.



Jeroen.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Stefan Behnel]

Ronald Oussoren schrieb am 17.06.2018 um 14:50:
> Why did you add a tp_ccalloffset slot to the type with the actual information 
> in instances instead of storing the information in a slot? 

If the configuration of the callable was in the type, you would need a
separate type for each kind of callable. That would quickly explode. Think
of this as a generalised PyCFunction interface to arbitrary callables.
There is a function pointer and some meta data, and both are specific to an
instance.

Also, there are usually only a limited number of callables around, so
memory doesn't matter. (And memory usage would be a striking reason to have
something in a type rather than an instance.)

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Ronald Oussoren]

> On 17 Jun 2018, at 11:00, Jeroen Demeyer  wrote:
> 
> Hello,
> 
> I have been working on a slightly different PEP to use a new type slot 
> tp_ccalloffset instead the base_function base class. You can see the work in 
> progress here:
> 
> https://github.com/jdemeyer/PEP-ccall
> 
> By creating a new protocol that each class can implement, there is a full 
> decoupling between the features of a class and between the class hierarchy 
> (such coupling was complained about during the PEP 575 discussion). So I got 
> convinced that this is a better approach.
> 
> It also has the advantage that changes can be made more gradually: this PEP 
> changes nothing at all on the Python side, it only changes the CPython 
> implementation. I still think that it would be a good idea to refactor the 
> class hierarchy, but that's now an independent issue.
> 
> Another advantage is that it's more general and easier for existing classes 
> to use the protocol (PEP 575 on the other hand requires subclassing from 
> base_function which may not be compatible with an existing class hierarchy).

This looks interesting. Why did you add a tp_ccalloffset slot to the type with 
the actual information in instances instead of storing the information in a 
slot? 

Ronald

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Nick Coghlan]

On 17 June 2018 at 19:00, Jeroen Demeyer  wrote:
> Hello,
>
> I have been working on a slightly different PEP to use a new type slot
> tp_ccalloffset instead the base_function base class. You can see the work in
> progress here:
>
> https://github.com/jdemeyer/PEP-ccall
>
> By creating a new protocol that each class can implement, there is a full
> decoupling between the features of a class and between the class hierarchy
> (such coupling was complained about during the PEP 575 discussion). So I got
> convinced that this is a better approach.
>
> It also has the advantage that changes can be made more gradually: this PEP
> changes nothing at all on the Python side, it only changes the CPython
> implementation. I still think that it would be a good idea to refactor the
> class hierarchy, but that's now an independent issue.
>
> Another advantage is that it's more general and easier for existing classes
> to use the protocol (PEP 575 on the other hand requires subclassing from
> base_function which may not be compatible with an existing class hierarchy).

Ah, this looks *very* nice :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

Hello,

I have been working on a slightly different PEP to use a new type slot 
tp_ccalloffset instead the base_function base class. You can see the 
work in progress here:


https://github.com/jdemeyer/PEP-ccall

By creating a new protocol that each class can implement, there is a 
full decoupling between the features of a class and between the class 
hierarchy (such coupling was complained about during the PEP 575 
discussion). So I got convinced that this is a better approach.


It also has the advantage that changes can be made more gradually: this 
PEP changes nothing at all on the Python side, it only changes the 
CPython implementation. I still think that it would be a good idea to 
refactor the class hierarchy, but that's now an independent issue.


Another advantage is that it's more general and easier for existing 
classes to use the protocol (PEP 575 on the other hand requires 
subclassing from base_function which may not be compatible with an 
existing class hierarchy).



Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Robert Bradshaw]

Having had some time to let this settle for a bit, I hope it doesn't
get abandoned just because it was to complicated to come to a
conclusion.

I'd like to attempt to summarize the main ideas as follows.

1) Currently the "fast call" optimization is implemented by by
checking explicitly for a set of types (builtin functions, methods
method descriptors, and functions). This is both ugly, as it requires
listing several cases, and also locks out any other types from
participating in this protocol. This PEP proposes elevating this to a
contract that other types can participate in.

2) Inspect and friends are hard-coded checks on these non-extendable
types, again making it difficult for other types to be truly first
class citizens, and breaks attempts at duck typing.

3) The current hierarchy of builtin_function_or_method vs. function
vs. instancemethod could use some cleanup for consistency and
extensibility.

PEP 575 solves all of these by introducing a common base class, but
they are somewhat separable. As for complexity, there are two metrics,
the complexity of the delta (e.g. more lines of code in trickier
places = worse, paid once) and of the final result (less code, less
special casing = better, paid as long as the code is in use). I tend
to think it's a good tradeoff to pay former to improve the latter.

Jeroen, is this a fair summary? Are they fully separable?

Others, are these three valuable goals? At what cost (e.g. (3) may
have backwards compatibility concerns if taken as far as possible.)

- Robert


On Sun, May 20, 2018 at 1:15 PM, Jeroen Demeyer  wrote:
> On 2018-05-19 15:29, Nick Coghlan wrote:
>>
>> That's not how code reviews work, as their complexity is governed by the
>> number of lines changed (added/removed/modified), not just the number of
>> lines that are left at the end.
>
>
> Of course, you are right. I didn't mean literally that only the end result
> matters. But it should certainly be considered.
>
> If you only do small incremental changes, complexity tends to build up
> because choices which are locally optimal are not always globally optimal.
> Sometimes you need to do some refactoring to revisit some of that
> complexity. This is part of what PEP 575 does.
>
>> That said, "deletes more lines than it
>> adds" is typically a point strongly in favour of a particular change.
>
>
> This certainly won't be true for my patch, because there is a lot of code
> that I need to support for backwards compatibility (all the old code for
> method_descriptor in particular).
>
>
> Going back to the review of PEP 575, I see the following possible outcomes:
>
> (A) Accept it as is (possibly with minor changes).
>
> (B) Accept the general idea but split the details up in several PEPs which
> can still be discussed individually.
>
> (C) Accept a minimal variant of PEP 575, only changing existing classes but
> not changing the class hierarchy.
>
> (D) Accept some yet-to-be-written variant of PEP 575.
>
> (E) Don't fix the use case that PEP 575 wants to address.
>
>
> Petr Viktorin suggests (C). I am personally quite hesitant because that only
> adds complexity and it wouldn't be the best choice for the future
> maintainability of CPython. I also fear that this hypothetical PEP variant
> would be rejected because of that reason. Of course, if there is some
> general agreement that (C) is the way to go, then that is fine for me.
>
> If people feel that PEP 575 is currently too complex, I think that (B) is a
> very good compromise. The end result would be the same as what PEP 575
> proposes. Instead of changing many things at once, we could handle each
> class in a separate PEP. But the motivation of those mini-PEPs will still be
> PEP 575. So, in order for this to make sense, the general idea of PEP 575
> needs to be accepted: adding a base_function base class and making various
> existing classes subclasses of that.
>
>
>
> Jeroen.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/robertwb%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-05-19 15:29, Nick Coghlan wrote:

That's not how code reviews work, as their complexity is governed by the
number of lines changed (added/removed/modified), not just the number of
lines that are left at the end.


Of course, you are right. I didn't mean literally that only the end 
result matters. But it should certainly be considered.


If you only do small incremental changes, complexity tends to build up 
because choices which are locally optimal are not always globally 
optimal. Sometimes you need to do some refactoring to revisit some of 
that complexity. This is part of what PEP 575 does.



That said, "deletes more lines than it
adds" is typically a point strongly in favour of a particular change.


This certainly won't be true for my patch, because there is a lot of 
code that I need to support for backwards compatibility (all the old 
code for method_descriptor in particular).



Going back to the review of PEP 575, I see the following possible outcomes:

(A) Accept it as is (possibly with minor changes).

(B) Accept the general idea but split the details up in several PEPs 
which can still be discussed individually.


(C) Accept a minimal variant of PEP 575, only changing existing classes 
but not changing the class hierarchy.


(D) Accept some yet-to-be-written variant of PEP 575.

(E) Don't fix the use case that PEP 575 wants to address.


Petr Viktorin suggests (C). I am personally quite hesitant because that 
only adds complexity and it wouldn't be the best choice for the future 
maintainability of CPython. I also fear that this hypothetical PEP 
variant would be rejected because of that reason. Of course, if there is 
some general agreement that (C) is the way to go, then that is fine for me.


If people feel that PEP 575 is currently too complex, I think that (B) 
is a very good compromise. The end result would be the same as what PEP 
575 proposes. Instead of changing many things at once, we could handle 
each class in a separate PEP. But the motivation of those mini-PEPs will 
still be PEP 575. So, in order for this to make sense, the general idea 
of PEP 575 needs to be accepted: adding a base_function base class and 
making various existing classes subclasses of that.



Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Nick Coghlan]

On 17 May 2018 at 06:34, Jeroen Demeyer  wrote:

> On 2018-05-16 17:31, Petr Viktorin wrote:
>
>> The larger a change is, the harder it is to understand
>>
>
> I already disagree here...
>
> I'm afraid that you are still confusing the largeness of the *change* with
> the complexity of the *result* after the change was implemented.
>

That's not how code reviews work, as their complexity is governed by the
number of lines changed (added/removed/modified), not just the number of
lines that are left at the end. That said, "deletes more lines than it
adds" is typically a point strongly in favour of a particular change.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-05-16 17:31, Petr Viktorin wrote:

Less disruptive changes tend to have a better backwards compatibility story.
A less intertwined change makes it easier to revert just a single part,
in case that becomes necessary.


I'll just repeat what I said in a different post on this thread: we can 
still *implement* the PEP in a less intertwined and more gradual way. 
The PEP deals with several classes and each class can be changed separately.


However, there is not much point in starting this process if you don't 
intend to go all the way. The power of PEP 575 is really using this 
base_function class in many places.


A PEP just adding the class base_function as base class of 
buitin_function_or_method without using it anywhere else would make no 
sense by itself. Still, that could be a first isolated step in the 
implementation.


If PEP 575 is accepted, I would like to follow it up with PEPs to add 
more classes to the base_function hierarchy (candidates: staticmethod, 
classmethod, classmethod_descriptor, method-wrapper, slot wrapper, 
functools.lru_cache).



Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-05-16 17:31, Petr Viktorin wrote:

The larger a change is, the harder it is to understand


I already disagree here...

I'm afraid that you are still confusing the largeness of the *change* 
with the complexity of the *result* after the change was implemented.
A change that *removes* complexity should be considered a good thing, 
even if it's a large change.


That being said, if you want me to make smaller changes, I could do it. 
But I would do it for *you* personally because I'm afraid that other 
people might rightly complain that I'm making things too complicated.


So I would certainly like some feedback from others on this point.


Less disruptive changes tend to have a better backwards compatibility story.


Maybe in very general terms, yes. But I believe that the "disruptive" 
changes that I'm making will not contribute to backwards 
incompatibility. Adding new ml_flags flags shouldn't break anything and 
adding a base class shouldn't either (I doubt that there is code relying 
on the fact that type(len).__base__ is object).


In my opinion, the one change that is most likely to cause backwards 
compatibility problems is changing the type of bound methods of 
extension types. And that change is even in the less disruptive PEP 576.



Mark Shannon has an upcoming PEP with an alternative to some of the
issues.


I'm looking forward to a serious discussion about that. However, from a 
first reading, I'm not very optimistic about its performance implications.



Currently, the "outside" of a function (how it looks when introspected)
is tied to the "inside" (what happens internally when it's called).
Can we better enable pydoc/IPython developers to tackle introspection
problems without wading deep in the internals and call optimizations?


I proposed complete decoupling in https://bugs.python.org/issue30071 and 
that was rejected. Anyway, decoupling of introspection is not the 
essence of this PEP. This PEP is really about allowing custom built-in 
function subclasses. That's the hard part where CPython internals come 
in. So I suggest that we leave the discussion about introspection and 
focus on the function classes.



But, it still has to inherit from base_function to "look like a
function". Can we remove that limitation in favor of duck typing?


Duck typing is a Python thing, I don't know what "duck typing" would 
mean on the C level. We could change the existing isinstance(..., 
base_function) check by a different fast check. For example, we 
(together with the Cython devs) have been pondering about a new "type" 
field, say tp_cfunctionoffset pointing to a certain C field in the 
object structure. That would work but it would not be so fundamentally 
different from the current PEP.



*PS*: On friday, I'm leaving for 2 weeks on holidays. So if I don't 
reply to comments on PEP 575 or alternative proposals, don't take it as 
a lack of interest.



Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Stefan Behnel]

Petr Viktorin schrieb am 15.05.2018 um 18:36:
> On 05/15/18 05:15, Jeroen Demeyer wrote:
>> An important note is that it was never my goal to create a minimal PEP. I
>> did not aim for changing as little as possible. I was thinking: we are
>> changing functions, what would be the best way to implement them?
> 
> That might be a problem. For the change to be accepted, a core developer
> will need to commit to maintaining the code, understand it, and accept
> responsibility for anything that's broken. Naturally, large-scale changes
> have less of a chance there.

Honestly, the current implementation involves such a clutter of special
cases that the internal code simplification that this PEP allows should
make every core developer who needs to make their hands dirty with the
current code bow to Jeroen for coming up with this PEP and even
implementing it.

Just my personal opinion.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/15/18 17:55, Jeroen Demeyer wrote:

On 2018-05-15 18:36, Petr Viktorin wrote:

Naturally, large-scale
changes have less of a chance there.


Does it really matter that much how large the change is? I think you are 
focusing too much on the change instead of the end result.


As I said in my previous post, I could certainly make less disruptive 
changes. But would that really be better? (If you think that the answer 
is "yes" here, I honestly want to know).


Yes, I believe it is better.
The larger a change is, the harder it is to understand, meaning that 
less people can meaningfully join the conversation, think about how it 
interacts with their own use cases, and notice (and think through) any 
unpleasant details.

Less disruptive changes tend to have a better backwards compatibility story.
A less intertwined change makes it easier to revert just a single part, 
in case that becomes necessary.


I could make the code less different than today but at the cost of added 
complexity. Building on top of the existing code is like building on a 
bad foundation: the higher you build, the messier it gets. Instead, I 
propose a solid new foundation. Of course, that requires more work to 
build but once it is built, the finished building looks a lot better.


To continue the analogy: the tenants have been customizing their 
apartments inside that building, possibly depending on structural 
details that we might think should be hidden from them. And they expect 
to continue living there while the foundation is being swapped under them :)



With such a "finished product" PEP, it's hard to see if some of the
various problems could be solved in a better way -- faster, more
maintainable, or less disruptive.


With "faster", you mean runtime speed? I'm pretty confident that we 
won't lose anything there.


As I argued above, my PEP might very well make things "more 
maintainable", but this is of course very subjective. And "less 
disruptive" was never a goal for this PEP.



It's also harder from a psychological point of view: you obviously
already put in a lot of good work, and it's harder to waste that work if
an even better solution is found.


I hope that this won't be my psychology. As a developer, I prefer to 
focus on problems rather than on solutions: I don't want to push a 
particular solution, I want to fix a particular problem. If an even 
better solution is accepted, I will be a very happy man.


What I would hate is that this PEP gets rejected because some people 
claim that the problem can be solved in a better way, but without 
actually suggesting such a better way.


Mark Shannon has an upcoming PEP with an alternative to some of the 
issues. (Not all of them – but less intertwined is better, all else 
being equal.)



Is a branching class hierarchy, with quite a few new of flags for
feature selection, the kind of simplicity we want?


Maybe yes because it *concentrates* all complexity in one small place. 
Currently, we have several independent classes 
(builtin_function_or_method, method_descriptor, function, method) which 
all require various forms of special casing in the interpreter with some 
code duplication. With my PEP, this all goes away and instead we need to 
understand just one class, namely base_function.



Would it be possible to first decouple things, reducing the complexity,
and then tackle the individual problems?


What do you mean with "decouple things"? Can you be more concrete?


Currently, the "outside" of a function (how it looks when introspected) 
is tied to the "inside" (what happens internally when it's called). 
That's what I'd like to see decoupled.
Can we better enable pydoc/IPython developers to tackle introspection 
problems without wading deep in the internals and call optimizations?



The class hierarchy still makes it hard to decouple the introspection
side (how functions look on the outside) from the calling mechanism (how
the calling works internally).


Any class who wants to profit from fast function calls can inherit from 
base_function. It can add whatever attributes it wants and it can choose 
to implement documentation and/or introspection in whatever way it 
wants. It can choose to not care about that at all. That looks very 
decoupled to me.


But, it still has to inherit from base_function to "look like a 
function". Can we remove that limitation in favor of duck typing?



Starting from an idea and ironing out the details it lets you (and, if
since you published results, everyone else) figure out the tricky
details. But ultimately it's exploring one path of doing things – it
doesn't necessarily lead to the best way of doing something.


So far I haven't seen any other proposals...


That's a good question. Maybe inspect.isfunction() serves too many use
cases to be useful. Cython functons should behave like "def" functions
in some cases, and like built-in functions in others.


 From the outside, i.e. user's point of view, I want them to behave like 

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-05-15 18:36, Petr Viktorin wrote:

Naturally, large-scale
changes have less of a chance there.


Does it really matter that much how large the change is? I think you are 
focusing too much on the change instead of the end result.


As I said in my previous post, I could certainly make less disruptive 
changes. But would that really be better? (If you think that the answer 
is "yes" here, I honestly want to know).


I could make the code less different than today but at the cost of added 
complexity. Building on top of the existing code is like building on a 
bad foundation: the higher you build, the messier it gets. Instead, I 
propose a solid new foundation. Of course, that requires more work to 
build but once it is built, the finished building looks a lot better.



With such a "finished product" PEP, it's hard to see if some of the
various problems could be solved in a better way -- faster, more
maintainable, or less disruptive.


With "faster", you mean runtime speed? I'm pretty confident that we 
won't lose anything there.


As I argued above, my PEP might very well make things "more 
maintainable", but this is of course very subjective. And "less 
disruptive" was never a goal for this PEP.



It's also harder from a psychological point of view: you obviously
already put in a lot of good work, and it's harder to waste that work if
an even better solution is found.


I hope that this won't be my psychology. As a developer, I prefer to 
focus on problems rather than on solutions: I don't want to push a 
particular solution, I want to fix a particular problem. If an even 
better solution is accepted, I will be a very happy man.


What I would hate is that this PEP gets rejected because some people 
claim that the problem can be solved in a better way, but without 
actually suggesting such a better way.



Is a branching class hierarchy, with quite a few new of flags for
feature selection, the kind of simplicity we want?


Maybe yes because it *concentrates* all complexity in one small place. 
Currently, we have several independent classes 
(builtin_function_or_method, method_descriptor, function, method) which 
all require various forms of special casing in the interpreter with some 
code duplication. With my PEP, this all goes away and instead we need to 
understand just one class, namely base_function.



Would it be possible to first decouple things, reducing the complexity,
and then tackle the individual problems?


What do you mean with "decouple things"? Can you be more concrete?


The class hierarchy still makes it hard to decouple the introspection
side (how functions look on the outside) from the calling mechanism (how
the calling works internally).


Any class who wants to profit from fast function calls can inherit from 
base_function. It can add whatever attributes it wants and it can choose 
to implement documentation and/or introspection in whatever way it 
wants. It can choose to not care about that at all. That looks very 
decoupled to me.



Starting from an idea and ironing out the details it lets you (and, if
since you published results, everyone else) figure out the tricky
details. But ultimately it's exploring one path of doing things – it
doesn't necessarily lead to the best way of doing something.


So far I haven't seen any other proposals...


That's a good question. Maybe inspect.isfunction() serves too many use
cases to be useful. Cython functons should behave like "def" functions
in some cases, and like built-in functions in others.


From the outside, i.e. user's point of view, I want them to behave like 
Python functions. Whether it's implemented in C or Python should just be 
an implementation detail. Of course there are attributes like __code__ 
which dive into implementation details, so there you will see the 
difference.



before we change how inspect.isfunction ultimately behaves,
I'd like to make its purpose clearer (and try to check how that meshes
with the current use cases).


The problem is that this is not easy to do. You could search CPython for 
occurrences of inspect.isfunction() and you could search your favorite 
Python projects. This will give you some indication, but I'm not sure 
whether that will be representative.


From what I can tell, inspect.isfunction() is mainly used as guard for 
attribute access: it implies for example that a __globals__ attribute 
exists. And it's used by documentation tools to decide that it should be 
documented as Python function whose signature can be extracted using 
inspect.signature().



Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-05-15 18:36, Petr Viktorin wrote:

What is your ultimate use case?


(I'll just answer this one question now and reply to the more technical 
comments in another thread)


My ultimate use case is being able to implement functions and methods 
which are


(A) equally fast as the existing built-in function and methods

(B) and behave from a user's point of view like Python functions.

With objective (A) I want no compromises. CPython has many optimizations 
for built-in functions and all of them should work for my new functions.


Objective (B) means more precisely:

1. Implementing __get__ to turn a function in a method.

2. Being recognized as "functions" by tools like Sphinx and IPython.

3. Introspection support such as inspect.signature() and 
inspect.getsource().



Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/15/18 05:15, Jeroen Demeyer wrote:

On 2018-05-14 19:56, Petr Viktorin wrote:

It does quite a lot of things, and the changes are all intertwined,
which will make it hard to get reviewed and accepted.


The problem is that many things *are* already intertwined currently. You 
cannot deal with functions without involving methods for example.


An important note is that it was never my goal to create a minimal PEP. 
I did not aim for changing as little as possible. I was thinking: we are 
changing functions, what would be the best way to implement them?


That might be a problem. For the change to be accepted, a core developer 
will need to commit to maintaining the code, understand it, and accept 
responsibility for anything that's broken. Naturally, large-scale 
changes have less of a chance there.


With such a "finished product" PEP, it's hard to see if some of the 
various problems could be solved in a better way -- faster, more 
maintainable, or less disruptive.
It's also harder from a psychological point of view: you obviously 
already put in a lot of good work, and it's harder to waste that work if 
an even better solution is found. (I always tell Marcel to view 
large-scale changes as a hands-on learning experiment -- more likely to 
be thrown away than accepted -- rather than as creating a finished project.)


The main goal was fixing introspection but a secondary goal was fixing 
many of the existing warts with functions. Probably this secondary goal 
will in the end be more important for the general Python community.


I would argue that my PEP may look complicated, but I'm sure that the 
end result will be a simpler implementation than we have today. Instead 
of having four related classes implementing similar functionality 
(builtin_function_or_method, method, method_descriptor and function), we 
have just one (base_function). The existing classes like method still 
exist with my PEP but a lot of the core functionality is implemented in 
the common base_function.


Is a branching class hierarchy, with quite a few new of flags for 
feature selection, the kind of simplicity we want?
Would it be possible to first decouple things, reducing the complexity, 
and then tackle the individual problems?


This is really one of the key points: while my PEP *could* be 
implemented without the base_function class, the resulting code would be 
far more complicated.



Are there parts that can be left to a subsequent PEP, to simplify the
document (and implementation)?


It depends. The current PEP is more or less a finished product. You can 
of course pick parts of the PEP and implement those, but then those 
parts will be somewhat meaningless individually.


But if PEP 575 is accepted "in principle" (you accept the new class 
hierarchy for functions), then the details could be spread over several 
PEPs. But those individual PEPs would only make sense in the light of 
PEP 575.


Well, that's the thing I'm not sure about.
The class hierarchy still makes it hard to decouple the introspection 
side (how functions look on the outside) from the calling mechanism (how 
the calling works internally). It fear that it is replacing complexity 
with a different kind of complexity.
So my main question now is, can this all be *simplified* rather than 
*reorganized*? It's a genuine question – I don't know, but I feel it 
should be explored more.


A few small details could be left out, such as METH_BINDING. But that 
wouldn't yield a significant simplification.



It seems to me that the current complexity is (partly) due to the fact
that how functions are *called* is tied to how they are *introspected*.


The *existing* situation is that introspection is totally tied to how 
functions are called. So I would argue that my PEP improves on that by 
removing some of those ties by moving __call__ to a common base class.



Maybe we can change `inspect` to use duck-typing instead of isinstance?


That was rejected on https://bugs.python.org/issue30071


Then, if built-in functions were subclassable, Cython functions could
need to provide appropriate __code__/__defaults__/__kwdefaults__
attributes that inspect would pick up.


Of course, that's possible. I don't think that it would be a *better* 
solution than my PEP though.


Essentially, my PEP started from that idea. But then you realize that 
you'll need to handle not only built-in functions but also method 
descriptors (unbound methods of extension types). And you'll want to 
allow __get__ for the new subclasses. For efficiency, you really want to 
implement __get__ in the base classes (both builtin_function_or_method 
and method_descriptor) because of optimizations combining __get__ and 
__call__ (the LOAD_METHOD and CALL_METHOD opcodes). And then you realize 
that it makes no sense to duplicate all that functionality in both 
classes. So you add a new base class. You already end up with a major 
part of my PEP this way.


Starting from an idea and ironing out the 

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-05-14 22:38, Petr Viktorin wrote:

Why are these flags added?


I made a minor edit to the PEP to remove those flags:
https://github.com/python/peps/pull/649
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-05-14 19:56, Petr Viktorin wrote:

It does quite a lot of things, and the changes are all intertwined,
which will make it hard to get reviewed and accepted.


The problem is that many things *are* already intertwined currently. You 
cannot deal with functions without involving methods for example.


An important note is that it was never my goal to create a minimal PEP. 
I did not aim for changing as little as possible. I was thinking: we are 
changing functions, what would be the best way to implement them?


The main goal was fixing introspection but a secondary goal was fixing 
many of the existing warts with functions. Probably this secondary goal 
will in the end be more important for the general Python community.


I would argue that my PEP may look complicated, but I'm sure that the 
end result will be a simpler implementation than we have today. Instead 
of having four related classes implementing similar functionality 
(builtin_function_or_method, method, method_descriptor and function), we 
have just one (base_function). The existing classes like method still 
exist with my PEP but a lot of the core functionality is implemented in 
the common base_function.


This is really one of the key points: while my PEP *could* be 
implemented without the base_function class, the resulting code would be 
far more complicated.



Are there parts that can be left to a subsequent PEP, to simplify the
document (and implementation)?


It depends. The current PEP is more or less a finished product. You can 
of course pick parts of the PEP and implement those, but then those 
parts will be somewhat meaningless individually.


But if PEP 575 is accepted "in principle" (you accept the new class 
hierarchy for functions), then the details could be spread over several 
PEPs. But those individual PEPs would only make sense in the light of 
PEP 575.


A few small details could be left out, such as METH_BINDING. But that 
wouldn't yield a significant simplification.



It seems to me that the current complexity is (partly) due to the fact
that how functions are *called* is tied to how they are *introspected*.


The *existing* situation is that introspection is totally tied to how 
functions are called. So I would argue that my PEP improves on that by 
removing some of those ties by moving __call__ to a common base class.



Maybe we can change `inspect` to use duck-typing instead of isinstance?


That was rejected on https://bugs.python.org/issue30071


Then, if built-in functions were subclassable, Cython functions could
need to provide appropriate __code__/__defaults__/__kwdefaults__
attributes that inspect would pick up.


Of course, that's possible. I don't think that it would be a *better* 
solution than my PEP though.


Essentially, my PEP started from that idea. But then you realize that 
you'll need to handle not only built-in functions but also method 
descriptors (unbound methods of extension types). And you'll want to 
allow __get__ for the new subclasses. For efficiency, you really want to 
implement __get__ in the base classes (both builtin_function_or_method 
and method_descriptor) because of optimizations combining __get__ and 
__call__ (the LOAD_METHOD and CALL_METHOD opcodes). And then you realize 
that it makes no sense to duplicate all that functionality in both 
classes. So you add a new base class. You already end up with a major 
part of my PEP this way.


That still leaves the issue of what inspect.isfunction() should do. 
Often, "isfunction" is used to check for "has introspection" so you 
certainly want to allow for custom built-in function classes to satisfy 
inspect.isfunction(). So you need to involve Python functions too in the 
class hierarchy. And that's more or less my PEP.



Jeroen.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-05-14 22:38, Petr Viktorin wrote:

Why are these flags added?
They aren't free – the space of available flags is not infinite. If
something (Cython?) needs eight of them, it would be nice to mention the
use case, at least as an example.

What should Python do with a m_methods entry that has METH_CUSTOM set?
Again it would be nice to have an example or use case.


They have no specific use case. I just added this because it made sense 
abstractly. I can remove this from my PEP to simplify it.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/05/18 04:55, Jeroen Demeyer wrote:

Hello all,

I have updated PEP 575 in response to some posts on this mailing list 
and to some discussions in person with the core Cython developers.

See https://www.python.org/dev/peps/pep-0575/

The main differences with respect to the previous version are:

* "builtin_function" was renamed to "cfunction". Since we are changing 
the name anyway, "cfunction" looked like a better choice because the 
word "built-in" typically refers to things from the builtins module.


* defined_function now only defines an API (it must support all 
attributes that a Python function has) without specifying the 
implementation.


* The "Two-phase Implementation" proposal for better backwards 
compatibility has been expanded and now offers 100% backwards 
compatibility for the classes and for the inspect functions.


The PEP says:


User flags: METH_CUSTOM and METH_USRx
These flags are meant for applications that want to use tp_methods for an 
extension type or m_methods for a module but that do not want the default 
built-in functions to be created. Those applications would set METH_CUSTOM. The 
application is also free to use METH_USR0, ..., METH_USR7 for its own purposes, 
for example to customize the creation of special function instances.

There is no immediate concrete use case, but we expect that tools which 
auto-generate functions or extension types may want to define custom flags. 
Given that it costs essentially nothing to have these flags, it seems like a 
good idea to allow it.


Why are these flags added?
They aren't free – the space of available flags is not infinite. If 
something (Cython?) needs eight of them, it would be nice to mention the 
use case, at least as an example.


What should Python do with a m_methods entry that has METH_CUSTOM set? 
Again it would be nice to have an example or use case.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Petr Viktorin]

On 05/05/18 04:55, Jeroen Demeyer wrote:

Hello all,

I have updated PEP 575 in response to some posts on this mailing list 
and to some discussions in person with the core Cython developers.

See https://www.python.org/dev/peps/pep-0575/

The main differences with respect to the previous version are:

* "builtin_function" was renamed to "cfunction". Since we are changing 
the name anyway, "cfunction" looked like a better choice because the 
word "built-in" typically refers to things from the builtins module.


* defined_function now only defines an API (it must support all 
attributes that a Python function has) without specifying the 
implementation.


* The "Two-phase Implementation" proposal for better backwards 
compatibility has been expanded and now offers 100% backwards 
compatibility for the classes and for the inspect functions.



Hi,
I'm reading the PEP thoroughly, trying to "swap it into my brain" for 
the next few days.
It does quite a lot of things, and the changes are all intertwined, 
which will make it hard to get reviewed and accepted.
Are there parts that can be left to a subsequent PEP, to simplify the 
document (and implementation)?
It seems to me that the current complexity is (partly) due to the fact 
that how functions are *called* is tied to how they are *introspected*. 
Perhaps starting to separate that is a better way to untangle things 
than arranging a class hierarchy?



Can the problem of allowing introspection ("It is currently not possible 
to implement a function efficiently in C (only built-in functions can do 
that) while still allowing introspection like inspect.signature or 
inspect.getsourcefile (only Python functions can do that)") be solved in 
a better way?


Maybe we can change `inspect` to use duck-typing instead of isinstance? 
Then, if built-in functions were subclassable, Cython functions could 
need to provide appropriate __code__/__defaults__/__kwdefaults__ 
attributes that inspect would pick up.
Maybe we could eve add more attributes (__isgenerator__?) to separate 
how a function is called from how it should be introspected -- e.g. make 
inspect not consult co_flags.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update: limit?

[Paddy McCarthy]

On Tue, 8 May 2018, 16:33 Jeroen Demeyer,  wrote:

> On 2018-05-06 09:35, Nick Coghlan wrote:
> > Thanks for this update Jeroen! If it doesn't come up otherwise, I'll try
> > to claim one of the lightning talk slots at the Language Summit to
> > discuss this with folks in person :)
>
> Sounds great! I'd love to hear what people think.
>
> As an example of how the new functionality of PEP 575 can be used, I
> changed functools.lru_cache to implement the _lru_cache_wrapper class as
> subclass of base_function. I added this to the reference implementation
> https://github.com/jdemeyer/cpython/tree/pep575
> ___
>
It is not so much the use, but the abuse that I am worried about. If their
was a way to limit the complexity of expression allowed around the use of
the name assigned to by :=, then that would be a start.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Jeroen Demeyer]

On 2018-05-06 09:35, Nick Coghlan wrote:

Thanks for this update Jeroen! If it doesn't come up otherwise, I'll try
to claim one of the lightning talk slots at the Language Summit to
discuss this with folks in person :)


Sounds great! I'd love to hear what people think.

As an example of how the new functionality of PEP 575 can be used, I 
changed functools.lru_cache to implement the _lru_cache_wrapper class as 
subclass of base_function. I added this to the reference implementation

https://github.com/jdemeyer/cpython/tree/pep575
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

[Nick Coghlan]

On 5 May 2018 at 18:55, Jeroen Demeyer  wrote:

> Hello all,
>
> I have updated PEP 575 in response to some posts on this mailing list and
> to some discussions in person with the core Cython developers.
> See https://www.python.org/dev/peps/pep-0575/
>
> The main differences with respect to the previous version are:
>
> * "builtin_function" was renamed to "cfunction". Since we are changing the
> name anyway, "cfunction" looked like a better choice because the word
> "built-in" typically refers to things from the builtins module.
>
> * defined_function now only defines an API (it must support all attributes
> that a Python function has) without specifying the implementation.
>
> * The "Two-phase Implementation" proposal for better backwards
> compatibility has been expanded and now offers 100% backwards compatibility
> for the classes and for the inspect functions.
>

Thanks for this update Jeroen! If it doesn't come up otherwise, I'll try to
claim one of the lightning talk slots at the Language Summit to discuss
this with folks in person :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com