[Python-ideas] Re: Extension methods in Python
On Wed, Jun 23, 2021 at 11:40 AM Steven D'Aprano wrote:
>
> On Tue, Jun 22, 2021 at 10:25:33PM +1000, Chris Angelico wrote:
>
> > > If its a problem for getattr, it is a problem for dot syntax, because
> > > they are essentially the same thing.
> >
> > Ahh but that is precisely the problem.
>
> Is it? Don't be shy. Tell us what the problem is and why its a problem.
That's exactly what the rest of the post is about.
> > > > How is getattr defined?
> > >
> > > The same as it is defined now, except with some minor tweaks to support
> > > extension methods.
> >
> > Do those tweaks include reaching back into the module that called it?
> > How magical will it be?
>
> I thought you agreed that we didn't need to discuss implementation until
> we had decided on the desired semantics?
>
> Let's just say it will be a well-defined, totally non-magical
> implementation (like everything else in Python) that manages to be
> equally efficient as regular attribute access. A high bar to set.
> Implementation issues may require us to dial that back a bit, or might
> even rule out the concept altogether, but let's start off by assuming
> the best and decide on the semantics first.
Semantics are *exactly* what I'm talking about.
> > This means that the getattr() function, being a perfectly
> > straight-forward function, is not going to see any extension methods.
>
> Does getattr see slots (both C-level and Python)? Yes. Does it see
> attributes in instance and class dicts? Yes. Does it see dynamic
> attributes that use `__getattr__`? Yes. Does it understand the
> descriptor protocol? Yes.
>
> It does everything else dot notation does. Why wouldn't it see extension
> methods? (Apart from spite.)
>
> The getattr builtin is just a public interface to whatever internal
> function or functions the interpreter uses to look up attributes.
Okay. Lemme give it to you *even more clearly* since the previous
example didn't satisfy.
# file1.py
@extend(list)
def in_order(self):
return sorted(self)
def frob(stuff):
return stuff.in_order()
# file2.py
from file1 import frob
thing = [1, 5, 2]
frob(thing) # == [1, 2, 5]
def otherfrob(stuff):
return stuff.in_order()
otherfrob(thing) # AttributeError
Am I correct so far? The function imported from file1 has the
extension method, the code in file2 does not. That's the entire point
here, right?
Okay. Now, what if getattr is brought into the mix?
# file3.py
@extend(list)
def in_order(self):
return sorted(self)
def fetch1(stuff, attr):
if attr == "in_order": return stuff.in_order
if attr == "unordered": return stuff.unordered
return getattr(stuff, attr)
def fetch2(stuff, attr):
return getattr(stuff, attr)
# file4.py
from file3 import fetch1, fetch2
import random
@extend(list)
def unordered(self):
return random.shuffle(self[:])
def fetch3(stuff, attr):
if attr == "in_order": return stuff.in_order
if attr == "unordered": return stuff.unordered
return getattr(stuff, attr)
def fetch4(stuff, attr):
return getattr(stuff, attr)
thing = [1, 5, 2]
fetch1(thing, "in_order")()
fetch2(thing, "in_order")()
fetch3(thing, "in_order")()
fetch4(thing, "in_order")()
fetch1(thing, "unordered")()
fetch2(thing, "unordered")()
fetch3(thing, "unordered")()
fetch4(thing, "unordered")()
Okay. *NOW* which ones raise AttributeError, and which ones give the
extension method? What exactly are the semantics of getattr? Is it a
magical function that can reach back into the module that called it,
or is it actually a function of its own? And if getattr is supposed to
reach back into the other module, why shouldn't other functions be
able to?
Please explain exactly what the semantics of getattr are, and exactly
which modules it is supposed to be able to see. Remember, it is not a
compiler construct or an operator. It is a function, and it lives in
its own module (the builtins).
> Not a rhetorical question: is that how it works in something like Swift,
> or Kotlin?
I have no idea. I'm just asking how you intend it to work in Python.
If you want to cite other languages, go ahead, but I'm not assuming
that they already have the solution, because they are different
languages. Also not a rhetorical question: Is their getattr equivalent
actually an operator or compiler construct, rather than being a
regular function? Because if it is, then the entire problem doesn't
exist.
> > And what about this?
> >
> > f = functools.partial(getattr, stuff)
> > f("in_order")
> >
> > NOW which extension methods should apply? Those registered here? Those
> > registered in the builtins? Those registered in functools?
>
> partial is just a wrapper around its function argument, so that should
> behave *exactly* the same as `getattr(stuff, 'in_order')`.
So if it behaves exactly the same way that getattr would, then is it
exactly the same as fetch2 and fetch4? If not, how is it different?
What about other functions implemented in C? If I write a C module
that calls
[Python-ideas] Re: Extension methods in Python
On Tue, Jun 22, 2021 at 10:25:33PM +1000, Chris Angelico wrote:
> > If its a problem for getattr, it is a problem for dot syntax, because
> > they are essentially the same thing.
>
> Ahh but that is precisely the problem.
Is it? Don't be shy. Tell us what the problem is and why its a problem.
> > > How is getattr defined?
> >
> > The same as it is defined now, except with some minor tweaks to support
> > extension methods.
>
> Do those tweaks include reaching back into the module that called it?
> How magical will it be?
I thought you agreed that we didn't need to discuss implementation until
we had decided on the desired semantics?
Let's just say it will be a well-defined, totally non-magical
implementation (like everything else in Python) that manages to be
equally efficient as regular attribute access. A high bar to set.
Implementation issues may require us to dial that back a bit, or might
even rule out the concept altogether, but let's start off by assuming
the best and decide on the semantics first.
[...]
> Let me clarify then.
Thank you, that would be helpful.
> We shall assume for the moment that the builtins module does not have
> any extension methods registered. (I suppose it could, but then you
> get all the action-at-a-distance of monkey-patching AND the problems
> of extension methods, so I would hope people don't do this.)
How do you know that the builtins aren't already using extension
methods?
Let's pretend that you didn't know that CPython's implementation was C
rather than C#. Or that C has support for something similar to extension
methods. (I daresay you could simulate it, somehow.) Or that we're
talking about IronPython, for example, which is implemented in C#. I
might tell you that list.sort and list.index are regular methods, and
that list.append and list.reverse are extension methods.
Short of looking at the source code, there would be absolutely no way
for you to tell if I were correct or not.
So if builtins used extension methods, that would be indistinguishable
from builtins not using extension methods.
(To be pedantic: this would only be true if those extension methods were
added at interpreter startup, before the interpreter ran any user code.
Otherwise you could take a snapshot of `dir(list)` before and after, and
inspect the differences.)
To be clear, this is distinct from *a user module* using extension
methods on a builtin type, which is normal and the point of the
exercise.
Do I need to explain the difference between the interpreter using
extension methods as part of the builtins implementation, and user-
written modules ("spam.py") extending builtin classes with extension
types? Because they are completely different things.
> This means that the getattr() function, being a perfectly
> straight-forward function, is not going to see any extension methods.
Does getattr see slots (both C-level and Python)? Yes. Does it see
attributes in instance and class dicts? Yes. Does it see dynamic
attributes that use `__getattr__`? Yes. Does it understand the
descriptor protocol? Yes.
It does everything else dot notation does. Why wouldn't it see extension
methods? (Apart from spite.)
The getattr builtin is just a public interface to whatever internal
function or functions the interpreter uses to look up attributes.
> # whatever the actual syntax is
> @extend(list)
> def in_order(self):
> return sorted(self)
>
> stuff = [1, 5, 2]
> stuff.in_order() # == [1, 2, 5]
> getattr(stuff, "in_order")() # AttributeError
What reason do you have for thinking that would be how it works?
Not a rhetorical question: is that how it works in something like Swift,
or Kotlin?
> Does the getattr function see the extension methods? If so, which?
Yes, and the same ones you would see if you used dot syntax.
> If not, how can getattr return the same thing as attribute lookup
> does?
I think you've just answered your own question. getattr has to return
the same thing as attribute lookup, because if it didn't, it wouldn't be
returning the same thing as attribute lookup, which is getattr's reason
to exist.
> How do you inform getattr of which extension methods it should be looking at?
You don't. You inform the interpreter tha you are opting in to use
extension methods on a type, the interpreter does whatever it needs to
do to make it work (implementation), and then it Just Works™.
> And what about this?
>
> f = functools.partial(getattr, stuff)
> f("in_order")
>
> NOW which extension methods should apply? Those registered here? Those
> registered in the builtins? Those registered in functools?
partial is just a wrapper around its function argument, so that should
behave *exactly* the same as `getattr(stuff, 'in_order')`.
> Yes, monkey-patching *is* cleaner, because the object is the same
> object no matter how you look it up.
Oh for heaven's sake, I'm not proposing changes to Python's object
identity model! Please don't
[Python-ideas] Re: Extension methods in Python
On 2021-06-22 8:11 p.m., Chris Angelico wrote: > On Wed, Jun 23, 2021 at 9:06 AM Soni L. wrote: > > On 2021-06-22 7:38 p.m., Chris Angelico wrote: > > > Have you actually tried designing this into a larger project to see > > > what problems you run into, or is this something you've only > > > considered at this trivial level? > > > > 1. It's opt-in. > > 2. It's designed to be used by a hypothetical extension methods module, > > but without imposing any design constraints on such module. It could > > return a named function every time a given name is looked up (a la "bind > > the first argument" operator), or do dynamic dispatch based on types or > > ABCs (a la proper extension methods). > > > > In practice, you don't def your own __dot__, but rather use someone > > else's "__dot__ builder". If you don't wanna deal with it, just don't > > use __dot__. > > > > It's also useful for the occasional domain-specific language. > > > > What I'm hearing from you is: "No". Lots of words to say it, but, the > answer to my question seems to be no. We have never felt a need for extension methods in python, no. > > ChrisA > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/BZ7BR5ACN2VXJQA6LVV6FNSOCKFVQTWT/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/Q57V3QDZ6O6WZZXYQDG4E3RRFSPZO62L/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On Wed, Jun 23, 2021 at 9:06 AM Soni L. wrote: > On 2021-06-22 7:38 p.m., Chris Angelico wrote: > > Have you actually tried designing this into a larger project to see > > what problems you run into, or is this something you've only > > considered at this trivial level? > > 1. It's opt-in. > 2. It's designed to be used by a hypothetical extension methods module, > but without imposing any design constraints on such module. It could > return a named function every time a given name is looked up (a la "bind > the first argument" operator), or do dynamic dispatch based on types or > ABCs (a la proper extension methods). > > In practice, you don't def your own __dot__, but rather use someone > else's "__dot__ builder". If you don't wanna deal with it, just don't > use __dot__. > > It's also useful for the occasional domain-specific language. > What I'm hearing from you is: "No". Lots of words to say it, but, the answer to my question seems to be no. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BZ7BR5ACN2VXJQA6LVV6FNSOCKFVQTWT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On 2021-06-22 7:38 p.m., Chris Angelico wrote: > On Wed, Jun 23, 2021 at 8:30 AM Soni L. wrote: > > > > > > > > On 2021-06-22 5:54 p.m., Chris Angelico wrote: > > > On Wed, Jun 23, 2021 at 6:41 AM Soni L. wrote: > > > > It would have local scope, similar to uh... locals. Y'know how locals > > > > are just sugar for locals()['foo'] and stuff? Yeah. > > > > > > Not really, no, they're not. :) The dictionary returned by locals() > > > isn't actually an implementation detail of local name lookups. > > > > It's... part of the language. Not an implementation detail. The > > dictionary returned by locals() is an inherent part of local name > > lookups, isn't it? > > No, it's not. Most definitely not. > > https://docs.python.org/3/library/functions.html#locals Ohh. Fair enough, sorry. > > > Have you put any thought into how you would deal with the problem of > > > recursive __dot__ calls? > > > > Let it recurse! > > > > Globals and locals don't go through __dot__, so you can just... use > > them. In particular, you can always use getattr(), and probably should. > > Or even set __dot__ to getattr inside it, like so: > > > > def __dot__(left, right): > > __dot__ = getattr > > foo.bar # same as getattr(foo, "bar") because we set (local) __dot__ > > to getattr above > > I can't actually pin down what I'm averse to here, but it gives me a > really REALLY bad feeling. You're expecting every attribute lookup to > now look for a local or global name __dot__ (or, presumably, a > nonlocal, class, or builtin), and do whatever that does. That seems > like a really effective foot-gun. > > Have you actually tried designing this into a larger project to see > what problems you run into, or is this something you've only > considered at this trivial level? 1. It's opt-in. 2. It's designed to be used by a hypothetical extension methods module, but without imposing any design constraints on such module. It could return a named function every time a given name is looked up (a la "bind the first argument" operator), or do dynamic dispatch based on types or ABCs (a la proper extension methods). In practice, you don't def your own __dot__, but rather use someone else's "__dot__ builder". If you don't wanna deal with it, just don't use __dot__. It's also useful for the occasional domain-specific language. > > ChrisA > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/YOPTCIYV4GSW3N7EA7KPLJBKNVSNANXZ/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LQW3GLWRGPOZDOAS7LGUL4WLP6JOUKUR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On Wed, Jun 23, 2021 at 8:30 AM Soni L. wrote: > > > > On 2021-06-22 5:54 p.m., Chris Angelico wrote: > > On Wed, Jun 23, 2021 at 6:41 AM Soni L. wrote: > > > It would have local scope, similar to uh... locals. Y'know how locals > > > are just sugar for locals()['foo'] and stuff? Yeah. > > > > Not really, no, they're not. :) The dictionary returned by locals() > > isn't actually an implementation detail of local name lookups. > > It's... part of the language. Not an implementation detail. The > dictionary returned by locals() is an inherent part of local name > lookups, isn't it? No, it's not. Most definitely not. https://docs.python.org/3/library/functions.html#locals > > Have you put any thought into how you would deal with the problem of > > recursive __dot__ calls? > > Let it recurse! > > Globals and locals don't go through __dot__, so you can just... use > them. In particular, you can always use getattr(), and probably should. > Or even set __dot__ to getattr inside it, like so: > > def __dot__(left, right): > __dot__ = getattr > foo.bar # same as getattr(foo, "bar") because we set (local) __dot__ > to getattr above I can't actually pin down what I'm averse to here, but it gives me a really REALLY bad feeling. You're expecting every attribute lookup to now look for a local or global name __dot__ (or, presumably, a nonlocal, class, or builtin), and do whatever that does. That seems like a really effective foot-gun. Have you actually tried designing this into a larger project to see what problems you run into, or is this something you've only considered at this trivial level? ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/YOPTCIYV4GSW3N7EA7KPLJBKNVSNANXZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On 2021-06-22 5:34 p.m., Brendan Barnwell wrote: > On 2021-06-22 13:09, Soni L. wrote: >> Think about it like this, extension methods give you the ability to make >> imported functions that look like this: >> >> foo(bar, baz) >> >> look like this instead: >> >> bar.foo(baz) >> >> That's all there is to them. They're just a lie to change how you >> read/write the code. Some languages have an whole operator that has a >> similar function, where something like bar->foo(baz) is sugar for >> foo(bar, baz). The OP doesn't specify any particular mechanism for >> extension methods, so e.g. making the dot operator be implemented by a >> local function in the module, which delegates to the current attribute >> lookup mechanism by default, would be perfectly acceptable. It's like >> deprecating the existing dot operator and introducing a completely >> different one that has nothing to do with attribute lookup! > > Okay, if that's the case, then I just think it's a bad idea. :-) > > We already have a definition for what bar.foo does, and it's > totally under the control of the bar object (via the > __getattr__/__getattribute__ mechanism). The idea that other things > would be able to hook in there does not appeal to me at all. > > I don't really understand why you would want such a thing, to be > honest. I feel it would make code way more difficult to reason about, > as it would break locality constraints every which way. Now every > time you see`bar.foo` you would have to think about all kinds of other > modules that may be hooking in and adding their own complications. > What's the point? > > Mostly the whole benefit of the dot notation is that it specifies > a locally constrained relationship between the object and the > attribute: you know that bar.foo does what bar decides, and no one > else gets any say (unless bar asks for their opinion, e.g. by > consulting global variables or whatever). If we want to write > foo(bar, baz). . . well, we can just do that! What you're describing > would just make existing attribute usages harder to understand while > only "adding" something we can already do quite straightforwardly. > > Imagine a similar proposal for other syntax. Suppose that in any > module I could define a function called operator_add and then other > modules could "import" this "extension" so that every use of the + > operator would somehow hook into this operator_add function. So now > every time you do 2 + 2 you might be invoking some extension behavior. > In my view that is unambiguously a road to madness, and as far as I > can tell the extension mechanism you're proposing is equally ill-advised. > Imagine if Python didn't have an + operator, but instead an + *infix function*. Thus, every module would automatically include the global def infix +(left, right): ... And indeed, you could say we already have this. Except currently you can't define your own local infix +. But what if you *could*? What if you could just, # file1.py def infix +(left, right): return left << right x = 4 + 4 # file2.py def infix +(left, right): return left ** right x = 4 + 4 # file3.py import file1 import file2 print(file1.x) # 64 print(file2.x) # 256 print(4 + 4) # 8 How does this break locality? Same idea with the dot operator, really. (Some languages don't have operators, but only functions. They let you do just this.) ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/QBQCSJELOHSPOG24X2LMSYBSKCGQRVYP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On 2021-06-22 5:54 p.m., Chris Angelico wrote: > On Wed, Jun 23, 2021 at 6:41 AM Soni L. wrote: > > > > > > > > On 2021-06-22 5:23 p.m., Chris Angelico wrote: > > > On Wed, Jun 23, 2021 at 6:13 AM Soni L. wrote: > > > > Think about it like this, extension methods give you the ability to make > > > > imported functions that look like this: > > > > > > > > foo(bar, baz) > > > > > > > > look like this instead: > > > > > > > > bar.foo(baz) > > > > > > > > That's all there is to them. They're just a lie to change how you > > > > read/write the code. Some languages have an whole operator that has a > > > > similar function, where something like bar->foo(baz) is sugar for > > > > foo(bar, baz). The OP doesn't specify any particular mechanism for > > > > extension methods, so e.g. making the dot operator be implemented by a > > > > local function in the module, which delegates to the current attribute > > > > lookup mechanism by default, would be perfectly acceptable. It's like > > > > deprecating the existing dot operator and introducing a completely > > > > different one that has nothing to do with attribute lookup! > > > > > > uh... I'm lost. Are you saying that that's a good thing? You *want* to > > > replace the existing dot operator with one that has nothing to do with > > > attribute lookup?? I don't get it. > > > > Sure! As long as the new one can call getattr! > > > > Let's say the new dot operator looks like this: > > > > # file1.py > > def __dot__(left, right): > > print(left) > > print(right) > > ... > > foo = [] > > foo.bar > > > > Now, this would actually print the list [] and the string "bar". > > > > Then you can just use getattr to get attribute lookup behaviour out of it! > > > > def __dot__(left, right): > > return getattr(left, right) > > foo = [] > > foo.bar > > > > It would have local scope, similar to uh... locals. Y'know how locals > > are just sugar for locals()['foo'] and stuff? Yeah. > > Not really, no, they're not. :) The dictionary returned by locals() > isn't actually an implementation detail of local name lookups. It's... part of the language. Not an implementation detail. The dictionary returned by locals() is an inherent part of local name lookups, isn't it? > Have you put any thought into how you would deal with the problem of > recursive __dot__ calls? Let it recurse! Globals and locals don't go through __dot__, so you can just... use them. In particular, you can always use getattr(), and probably should. Or even set __dot__ to getattr inside it, like so: def __dot__(left, right): __dot__ = getattr foo.bar # same as getattr(foo, "bar") because we set (local) __dot__ to getattr above In languages with lexical scoping (instead of block scoping), the compiler doesn't see things that haven't yet been declared. In those languages, such a __dot__ function would actually inherit the global __dot__ rather than recursing. But as you can see from the above example, it's really not a big deal. > > ChrisA > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/NEI23TBWA3QKRGYIWTBJV3SK5O6MWXZC/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/X7J5SNYPDJTCC5LKBG4SHA5YEQAU5AMA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On Wed, Jun 23, 2021 at 6:41 AM Soni L. wrote: > > > > On 2021-06-22 5:23 p.m., Chris Angelico wrote: > > On Wed, Jun 23, 2021 at 6:13 AM Soni L. wrote: > > > Think about it like this, extension methods give you the ability to make > > > imported functions that look like this: > > > > > > foo(bar, baz) > > > > > > look like this instead: > > > > > > bar.foo(baz) > > > > > > That's all there is to them. They're just a lie to change how you > > > read/write the code. Some languages have an whole operator that has a > > > similar function, where something like bar->foo(baz) is sugar for > > > foo(bar, baz). The OP doesn't specify any particular mechanism for > > > extension methods, so e.g. making the dot operator be implemented by a > > > local function in the module, which delegates to the current attribute > > > lookup mechanism by default, would be perfectly acceptable. It's like > > > deprecating the existing dot operator and introducing a completely > > > different one that has nothing to do with attribute lookup! > > > > uh... I'm lost. Are you saying that that's a good thing? You *want* to > > replace the existing dot operator with one that has nothing to do with > > attribute lookup?? I don't get it. > > Sure! As long as the new one can call getattr! > > Let's say the new dot operator looks like this: > > # file1.py > def __dot__(left, right): > print(left) > print(right) > ... > foo = [] > foo.bar > > Now, this would actually print the list [] and the string "bar". > > Then you can just use getattr to get attribute lookup behaviour out of it! > > def __dot__(left, right): > return getattr(left, right) > foo = [] > foo.bar > > It would have local scope, similar to uh... locals. Y'know how locals > are just sugar for locals()['foo'] and stuff? Yeah. Not really, no, they're not. :) The dictionary returned by locals() isn't actually an implementation detail of local name lookups. Have you put any thought into how you would deal with the problem of recursive __dot__ calls? ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/NEI23TBWA3QKRGYIWTBJV3SK5O6MWXZC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On 2021-06-22 13:09, Soni L. wrote: Think about it like this, extension methods give you the ability to make imported functions that look like this: foo(bar, baz) look like this instead: bar.foo(baz) That's all there is to them. They're just a lie to change how you read/write the code. Some languages have an whole operator that has a similar function, where something like bar->foo(baz) is sugar for foo(bar, baz). The OP doesn't specify any particular mechanism for extension methods, so e.g. making the dot operator be implemented by a local function in the module, which delegates to the current attribute lookup mechanism by default, would be perfectly acceptable. It's like deprecating the existing dot operator and introducing a completely different one that has nothing to do with attribute lookup! Okay, if that's the case, then I just think it's a bad idea. :-) We already have a definition for what bar.foo does, and it's totally under the control of the bar object (via the __getattr__/__getattribute__ mechanism). The idea that other things would be able to hook in there does not appeal to me at all. I don't really understand why you would want such a thing, to be honest. I feel it would make code way more difficult to reason about, as it would break locality constraints every which way. Now every time you see`bar.foo` you would have to think about all kinds of other modules that may be hooking in and adding their own complications. What's the point? Mostly the whole benefit of the dot notation is that it specifies a locally constrained relationship between the object and the attribute: you know that bar.foo does what bar decides, and no one else gets any say (unless bar asks for their opinion, e.g. by consulting global variables or whatever). If we want to write foo(bar, baz). . . well, we can just do that! What you're describing would just make existing attribute usages harder to understand while only "adding" something we can already do quite straightforwardly. Imagine a similar proposal for other syntax. Suppose that in any module I could define a function called operator_add and then other modules could "import" this "extension" so that every use of the + operator would somehow hook into this operator_add function. So now every time you do 2 + 2 you might be invoking some extension behavior. In my view that is unambiguously a road to madness, and as far as I can tell the extension mechanism you're proposing is equally ill-advised. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4T35JHWPGIPWONX4PD7YPWEA2RXROSZI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On 2021-06-22 5:23 p.m., Chris Angelico wrote: > On Wed, Jun 23, 2021 at 6:13 AM Soni L. wrote: > > Think about it like this, extension methods give you the ability to make > > imported functions that look like this: > > > > foo(bar, baz) > > > > look like this instead: > > > > bar.foo(baz) > > > > That's all there is to them. They're just a lie to change how you > > read/write the code. Some languages have an whole operator that has a > > similar function, where something like bar->foo(baz) is sugar for > > foo(bar, baz). The OP doesn't specify any particular mechanism for > > extension methods, so e.g. making the dot operator be implemented by a > > local function in the module, which delegates to the current attribute > > lookup mechanism by default, would be perfectly acceptable. It's like > > deprecating the existing dot operator and introducing a completely > > different one that has nothing to do with attribute lookup! > > uh... I'm lost. Are you saying that that's a good thing? You *want* to > replace the existing dot operator with one that has nothing to do with > attribute lookup?? I don't get it. Sure! As long as the new one can call getattr! Let's say the new dot operator looks like this: # file1.py def __dot__(left, right): print(left) print(right) ... foo = [] foo.bar Now, this would actually print the list [] and the string "bar". Then you can just use getattr to get attribute lookup behaviour out of it! def __dot__(left, right): return getattr(left, right) foo = [] foo.bar It would have local scope, similar to uh... locals. Y'know how locals are just sugar for locals()['foo'] and stuff? Yeah. > > ChrisA > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/XDY3VAYX3FXJI7ZVT3AYAFA7PGQMBWAA/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RGLMGLHAHLJVN3G5WEQSEKURUY7OP2UD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On Wed, Jun 23, 2021 at 6:13 AM Soni L. wrote: > Think about it like this, extension methods give you the ability to make > imported functions that look like this: > > foo(bar, baz) > > look like this instead: > > bar.foo(baz) > > That's all there is to them. They're just a lie to change how you > read/write the code. Some languages have an whole operator that has a > similar function, where something like bar->foo(baz) is sugar for > foo(bar, baz). The OP doesn't specify any particular mechanism for > extension methods, so e.g. making the dot operator be implemented by a > local function in the module, which delegates to the current attribute > lookup mechanism by default, would be perfectly acceptable. It's like > deprecating the existing dot operator and introducing a completely > different one that has nothing to do with attribute lookup! uh... I'm lost. Are you saying that that's a good thing? You *want* to replace the existing dot operator with one that has nothing to do with attribute lookup?? I don't get it. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XDY3VAYX3FXJI7ZVT3AYAFA7PGQMBWAA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On 2021-06-22 3:43 p.m., Brendan Barnwell wrote:
> On 2021-06-22 05:14, Chris Angelico wrote:
>> Fair point. However, I've worked with a good number of languages that
>> have some notion of object methods, and generally, an object has or
>> doesn't have a method based on what the object*is*, not on who's
>> asking.
>
> I agree, and this is the aspect of the proposal that most confuses
> me. I still can't understand concretely what is being proposed,
> though, so I'm not sure I even understand it. Can someone clarify?
> Suppose I have this
>
> *
> ### file1.py
> @extend(list)
> def len2(self):
> return len(self)**2
>
> ### file2.py
> # or whatever I do to say "I want to use extensions to list defined in
> file1"
> from file1 extend list
>
> def coolness(some_list):
> return some_list.len2() + 1
>
> my_list = [1, 2, 3]
> print("My list len2:", my_list.len2())
> print("My list coolness:", coolness(my_list))
>
> ### file3.py
> import file2
>
> other_list = [1, 2, 3, 4]
> print("Other list len2:", other_list.len2())
> print("other list coolness:", file2.coolness(other_list))
> print("My list len2 from outside:", file2.my_list.len2())
> print("My list coolness from outside:", file2.coolness(file2.my_list))
> *
>
> What exactly is supposed to happen here if I run file3? file2
> declares use of file1's extensions. file2 does not. But file3 uses a
> function in file2 that makes use of such extensions. Who sees the
> extension?
>
NameError, value, NameError, value, respectively.
> The list object my_list in file2 is the same object accessed as
> file2.my_list in file3. Likewise coolness and file2.coolness. It is
> going to be super confusing if calling the same function object with
> the same list object argument gives different results depending on
> which file you're in. Likewise it's going to be confusing if the same
> list object sometimes has a .len2 method and sometimes doesn't.
It isn't the list object that has the extension method.
>
> But if it doesn't work that way, then it would seem to mean either
> every module sees the extensions (even if they didn't opt in), or else
> my_list in file2 is not the same object as file2.my_list in file3.
> And that would be even worse. (In this example it may seem okay
> because you can ask why I would call len2 from file3 if I didn't want
> to use it. But what if the extension is an override of an existing
> method? Is that not allowed?)
>
> In addition, if there is a difference between my_list and
> other_list, then that apparently means that the syntax for lists now
> does something different in the two files. This is maybe the most
> reasonable approach, since it's at least remotely reminiscent of a
> __future__ import, which changes syntactic behavior. But what exactly
> is the difference between the two objects here? Are both objects
> lists? If they are, then how can they have different methods? If
> they're not, then what are they?
>
> Most __future__ imports don't work like this. Maybe the closest
> thing is the generator_stop one, but at least that places a flag on
> the code object to indicate the difference. Would "extended lists"
> have some kind of magic attribute indicating which extensions they're
> using? That may have been marginally acceptable in the case of PEP
> 479, which was essentially a bugfix, and set the attribute on code
> objects which are an obscure internal data structure. But allowing
> this kind of thing for "user-facing" objects like lists would create a
> profusion of different list objects with different behavior depending
> on some combination of attributes indicating "what extends me" --- or,
> even worse, create different behavior without any such overt
> indication of which extensions are in use for a given object.
>
> The idea that the file in which code is written would somehow
> determine this type of runtime behavior seems to me to break my
> assumption that by knowing an object's identity I should have all the
> information I need to know about how to use it. Some of the posts
> earlier in this thread seem to suggest that somehow the module where
> something was defined (something --- not sure what --- maybe the
> object with the extended method? maybe the extended method itself?)
> would somehow get a hook to override attribute access on some objects
> (again, not sure which objects).
>
> That to me is the exact opposite of encapsulation. Encapsulation
> means the object itself contains all its behavior. If there is some
> getattr-like hook in some other module somewhere that is lying in wait
> to override attribute access on a given object "only sometimes" then
> that's not encapsulation at all. It's almost as bad as the infamous
> COME FROM statement!
>
> Existing mechanisms like __getattribute__ are not parallel at all.
> When you know an object's identity, you know its MRO, which tells you
> all you need to know about what __getattribute__
[Python-ideas] Re: Extension methods in Python
On 2021-06-22 05:14, Chris Angelico wrote:
Fair point. However, I've worked with a good number of languages that
have some notion of object methods, and generally, an object has or
doesn't have a method based on what the object*is*, not on who's
asking.
I agree, and this is the aspect of the proposal that most confuses me.
I still can't understand concretely what is being proposed, though, so
I'm not sure I even understand it. Can someone clarify? Suppose I have
this
*
### file1.py
@extend(list)
def len2(self):
return len(self)**2
### file2.py
# or whatever I do to say "I want to use extensions to list defined in
file1"
from file1 extend list
def coolness(some_list):
return some_list.len2() + 1
my_list = [1, 2, 3]
print("My list len2:", my_list.len2())
print("My list coolness:", coolness(my_list))
### file3.py
import file2
other_list = [1, 2, 3, 4]
print("Other list len2:", other_list.len2())
print("other list coolness:", file2.coolness(other_list))
print("My list len2 from outside:", file2.my_list.len2())
print("My list coolness from outside:", file2.coolness(file2.my_list))
*
What exactly is supposed to happen here if I run file3? file2 declares
use of file1's extensions. file2 does not. But file3 uses a function
in file2 that makes use of such extensions. Who sees the extension?
The list object my_list in file2 is the same object accessed as
file2.my_list in file3. Likewise coolness and file2.coolness. It is
going to be super confusing if calling the same function object with the
same list object argument gives different results depending on which
file you're in. Likewise it's going to be confusing if the same list
object sometimes has a .len2 method and sometimes doesn't.
But if it doesn't work that way, then it would seem to mean either
every module sees the extensions (even if they didn't opt in), or else
my_list in file2 is not the same object as file2.my_list in file3. And
that would be even worse. (In this example it may seem okay because you
can ask why I would call len2 from file3 if I didn't want to use it.
But what if the extension is an override of an existing method? Is that
not allowed?)
In addition, if there is a difference between my_list and other_list,
then that apparently means that the syntax for lists now does something
different in the two files. This is maybe the most reasonable approach,
since it's at least remotely reminiscent of a __future__ import, which
changes syntactic behavior. But what exactly is the difference between
the two objects here? Are both objects lists? If they are, then how
can they have different methods? If they're not, then what are they?
Most __future__ imports don't work like this. Maybe the closest thing
is the generator_stop one, but at least that places a flag on the code
object to indicate the difference. Would "extended lists" have some
kind of magic attribute indicating which extensions they're using? That
may have been marginally acceptable in the case of PEP 479, which was
essentially a bugfix, and set the attribute on code objects which are an
obscure internal data structure. But allowing this kind of thing for
"user-facing" objects like lists would create a profusion of different
list objects with different behavior depending on some combination of
attributes indicating "what extends me" --- or, even worse, create
different behavior without any such overt indication of which extensions
are in use for a given object.
The idea that the file in which code is written would somehow determine
this type of runtime behavior seems to me to break my assumption that by
knowing an object's identity I should have all the information I need to
know about how to use it. Some of the posts earlier in this thread seem
to suggest that somehow the module where something was defined
(something --- not sure what --- maybe the object with the extended
method? maybe the extended method itself?) would somehow get a hook to
override attribute access on some objects (again, not sure which objects).
That to me is the exact opposite of encapsulation. Encapsulation means
the object itself contains all its behavior. If there is some
getattr-like hook in some other module somewhere that is lying in wait
to override attribute access on a given object "only sometimes" then
that's not encapsulation at all. It's almost as bad as the infamous
COME FROM statement!
Existing mechanisms like __getattribute__ are not parallel at all.
When you know an object's identity, you know its MRO, which tells you
all you need to know about what __getattribute__ calls might happen.
You don't need to know anything about where the object "came from" or
what file you're using it in. But it seems with this proposal you would
need to know, and that's kind of creepy to me.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
[Python-ideas] Re: Extension methods in Python
On 2021-06-22 9:25 a.m., Chris Angelico wrote: > (Oh, and another wrinkle, although a small one: Code objects would > need to keep track of their modules. Currently functions do, but code > objects don't. But that seems unlikely to introduce further > complications.) What? No you just stop emitting LOAD_ATTR and instead emit LOAD_LOCAL '__opcode_load_attr_impl__' and call it with the relevant values. > > ChrisA > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/H5QSWNLWC4HOR3JPXBNDWCSE2QFHXPS6/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GLM2YWGRWHOAIQ7EJOXUNPGPY7HEEZPN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On Tue, Jun 22, 2021 at 9:56 PM Steven D'Aprano wrote:
>
> On Tue, Jun 22, 2021 at 09:12:53PM +1000, Chris Angelico wrote:
>
> > > The must be no semantic difference between:
> > >
> > > obj.method(arg)
> > >
> > > and
> > >
> > > getattr(obj, 'method')(arg)
> > >
> > > regardless of whether `method` is a regular method or an extension
> > > method.
> >
> > And this is a problem.
>
> If its a problem for getattr, it is a problem for dot syntax, because
> they are essentially the same thing.
Ahh but that is precisely the problem.
> > How is getattr defined?
>
> The same as it is defined now, except with some minor tweaks to support
> extension methods.
Do those tweaks include reaching back into the module that called it?
How magical will it be?
> > Is it counted as being in the current module?
>
> `getattr`? No, that's a builtin. You can shadow it or delete it if you
> want, it's just a public API to the underlying functionality built
> into the interpreter. Dot syntax won't be affected.
Let me clarify then.
We shall assume for the moment that the builtins module does not have
any extension methods registered. (I suppose it could, but then you
get all the action-at-a-distance of monkey-patching AND the problems
of extension methods, so I would hope people don't do this.) This
means that the getattr() function, being a perfectly straight-forward
function, is not going to see any extension methods.
Okay then.
# whatever the actual syntax is
@extend(list)
def in_order(self):
return sorted(self)
stuff = [1, 5, 2]
stuff.in_order() # == [1, 2, 5]
getattr(stuff, "in_order")() # AttributeError
Does the getattr function see the extension methods? If so, which? If
not, how can getattr return the same thing as attribute lookup does?
How do you inform getattr of which extension methods it should be looking at?
And what about this?
f = functools.partial(getattr, stuff)
f("in_order")
NOW which extension methods should apply? Those registered here? Those
registered in the builtins? Those registered in functools?
Yes, monkey-patching *is* cleaner, because the object is the same
object no matter how you look it up.
(Oh, and another wrinkle, although a small one: Code objects would
need to keep track of their modules. Currently functions do, but code
objects don't. But that seems unlikely to introduce further
complications.)
ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/H5QSWNLWC4HOR3JPXBNDWCSE2QFHXPS6/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Off topic: Rational discursive thought and helping others
Off-topic Someone on this list wrote: Mu. > > https://en.wikipedia.org/wiki/Mu_(negative)#%22Unasking%22_the_question This is a koan, a device in Zen Buddhism used by a teacher to help the student liberate themselves from being imprisoned by rational discursive thought. Here is another koan. A Zen student told Ummon: "Brilliancy of Buddha illuminates the whole universe." Before he finished the phrase Ummon asked: "You are reciting another's poem, are you not?" "Yes," answered the student. "You are sidetracked," said Ummon. Here's my commentary. The teacher neither agrees or disagrees with the student's statement. This is "Mu" if you wish. Instead the teacher directly observes the student's mental processes, confirms the observation, and provides helpful advice. The teacher is not trapped by rational discursive thought. Here the teacher uses only 12 words. Sometimes a teacher uses a single word, or no words at all. You'll find this koan and further commentary at https://www.sacred-texts.com/bud/glg/glg39.htm. -- Jonathan ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4BB6H5IDADCFOBWQTPDG7442P2DXWYRI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On Tue, Jun 22, 2021 at 9:23 PM Steven D'Aprano wrote: > > On Tue, Jun 22, 2021 at 05:50:48PM +1000, Chris Angelico wrote: > > > Hmm, that's not what I'd usually understand "encapsulation" to mean. > > That's what would normally be called "namespacing". > > Pfft, who you going to believe, me or some random folx on the internet > editing Wikipedia? *wink* > > Okay, using the Wikipedia/OOP definition still applies. Presumably most > extension methods are going to be regular instance methods or class > methods, rather than staticmethod. So they will take a `self` (or `cls`) > parameter, and presumably most such extension methods will actually act > on that self parameter in some way. > > There is your "bundling of data with the methods that operate on that > data", as required :-) Okay, that's fair. Granted. It's not ALL of encapsulation, but it is, to an extent encapsulation. (It is also namespacing, and your justification of it was actually a justification of namespacing; but this is Python, and I think we all agree that namespacing is good!) > Regarding your example, you're only confused because you haven't take on > board the fact that extension methods aren't interpreter-global, > just module-global. Because it's new and unfamiliar. But we can do > exactly the same thing, right now, with functions instead of methods, > and you will find it trivially easy to diagnose the fault: > > > # file1.py > from file2 import func > # instead of "applying an extension method" from elsewhere, > # import a function from elsewhere > from extra_functions import g > def spamify(obj): > print(g(obj)) # works fine > print(func(obj)) # fails > > > # file2.py > def func(obj): > return g(obj) # NameError > > > This example looks easy and not the least bit scary to you because > you've been using Python for a while and it has become second nature to > you. But you might remember back when you were a n00b, it probably > confused you: why doesn't `g(obj)` work when you imported it? How > weird and confusing! What do you mean, if I want to use g, I have to > import it in each and every module where I want to use it? That's just > dumb. Importing g once should make it available EVERYWHERE, right? > > Been there, done that. Fair point. However, I've worked with a good number of languages that have some notion of object methods, and generally, an object has or doesn't have a method based on what the object *is*, not on who's asking. It's going to make for some extremely confusing results. Is getattr() going to act as part of the builtins module or the module that's calling it? What about hasattr? What about an ABC's instance check, or anything else? How do other languages deal with this? How do they have a getattr-like function? Does it have to be a compiler construct? > Chris, here you are defending monkey-patching, not just as a necessary > evil under some circumstances, but as a "cleaner" option, and then in > your very next sentence: > > > And the Ruby community is starting to see the risks of > > monkey-patching. > Yes. That is exactly right. I am claiming that monkey-patching is, in many MANY cases, a cleaner option than extension methods. And then I am saying that monkey-patching is usually a bad thing. This is not incompatible, and it forms a strong view of my opinion of extension methods. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RCIR6GEJ6YA2Z2ZAFJJ7V2CJOGT6BRKP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On Tue, Jun 22, 2021 at 09:12:53PM +1000, Chris Angelico wrote: > > The must be no semantic difference between: > > > > obj.method(arg) > > > > and > > > > getattr(obj, 'method')(arg) > > > > regardless of whether `method` is a regular method or an extension > > method. > > And this is a problem. If its a problem for getattr, it is a problem for dot syntax, because they are essentially the same thing. > How is getattr defined? The same as it is defined now, except with some minor tweaks to support extension methods. Do you remember when we introduced `__slots__` (version 2.2 or 2.3, I think?), and added a whole new mechanism to look up ordinary attributes and slot attributes? No, neither do I, because we didn't. We have a single mechanism for looking up attributes, including methods, which works with instances and classes, descriptors and non-descripters, C-level slots and Python `__slots__` and `__dicts__` and `__getattr__` and `__getattribute__`, and I am absolutely positively sure that if Python ever adds a new implementation for attribute lookup, it will still be handled by the same getattr mechanism, which is built into the interpreter. > Is it counted as being in the current module? `getattr`? No, that's a builtin. You can shadow it or delete it if you want, it's just a public API to the underlying functionality built into the interpreter. Dot syntax won't be affected. > If it is, then has getattr magically become > part of the module it's called from? Or do ALL lookups depend on where > the function was called, rather than where it's defined? Can you be a bit more precise? I'm not suggesting that we introduce dynamic scoping instead of lexical scoping, if that's what you mean. Attribute lookups already depend on the state of the object at the time the lookup is made. This is just more of the same. class K: pass K.attr # this is an AttributeError K.attr = 'extension' K.attr # this is fine > If 'method' is an extension method, where exactly is it visible? I believe that TypeScript uses "import" for this, so it would be visible from anywhere that imports it: https://putridparrot.com/blog/extension-methods-in-typescript/ -- Steve ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5CEA6XWEX7VP554RIC4TDTJMURRVAOJL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
Oh this is a long one. Hypothetically, let's say you have a proxy object: class Foo: def __getattribute__(self, thing): return getattr(super().__getattribute__(self, "proxied"), thing) Should this really include extension methods into it by default? This is clearly wrong. The local override for the LOAD_ATTR opcode should NOT apply to proxy methods except where explicitly requested. Also sometimes you are supposed to call the dunder directly, like in the above example. It's not *bad* to do it if you know what you're doing. The point is that the caller using your proxy object should opt-in to the extension methods, rather than break with no way to opt-out of them. Your extension methods shouldn't propagate to proxy objects. To go even further, should all your class definitions that happen to extend a class with in-scope extension methods automatically gain those extension methods? Because with actual extension methods, that doesn't happen. You can have class MyList(list): pass and other callers would not get MyList.flatten even with you being able to use MyList.flatten locally. Extension methods are more like Rust traits than inheritance-based OOP. Also note that they use instance method syntax, but no other. That is they apply to LOAD_ATTR opcodes but should not apply to getattr! (Indeed, reflection in C#/Kotlin doesn't see the extension methods!) On 2021-06-22 6:57 a.m., Steven D'Aprano wrote: > I'm sorry Soni, I don't understand what you are arguing here. See below. > > > On Mon, Jun 21, 2021 at 10:09:17PM -0300, Soni L. wrote: > > > > > > On 2021-06-21 9:39 p.m., Steven D'Aprano wrote: > > > > > > > > > > Fourth step is that you go ahead and use lists as normal. Whether you > > > use getattr or dot syntax, any extension methods defined in spam.py will > > > show up, as if they were actual list methods. > > > > > > hasattr([], 'head') # returns True > > > list.tail # returns the spam.tail function object (unbound method) > > > > > > They're not monkey-patched: other modules don't see that. > > > > > > > > > > Python is a dynamic language. Maybe you're using hasattr/getattr to > > forward something from A to B. If "other modules don't see that" then > > this must work as if there were no extension methods in place. > > What's "forward something from A to B" mean? What are A and B? > > If "this" (method lookups) "must work as if there were no extension > methods in place" then extension methods are a no-op and are pointless. > You write an extension method, register it as applying to a type, the > caller opts-in to use it, and then... nothing happens, because it "must > work as if there were no extension methods in place". > > Surely that isn't what you actually want to happen. But if not, I have > no idea what you mean. > > The whole point of extension methods is that once the caller opts in to > use them, method look ups (and that includes hasattr and getattr) must > work as if the extension methods **are in place**. > > The must be no semantic difference between: > > obj.method(arg) > > and > > getattr(obj, 'method')(arg) > > regardless of whether `method` is a regular method or an extension > method. > > > > So you > > actually wouldn't want the local load_attr override to apply to those. > > If you did... well, just call the override directly. > > I have no idea what that means. What is "the local load_attr override"? > > > > If the override was > > called __opcode_load_attr_impl__ you'd just call > > __opcode_load_attr_impl__ directly instead of going through getattr. > > As a general rule, you should not be calling dunders directly. > > You seem to have missed the point that extension methods are intended as > a mechanism to **extend a type** by giving it new methods on an opt-in > basis. I want to call them "virtual methods" except that would add > confusion regarding virtual subclasses and ABCs etc. > > Maybe you need to read the Kotlin docs: > > https://kotlinlang.org/docs/extensions.html > > and the C# docs: > > https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/extension-methods > > Wikipedia also has a broad overview from a language-agnostic > perspective: > > https://en.wikipedia.org/wiki/Extension_method > > Here's an example in TypeScript and Javascript: > > https://putridparrot.com/blog/extension-methods-in-typescript/ > > > > In particular note these comments: > > # Kotlin > "Such functions are available for calling in the usual way as if they > were methods of the original class." > > # C# > "Extension methods are only in scope when you explicitly import the > namespace into your source code with a using directive." > > Both C# and Kotlin are statically typed languages, and Python is not, > but we ought to aim to minimise the differences in semantics. Aside from > extension methods being resolved at runtime instead of at compile time, > the behaviour ought to be as close as possible. > > Just as
[Python-ideas] Re: Extension methods in Python
On Tue, Jun 22, 2021 at 05:50:48PM +1000, Chris Angelico wrote: > Hmm, that's not what I'd usually understand "encapsulation" to mean. > That's what would normally be called "namespacing". Pfft, who you going to believe, me or some random folx on the internet editing Wikipedia? *wink* Okay, using the Wikipedia/OOP definition still applies. Presumably most extension methods are going to be regular instance methods or class methods, rather than staticmethod. So they will take a `self` (or `cls`) parameter, and presumably most such extension methods will actually act on that self parameter in some way. There is your "bundling of data with the methods that operate on that data", as required :-) The fact that the methods happen to be written in an separate file, and (in some sense) added to the class as extension methods, is neither here nor there. While we *could* write an extension method that totally ignored `self` and instead operated entirely on global variables, most people won't -- and besides, we can already do that with regular methods. class Weird: def method(self, arg): global data, more_data, unbundled_data, extra_data del self # don't need it, don't want it do_stuff_with(data, more_data, unbundled_data, extra_data) So the point is that extension methods are no less object-orientey than regular methods. They ought to behave just like regular methods with respect to encapsulation, namespacing, inheritance etc, modulo any minor and necessary differences. E.g. in C# extension methods can only extend a class, not override an existing method. [...] > I don't think it's safer necessarily. With this proposal, we have the > notion that obj.method() can mean two completely different things *at > the same time* and *on the same object* depending on how you refactor > the code. > > # file1.py > from file2 import func > # and apply some extension methods > def spamify(obj): > print(obj.method()) > print(func(obj)) > > # file2.py > def func(obj): > return obj.method() Yes, we can write non-obvious code in any language, using all sorts of "confusing" techniques, especially when you do stuff dynamically. class K: def __getattr__(self, attrname): if attrname == 'method': if __name__ == '__main__': raise AttributeError return something() Regarding your example, you're only confused because you haven't take on board the fact that extension methods aren't interpreter-global, just module-global. Because it's new and unfamiliar. But we can do exactly the same thing, right now, with functions instead of methods, and you will find it trivially easy to diagnose the fault: # file1.py from file2 import func # instead of "applying an extension method" from elsewhere, # import a function from elsewhere from extra_functions import g def spamify(obj): print(g(obj)) # works fine print(func(obj)) # fails # file2.py def func(obj): return g(obj) # NameError This example looks easy and not the least bit scary to you because you've been using Python for a while and it has become second nature to you. But you might remember back when you were a n00b, it probably confused you: why doesn't `g(obj)` work when you imported it? How weird and confusing! What do you mean, if I want to use g, I have to import it in each and every module where I want to use it? That's just dumb. Importing g once should make it available EVERYWHERE, right? Been there, done that. You learned about modules and namespaces, and why Python's design is *safer and better* than a single interpreter-global namespace, and now that doesn't confuse you one bit. And if you were using Kotlin, or C#, or Swift, or any one of a number of other languages with extension methods, you would likewise learn that extension methods work in a similar fashion. Why does obj.method raise AttributeError from file2? *Obviously* its because you neglected to "apply the extension method", duh. That's as obvious as neglecting to import something and getting a NameError. Maybe even more obvious, if your IDE or linter knows about extension methods. And its *safer and better* than monkey-patching. We have two people in this thread who know Kotlin and C#, at least one of them is a fan of the technique. Why don't we ask them how often this sort of error is a problem within the Kotlin and C# communities? > Is that really beneficial? All I'm seeing is myriad ways for things to > get confusing - just like in the very worst examples of > monkey-patching. > > And yes, I have some experience of monkey-patching in Python, > including a situation where I couldn't just "import A; import B", I > had to first import a helper for A, then import B, and finally import > A, because there were conflicting monkey-patches. But here's the > thing: extension
[Python-ideas] Re: Extension methods in Python
On Tue, Jun 22, 2021 at 8:01 PM Steven D'Aprano wrote: > The whole point of extension methods is that once the caller opts in to > use them, method look ups (and that includes hasattr and getattr) must > work as if the extension methods **are in place**. > > The must be no semantic difference between: > > obj.method(arg) > > and > > getattr(obj, 'method')(arg) > > regardless of whether `method` is a regular method or an extension > method. > And this is a problem. How is getattr defined? Is it counted as being in the current module? If it is, then has getattr magically become part of the module it's called from? Or do ALL lookups depend on where the function was called, rather than where it's defined? If 'method' is an extension method, where exactly is it visible? ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5YRFIE3O42WN3TCLLLR2NR5M6DZ2SUJU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
I'm sorry Soni, I don't understand what you are arguing here. See below. On Mon, Jun 21, 2021 at 10:09:17PM -0300, Soni L. wrote: > > > On 2021-06-21 9:39 p.m., Steven D'Aprano wrote: > > > > > > Fourth step is that you go ahead and use lists as normal. Whether you > > use getattr or dot syntax, any extension methods defined in spam.py will > > show up, as if they were actual list methods. > > > > hasattr([], 'head') # returns True > > list.tail # returns the spam.tail function object (unbound method) > > > > They're not monkey-patched: other modules don't see that. > > > > > > Python is a dynamic language. Maybe you're using hasattr/getattr to > forward something from A to B. If "other modules don't see that" then > this must work as if there were no extension methods in place. What's "forward something from A to B" mean? What are A and B? If "this" (method lookups) "must work as if there were no extension methods in place" then extension methods are a no-op and are pointless. You write an extension method, register it as applying to a type, the caller opts-in to use it, and then... nothing happens, because it "must work as if there were no extension methods in place". Surely that isn't what you actually want to happen. But if not, I have no idea what you mean. The whole point of extension methods is that once the caller opts in to use them, method look ups (and that includes hasattr and getattr) must work as if the extension methods **are in place**. The must be no semantic difference between: obj.method(arg) and getattr(obj, 'method')(arg) regardless of whether `method` is a regular method or an extension method. > So you > actually wouldn't want the local load_attr override to apply to those. > If you did... well, just call the override directly. I have no idea what that means. What is "the local load_attr override"? > If the override was > called __opcode_load_attr_impl__ you'd just call > __opcode_load_attr_impl__ directly instead of going through getattr. As a general rule, you should not be calling dunders directly. You seem to have missed the point that extension methods are intended as a mechanism to **extend a type** by giving it new methods on an opt-in basis. I want to call them "virtual methods" except that would add confusion regarding virtual subclasses and ABCs etc. Maybe you need to read the Kotlin docs: https://kotlinlang.org/docs/extensions.html and the C# docs: https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/extension-methods Wikipedia also has a broad overview from a language-agnostic perspective: https://en.wikipedia.org/wiki/Extension_method Here's an example in TypeScript and Javascript: https://putridparrot.com/blog/extension-methods-in-typescript/ In particular note these comments: # Kotlin "Such functions are available for calling in the usual way as if they were methods of the original class." # C# "Extension methods are only in scope when you explicitly import the namespace into your source code with a using directive." Both C# and Kotlin are statically typed languages, and Python is not, but we ought to aim to minimise the differences in semantics. Aside from extension methods being resolved at runtime instead of at compile time, the behaviour ought to be as close as possible. Just as single dispatch in Python is resolved dynamically, but aims to behave as close as possible to single dispatch in statically typed languages. Another important quote: "Because extension methods are called by using instance method syntax, no special knowledge is required to use them from client code. To enable extension methods for a particular type, just add a `using` directive for the namespace in which the methods are defined." "No special knowledge is required" implies that, aside from the opt-in step itself, extension methods must behave precisely the same as regular methods. That means they will be accessible as bound methods on the instance: obj.method and unbound methods (functions) on the type: type(obj).method and using dynamic lookup: getattr(obj, 'method') and they will fully participate in inheritance heirarchies if you have opted in to use them. > There needs to be an escape hatch for this. The escape hatch is to *not* opt-in to the extension method. If the caller doesn't opt-in, they don't get the extension methods. That is the critical difference between extension methods and monkey- patching the type. Monkey-patching effects everyone. Extension methods have to be opt-in. > Or you *could* have getattr be special (called by load_attr) and > overridable, and builtins.getattr be the escape hatch, but nobody would > like that. Huh? Unless you have shadowed getattr with a module-level function, getattr *is* builtins.getattr. -- Steve ___ Python-ideas mailing list --
[Python-ideas] Re: Deprecate sum of lists
As someone who originally worked on implementing PEP 448, I wanted to chime
in that while I agreed with the decision to be conservative when adding new
features, I originally hoped that [*x for x in xs] would eventually make
its way into Python. To me, it seemed intuitive and I hoped people would
eventually come around to seeing things that way, and to Steven's similar
suggestion for dicts {**x for x in xs}.
However, I had never seen Serhiy and Guido's point about the similarity to
[a, b for a, b in abs]. I agree that the comparison makes sense, and that
that notation is really confusing. Maybe this should be added to the PEP
(https://www.python.org/dev/peps/pep-0448/) as a reminder?
Given the counterpoint, I don't have a strong opinion either way. Maybe
it's better to remain conservative and see where the language is at in
another few years?
On another tangent, I'd like to see array[*x, y] working one day. If I
remember correctly, that was an oversight. It regularly comes up when
indexing numpy arrays, and needs an ugly workaround.
Best,
Neil
>___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/K7MYCJPHWMKW67XQAS244CJ4I22VHVSY/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Extension methods in Python
On Tue, Jun 22, 2021 at 10:40 AM Steven D'Aprano wrote: > > True, all true, but considering that this is *not* actually part of > > the class, some of that doesn't really apply. For instance, is it > > really encapsulation? What does that word even mean when you're > > injecting methods in from the outside? > > Sure it's encapsulation. We can already do this with non-builtin > classes: > > class SpammySpam: > def spam(self, arg): > ... > > from another_module import eggy_method > > def aardvarks(self, foo, bar): > ... > > SpammySpam.aardvarks = aardvarks > > > The fact that two of those methods have source code that wasn't indented > under the class statement is neither here nor there. Even the fact that > eggy_method was defined in another module is irrelevant. What matters is > that once I've put the class together, all three methods are fully > encapsulated into the SpammySpam class, and other classes can define > different methods with the same name. > > Encapsulation is less about where you write the source code, and more > about the fact that I can have > > SpammySpam().spam > > and > > Advertising().spam > > without the two spam methods stomping on each other. Hmm, that's not what I'd usually understand "encapsulation" to mean. That's what would normally be called "namespacing". "... encapsulation refers to the bundling of data with the methods that operate on that data, or the restricting of direct access to some of an object's components." https://en.wikipedia.org/wiki/Encapsulation_(computer_programming) > Extension methods let us extend classes without the downsides of > monkey-patching. Extension methods are completely opt-in while > monkey-patching is mandatory for everyone. If we could only have one, > extension methods would clearly be the safer choice. I don't think it's safer necessarily. With this proposal, we have the notion that obj.method() can mean two completely different things *at the same time* and *on the same object* depending on how you refactor the code. # file1.py from file2 import func # and apply some extension methods def spamify(obj): print(obj.method()) print(func(obj)) # file2.py def func(obj): return obj.method() Is that really beneficial? All I'm seeing is myriad ways for things to get confusing - just like in the very worst examples of monkey-patching. And yes, I have some experience of monkey-patching in Python, including a situation where I couldn't just "import A; import B", I had to first import a helper for A, then import B, and finally import A, because there were conflicting monkey-patches. But here's the thing: extension methods (by this pattern) would not have solved it, because the entire *point* of the monkey-patch was to fix an incompatibility. So it HAD to apply to a completely different module. That's why, despite its problems, I still think that monkey-patching is the cleaner option. It prevents objects from becoming context-dependent. > We don't make heavy use of monkey-patching, not because it isn't a > useful technique, but because: > > - unlike Ruby, we can't extend builtins without subclassing; > > - we're very aware that monkey-patching is a massively powerful > technique with huge foot-gun potential; > > - and most of all, the Python community is a hell of a lot more > conservative than Ruby. And the Ruby community is starting to see the risks of monkey-patching. (There's a quiz floating around the internet - "Ruby or Rails?" - that brings into sharp relief the incredibly far-reaching effects of using Rails. It includes quite a few methods on builtin objects.) So I am absolutely fine with being conservative. We have import hooks and MacroPy. Does anyone use them in production? I certainly don't - not because I can't, but because I won't without a VERY good reason. > Even basic techniques intentionally added to the language (like being > able to attach attributes onto function objects) are often looked at as > if they were the worst kind of obfuscated self-modifying code. Even when > those same techniques are used in the stdlib people are still reluctant > to use it. As a community, we're like cats: anything new and different > scares us, even if its actually been used for 30 years. > > We're a risk-adverse community. > I'm not sure why attaching attributes to functions is frowned upon; I'd personally make very good use of this for static variables, if only I could dependably refer to "this_function". But risk-averse is definitely preferable to the alternative. It means that Python is a language that can be learned as a whole, rather than being fragmented into "the NumPy flavour of Python" and "the Flask flavour of Python" and so on, with their own changes to the fabric of the language. So far, I'm not seeing anything in extension methods to make me want to change that stance. ChrisA ___ Python-ideas mailing list
