Observations on the three pillars of Python execution
In Python, three types of objects have special syntax and mechanics for their instantiation, during which a code object is generated: modules, classes, and functions. Each has its own role to play and the differences between them mostly reflect that. Here are some observations (based on default behavior): Object creation relative to code object creation: (M) before (C) after (F) after Object creation relative to code object execution: (M) before (C) after (F) before Object available during code object execution: (M) no (C) no (F) no Code object destiny: (M) executed once at definition time, then discarded (C) executed once at definition time, then discarded (F) not executed at definition time, stored on the function object for execution with each __call__() invocation. Execution locals is available during code object execution: (M) as globals() (C) as locals() (F) as locals() Object namespace is execution locals of code object: (M) yes (C) more or less (F) completely distinct Unique syntax: (M) 'import' statement (C) 'class' statement (F) 'def' statement Mechanism triggered by respective syntax: (M) import machinery (import.c/importlib) (C) internal, but partially exposed via metaclasses and the __build_class__ builtin (F) internal, nothing exposed Override capability: (M) complete capability through builtin __import__ and PEP 302 import hooks (C) partial capability, before code object execution through metaclass __prepare__() and after through __build_class__() and metaclass __call__(), __new__(), and __init__() (F) no capability Post instantiation modification capability: (M) yes (C) yes (F) mostly (some attributes are read-only) Mechanism to instantiate outside of respective unique syntax: (M) imp.new_module(), types.ModuleType(), type(module)() (C) type() (F) types.FunctionType(), type(f)() Type extensibility: (M) Not relative to 'import' syntax (by default) (C) Complete (F) No Name available during definition execution: (M) as __name__ (C) only through metaclass __prepare__() (F) through inspect.currentframe().f_code.co_name Name available on object as __name__: (M) yes (C) yes (F) yes Corrections, additions, and comment are welcome. -eric -- http://mail.python.org/mailman/listinfo/python-list
Re: Observations on the three pillars of Python execution
On 05/08/11 09:20, Eric Snow wrote: Object available during code object execution: (M) no (C) no (F) no (F) yes. cf. recursion. -- http://mail.python.org/mailman/listinfo/python-list
Recursive functions (was Re: Observations on the three pillars of Python execution)
On Fri, Aug 5, 2011 at 9:22 AM, Thomas Jollans t...@jollybox.de wrote: On 05/08/11 09:20, Eric Snow wrote: Object available during code object execution: (M) no (C) no (F) no (F) yes. cf. recursion. Is it? As I understand it, a Python function is not able to reference itself but must reference its own name. This means that assigning that function to something else stops it being recursive: # def foo(x): print(x) if x3: foo(x-1) foo(5) 5 4 3 bar=foo bar(5) 5 4 3 def foo(x): print(Foo,x) if x3: foo(x-1) foo(5) Foo 5 Foo 4 Foo 3 bar(5) 5 Foo 4 Foo 3 # bar() is not a recursive function, even though foo() was and is. I've just played around with a decorator, though; this appears to work: def recursive(f): l=list(f.__defaults__) l[-1]=f f.__defaults__=tuple(l) return f @recursive def quux(x,quux=None): print(Quux,x) if x3: quux(x-1) foo=quux def quux(x): print(Stopping.) quux(5) Stopping. foo(5) Quux 5 Quux 4 Quux 3 Effectively, the decorator binds the function's own self to its last default parameter, thus holding a cyclic reference (unlikely to be an issue - people won't be creating and destroying functions frequently, so GC issues shouldn't arise). The last parameter doesn't have to have the name of the function; this works with lambdas: recursive(lambda x,_=None: print(Lambda,x) or (x3 and _(x-1) or None))(5) Lambda 5 Lambda 4 Lambda 3 Yes, this is a pretty stupid example. But is this sort of decorator useful? It's not like people regularly want recursive lambdas. Chris Angelico -- http://mail.python.org/mailman/listinfo/python-list
Re: Observations on the three pillars of Python execution
Eric Snow wrote:
In Python, three types of objects have special syntax and mechanics
for their instantiation, during which a code object is generated:
modules, classes, and functions.
I believe you are labouring under a misapprehension. Modules and classes
don't generate code objects.
The simplest possible module just has a name:
import sys
module = type(sys)
module(name)
module 'name' (built-in)
Or if you prefer to do it the normal way by importing from source code,
the simplest source code of all is just an empty file.
A module is an object with a __dict__, a __name__, and a __docstring__ which
defaults to None. That's it. No code objects, unless you populate the
__dict__ with functions, but it is the *functions* that have the code
objects, not the module itself. The module is just a container.
Classes are similar. The simplest class:
class K:
pass
or if you prefer:
type('K', (), {})
class '__main__.K'
Again, no code objects unless you populate the dict with functions (which
get turned into methods). Again, the class object is just a container. A
more complex container than modules, with a richer set of inherited
behaviour, but still just a container.
The signature for creating a function is very different:
help(type(lambda: None)) =
class function(object)
| function(code, globals[, name[, argdefs[, closure]]])
|
| Create a function object from a code object and a dictionary.
| The optional name string overrides the name from the code object.
| The optional argdefs tuple specifies the default argument values.
| The optional closure tuple supplies the bindings for free variables.
The simplest function I can think of:
co = compile('pass', '', 'single')
f = type(lambda: None)(co, {})
Even simpler than
def f():
pass
Can you see why?
If you inspect a function object, you will see that it always has a code
object, even if the code object does nothing but return None:
import dis
dis.dis(f.func_code)
1 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
There may be some other obscure built-in type that includes code objects,
but I can't imagine what it would be. I feel confident in saying that
functions, and functions alone, contain code. Even methods are just
wrappers around functions. Even built-in functions like len don't contain
code! (Or at least, their code isn't accessible from Python.) Which makes
sense, if you think about it: their code is part of the Python virtual
machine, not the object.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Recursive functions (was Re: Observations on the three pillars of Python execution)
Chris Angelico wrote: On Fri, Aug 5, 2011 at 9:22 AM, Thomas Jollans t...@jollybox.de wrote: On 05/08/11 09:20, Eric Snow wrote: Object available during code object execution: (M) no (C) no (F) no (F) yes. cf. recursion. Is it? As I understand it, a Python function is not able to reference itself but must reference its own name. That is correct. Recursion in Python is implemented simply by name lookup. This means that assigning that function to something else stops it being recursive: An easier way to demonstrate the process: def f(x): ... print x ... if x 0: f(x-1) ... f(3) 3 2 1 0 g = f del f g(3) 3 Traceback (most recent call last): File stdin, line 1, in module File stdin, line 3, in f NameError: global name 'f' is not defined -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Observations on the three pillars of Python execution
On Fri, Aug 5, 2011 at 8:36 AM, Steven D'Aprano
steve+comp.lang.pyt...@pearwood.info wrote:
Eric Snow wrote:
In Python, three types of objects have special syntax and mechanics
for their instantiation, during which a code object is generated:
modules, classes, and functions.
I believe you are labouring under a misapprehension. Modules and classes
don't generate code objects.
Sorry for any confusion. I was talking specifically about the their
instantiation through their respective special syntax. During that
process a code object is generated for each. For classes and modules
the code object is executed and then discarded. I covered this
explicitly in one of the observations.
The simplest possible module just has a name:
import sys
module = type(sys)
module(name)
module 'name' (built-in)
Or if you prefer to do it the normal way by importing from source code,
the simplest source code of all is just an empty file.
A module is an object with a __dict__, a __name__, and a __docstring__ which
defaults to None. That's it. No code objects, unless you populate the
__dict__ with functions, but it is the *functions* that have the code
objects, not the module itself. The module is just a container.
Classes are similar. The simplest class:
class K:
pass
or if you prefer:
type('K', (), {})
class '__main__.K'
Again, no code objects unless you populate the dict with functions (which
get turned into methods). Again, the class object is just a container. A
more complex container than modules, with a richer set of inherited
behaviour, but still just a container.
The signature for creating a function is very different:
help(type(lambda: None)) =
class function(object)
| function(code, globals[, name[, argdefs[, closure]]])
|
| Create a function object from a code object and a dictionary.
| The optional name string overrides the name from the code object.
| The optional argdefs tuple specifies the default argument values.
| The optional closure tuple supplies the bindings for free variables.
The simplest function I can think of:
co = compile('pass', '', 'single')
f = type(lambda: None)(co, {})
Even simpler than
def f():
pass
Can you see why?
If you inspect a function object, you will see that it always has a code
object, even if the code object does nothing but return None:
import dis
dis.dis(f.func_code)
1 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
There may be some other obscure built-in type that includes code objects,
but I can't imagine what it would be. I feel confident in saying that
functions, and functions alone, contain code. Even methods are just
wrappers around functions. Even built-in functions like len don't contain
code! (Or at least, their code isn't accessible from Python.) Which makes
sense, if you think about it: their code is part of the Python virtual
machine, not the object.
Agreed that [non-builtin] functions are the only objects that have a
code object attribute. However, they are not the only objects for
which a code object is generated and executed.
I've always found your frequent responses very insightful and
thoughtful, Steven, and I appreciate that. In this case I can only
imagine that my opening statement was unclear enough that you did not
continue to read the rest, where the context of my point was more
clear (I hope :). Regardless, thanks for taking a look.
-eric
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: Observations on the three pillars of Python execution
Eric Snow wrote: On Fri, Aug 5, 2011 at 8:36 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Eric Snow wrote: In Python, three types of objects have special syntax and mechanics for their instantiation, during which a code object is generated: modules, classes, and functions. I believe you are labouring under a misapprehension. Modules and classes don't generate code objects. Sorry for any confusion. I was talking specifically about the their instantiation through their respective special syntax. During that process a code object is generated for each. Do you believe that this process of generating a code object and throwing it away is a part of the Python language specification, which any compiler must do in order to call itself Python, or a mere implementation detail? Is this documented somewhere? If it is not documented, what makes you think that it happens at all? You are writing as if it were self-evidently true, but I don't believe it is self-evident at all. I think you need to demonstrate the truth of two of those three pillars, not just take them for granted. For classes and modules the code object is executed and then discarded. I covered this explicitly in one of the observations. I think your definition of explicitly and mine differ here. Agreed that [non-builtin] functions are the only objects that have a code object attribute. However, they are not the only objects for which a code object is generated and executed. Are you talking about the fact that importing a module, class statements and function statements all involve executing a block of code? How does that differ from executing any other fragment of code? Modules are special, of course, because they can get compiled to byte-code, but I trust you're not talking about .pyc files. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Observations on the three pillars of Python execution
Steven D'Aprano wrote: There may be some other obscure built-in type that includes code objects, but I can't imagine what it would be. I feel confident in saying that functions, and functions alone, contain code. Even methods are just wrappers around functions. Even built-in functions like len don't contain code! (Or at least, their code isn't accessible from Python.) Which makes sense, if you think about it: their code is part of the Python virtual machine, not the object. Interesting question. Iterators seem to have code objects: Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) [GCC 4.4.3] on linux2 Type help, copyright, credits or license for more information. a = [1, 2, 3] b = (x for x in a) dir(b) ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__iter__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'close', 'gi_code', 'gi_frame', 'gi_running', 'next', 'send', 'throw'] for name in dir(b): ... print name, type (getattr (b, name)) ... __class__ type 'type' __delattr__ type 'method-wrapper' __doc__ type 'NoneType' __format__ type 'builtin_function_or_method' __getattribute__ type 'method-wrapper' __hash__ type 'method-wrapper' __init__ type 'method-wrapper' __iter__ type 'method-wrapper' __name__ type 'str' __new__ type 'builtin_function_or_method' __reduce__ type 'builtin_function_or_method' __reduce_ex__ type 'builtin_function_or_method' __repr__ type 'method-wrapper' __setattr__ type 'method-wrapper' __sizeof__ type 'builtin_function_or_method' __str__ type 'method-wrapper' __subclasshook__ type 'builtin_function_or_method' close type 'builtin_function_or_method' gi_code type 'code' gi_frame type 'frame' gi_running type 'int' next type 'method-wrapper' send type 'builtin_function_or_method' throw type 'builtin_function_or_method' in the form of the gi_code attribute. No idea what it's for, although no reason to believe it shouldn't be there. (Very interesting demo you gave of primitive object creation. I' awed.) Mel. -- http://mail.python.org/mailman/listinfo/python-list
Re: Observations on the three pillars of Python execution
On 8/5/2011 4:22 AM, Thomas Jollans wrote: On 05/08/11 09:20, Eric Snow wrote: Object available during code object execution: (M) no (C) no (F) no (F) yes. cf. recursion. Recursion only happens through runtime name resolution, not through direct access to the function or code object from within the code. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Observations on the three pillars of Python execution
On Fri, Aug 5, 2011 at 11:29 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Eric Snow wrote: On Fri, Aug 5, 2011 at 8:36 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Eric Snow wrote: In Python, three types of objects have special syntax and mechanics for their instantiation, during which a code object is generated: modules, classes, and functions. I believe you are labouring under a misapprehension. Modules and classes don't generate code objects. Sorry for any confusion. I was talking specifically about the their instantiation through their respective special syntax. During that process a code object is generated for each. Do you believe that this process of generating a code object and throwing it away is a part of the Python language specification, which any compiler must do in order to call itself Python, or a mere implementation detail? That's a great point which I hadn't considered. Honestly, I only used my experience with CPython in making these observations. After reviewing the language reference I see that I missed out on a bunch of nomenclature that would have made things more clear, and I got a few points wrong, which you pointed out. :) Regarding code objects and classes, your are right. The language reference indicates the following: The class’s suite is then executed in a new execution frame...When the class’s suite finishes execution, its execution frame is discarded but its local namespace is saved. [1] So the use of code objects for execution is an implementation detail. Instead of code object I should have referred to the code executed in the execution frame or just to the frame. Incidently, I had not realized that the builtin __build_class__() is also an implementation detail[3]. For modules, the language reference doesn't say anything about how execution is accomplished, which only matters when execution is involved in the creation of the module object. It does refer to importlib as a reference implementation[4]. The order-of-operations observations I made are based on that reference implementation. Is this documented somewhere? If it is not documented, what makes you think that it happens at all? You are writing as if it were self-evidently true, but I don't believe it is self-evident at all. I think you need to demonstrate the truth of two of those three pillars, not just take them for granted. For classes and modules the code object is executed and then discarded. I covered this explicitly in one of the observations. I think your definition of explicitly and mine differ here. Agreed that [non-builtin] functions are the only objects that have a code object attribute. However, they are not the only objects for which a code object is generated and executed. Are you talking about the fact that importing a module, class statements and function statements all involve executing a block of code? How does that differ from executing any other fragment of code? The difference is that modules, classes, and functions (really the function body) are code blocks tied to syntax that results in module, type, and function objects. There are other code blocks but none of them have a unique syntax, much less one that results in an object of the corresponding type[5]. This is relevant for trying to find the object that corresponds to an execution frame, which is what led me to my original post and drove the direction of the observations I made. Anyway, I appreciate the feedback! I'm going to have to revisit my observations with the language definition in hand. You've been really insightful, as usual. -eric [1] http://docs.python.org/dev/reference/compound_stmts.html#class-definitions [2] http://docs.python.org/dev/reference/datamodel.html#metaclasses [3] http://mail.python.org/pipermail/python-3000/2007-March/006338.html [4] http://docs.python.org/dev/reference/simple_stmts.html#the-import-statement [5] http://docs.python.org/dev/reference/executionmodel.html#naming Modules are special, of course, because they can get compiled to byte-code, but I trust you're not talking about .pyc files. -- Steven -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Observations on the three pillars of Python execution
Mel wrote: Steven D'Aprano wrote: There may be some other obscure built-in type that includes code objects, but I can't imagine what it would be. I feel confident in saying that functions, and functions alone, contain code. Even methods are just wrappers around functions. Even built-in functions like len don't contain code! (Or at least, their code isn't accessible from Python.) Which makes sense, if you think about it: their code is part of the Python virtual machine, not the object. Interesting question. Iterators seem to have code objects: [...] Generators. But nice catch, thank you! Iterators are *any* object which obeys the iterator protocol, that is, have a next() method and an __iter__() method which behave in the expected way. Iterators are a duck-type. Generators, whether created from a generator expression or a generator function, are an actual type. type(x for x in (1,2)) class 'generator' -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Observations on the three pillars of Python execution
Eric Snow wrote:
On Fri, Aug 5, 2011 at 11:29 AM, Steven D'Aprano
steve+comp.lang.pyt...@pearwood.info wrote:
[...]
Do you believe that this process of generating a code object and throwing
it away is a part of the Python language specification, which any
compiler must do in order to call itself Python, or a mere
implementation detail?
That's a great point which I hadn't considered. Honestly, I only used
my experience with CPython in making these observations. After
reviewing the language reference I see that I missed out on a bunch of
nomenclature that would have made things more clear, and I got a few
points wrong, which you pointed out. :)
Regarding code objects and classes, your are right. The language
reference indicates the following:
The class’s suite is then executed in a new execution frame...When
the class’s suite finishes execution, its execution frame is discarded
but its local namespace is saved. [1]
It turns out that in CPython 2.5 at least, I'm strictly wrong and you got it
right, at least for classes:
code = compile(class K: pass, '', 'exec')
dis.dis(code)
1 0 LOAD_CONST 0 ('K')
3 LOAD_CONST 3 (())
6 LOAD_CONST 1 (code object K at 0xb7e8ad10,
file , line 1)
9 MAKE_FUNCTION0
12 CALL_FUNCTION0
15 BUILD_CLASS
16 STORE_NAME 0 (K)
19 LOAD_CONST 2 (None)
22 RETURN_VALUE
So a code object is compiled, turned into a function, executed, the results
turned into a class, and the code object and function thrown away.
Is this an implementation detail? I would say so. The semantics of Python
the language are different from the details of it's virtual machine. Surely
we would be allowed to call something Python if it executed the body of the
class statement *without* creating a code object first? The important part
is *execute the body of the class statement*, not building the code object.
The later is merely a means to an end.
So the use of code objects for execution is an implementation detail.
Instead of code object I should have referred to the code executed
in the execution frame or just to the frame.
Unless you really intend to talk about implementation details, I think you
should keep the discussion as high-level as possible. I'd talk about
executing blocks of code, and not even mention execution frames unless you
need to understand the role of frames during execution.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
