Re: compiler package vs parser
Kay Schluehr wrote: I realize that I probably ought to be trying this out with the newer ast stuff, but currently I am supporting code back to 2.3 and there's not much hope of doing it right there without using the compiler package. You might consider using the *builtin* parser module and forget about the compiler package if it is broken ( I take your word that it is ) or modern ast representations which aren't really necessary for Python anyway. import parser tree = parser.suite(def foo():\n print 42}\n) code = tree.compile() exec code foo() 42 This is also not 100% reliable ( at least not for all statements in all Python versions ) but it uses the internal parser/compiler and not a standard library compiler package that might not be that well maintained. ... thinking about it I just made the wrong decision back in 2004; we observed a semantic change caused by the new scoping rules and tried to fix using the wrong model; back then we were probably supporting 2.0 as well so the parser module probably wasn't available everywhere anyway; even today the ast stuff isn't available in 2.4. I prefer the ast approach as preppy is effectively indentation free which makes the tree harder to synthesize for the parser tree. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
Kay Schluehr wrote: On 16 Apr., 11:41, Robin Becker ro...@reportlab.com wrote: Is the compiler package actually supposed to be equivalent to the parser module? No. The parser module creates a concrete parse tree ( CST ) whereas the compiler package transforms this CST into an AST for subsequent computations. In more recent versions those CST - AST transformations are performed by the runtime and the Python compiler uses those internally produced ASTs. The Python 2.6 API to ASTs is the ast module. .. OK having digested most of what Kay said (my simple engineer's brain being rather stretched at the best of times) I would like to rephrase my question. Is it intended that the compile package be able to create the same code as the normal parser chain. I think I understand now that the compiler used internally is now exposed in various ways as built in modules. The example I gave earlier seems to indicate that the compiler package may not be creating the same final bytecode at least using the compilation approach that I used (cbased on the compiler package itself). If I have messed that up then there should be some easy fix, otherwise if pycodegen is somehow not getting the semantics of the the variables i,j correct is there some way I can fix that. I realize that I probably ought to be trying this out with the newer ast stuff, but currently I am supporting code back to 2.3 and there's not much hope of doing it right there without using the compiler package. My analysis of the problem is that in start p.py def func(D): for k in D: exec '%s=D[%r]' % (k,k) print i, j, k print locals() print i, j, k if __name__=='__main__': func(dict(i=1,j=33)) end p.py the compiler package ends up treating i j as global, whereas the modern analysis doesn't (and doesn't say they're definitely local either). Looking at the code in Python/compile.c the compile_nameop code seems to check for scopes FREE, CELL, LOCAL, GLOBAL_IMPLICIT GLOBAL_EXPLICIT whereas pycodegen.CodeGenerator._nameOp seems not to know about GLOBAL_IMPLICIT/EXPLICIT but has only a single GLOBAL scope. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
In article mailman.4042.1239964684.11746.python-l...@python.org, Robin Becker ro...@reportlab.com wrote: My analysis of the problem is that in start p.py def func(D): for k in D: exec '%s=D[%r]' % (k,k) print i, j, k print locals() print i, j, k if __name__=='__main__': func(dict(i=1,j=33)) end p.py the compiler package ends up treating i j as global, whereas the modern analysis doesn't (and doesn't say they're definitely local either). Looking at the code in Python/compile.c the compile_nameop code seems to check for scopes FREE, CELL, LOCAL, GLOBAL_IMPLICIT GLOBAL_EXPLICIT whereas pycodegen.CodeGenerator._nameOp seems not to know about GLOBAL_IMPLICIT/EXPLICIT but has only a single GLOBAL scope. At this point, given the lack of response from people who actually know the compiler internals, I think it's fair of you to try going to python-dev -- normally questions like this are off-topic, and if someone complains, I'll take the blame. ;-) -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ If you think it's expensive to hire a professional to do the job, wait until you hire an amateur. --Red Adair -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
Robin Becker wrote: If I have messed that up then there should be some easy fix, otherwise if pycodegen is somehow not getting the semantics of the the variables i,j correct is there some way I can fix that def func(D): for k in D: exec '%s=D[%r]' % (k,k) print i, j, k print locals() print i, j, k if __name__=='__main__': func(dict(i=1,j=33)) end p.py the compiler package ends up treating i j as global, whereas the modern analysis doesn't (and doesn't say they're definitely local either). If they are not definitely local, they are non-local. Locals are determined at function definition time, not function execution time. So, your expectations about what the exec statement can do above are mistaken. You may try to work your way around it, but, IMHO, you will not succeed. If the code above were to work as you wish, every access to every non-local in code that contains an exec would have to check a new locals dictionary just in case the exec added a local. Think about what this code would have to do: i = j = 42 def func(D): print i, j, k for k in D: exec '%s=D[%r]' % (k,k) print i, j, k --Scott David Daniels scott.dani...@acm.org -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
En Fri, 17 Apr 2009 10:55:46 -0300, Scott David Daniels scott.dani...@acm.org escribió: Robin Becker wrote: def func(D): for k in D: exec '%s=D[%r]' % (k,k) print i, j, k print locals() print i, j, k if __name__=='__main__': func(dict(i=1,j=33)) end p.py the compiler package ends up treating i j as global, whereas the modern analysis doesn't (and doesn't say they're definitely local either). If they are not definitely local, they are non-local. Locals are determined at function definition time, not function execution time. So, your expectations about what the exec statement can do above are mistaken. You may try to work your way around it, but, IMHO, you will not succeed. If the code above were to work as you wish, every access to every non-local in code that contains an exec would have to check a new locals dictionary just in case the exec added a local. And that's what happens. In absence of an exec statement, locals are optimized: the compiler knows exactly how many of them exist, and its names, just by static code analysis. But when the function contains an exec statement (or an import * statement) this is not possible anymore, and the compiler has to switch to another strategy and generate code using a less-optimized approach. Unknown variables are accessed using LOAD_NAME/STORE_NAME (require a name lookup) instead of the normal LOAD_FAST/STORE_FAST for local variables and LOAD_GLOBAL/STORE_GLOBAL for global ones. Think about what this code would have to do: i = j = 42 def func(D): print i, j, k for k in D: exec '%s=D[%r]' % (k,k) print i, j, k I don't completely understand what you wanted to show, but try this: py i = j = 42 py def f(): ... print i, j ... exec k=1 ... print i, j, k ... exec i=5 ... print i, j, k ... py f() 42 42 42 42 1 5 42 1 py i 42 py j 42 py k Traceback (most recent call last): File stdin, line 1, in module NameError: name 'k' is not defined Note that i starts as a global name and after the exec i=5 it becomes a local variable; and k is always a local variable even if there is no explicit assignment to it (except in the exec). -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
I realize that I probably ought to be trying this out with the newer ast stuff, but currently I am supporting code back to 2.3 and there's not much hope of doing it right there without using the compiler package. You might consider using the *builtin* parser module and forget about the compiler package if it is broken ( I take your word that it is ) or modern ast representations which aren't really necessary for Python anyway. import parser tree = parser.suite(def foo():\n print 42}\n) code = tree.compile() exec code foo() 42 This is also not 100% reliable ( at least not for all statements in all Python versions ) but it uses the internal parser/compiler and not a standard library compiler package that might not be that well maintained. -- http://mail.python.org/mailman/listinfo/python-list
compiler package vs parser
Is the compiler package actually supposed to be equivalent to the parser module?
I ask because the following code
start p.py
def func(D):
for k in D:
exec '%s=D[%r]' % (k,k)
print i, j, k
print locals()
print i, j, k
if __name__=='__main__':
func(dict(i=1,j=33))
end p.py
when run through the compiler package to produce a module has different code to
that produced by the standard compilation (detected by dis.dis). In particular
the variables i and j in func above are treated differently; in the standard
compiler case LOAD_NAME is used and in the code from the package LOAD_GLOBAL is
used.
The code used to create the synthetic module is
start tp.py
from compiler import parse, pycodegen, misc, syntax
import time, struct, marshal
txt=open('p.py','r').read()
tree=parse(txt)
print 'tree\n',tree
def _get_tree(tree,filename):
misc.set_filename(filename, tree)
syntax.check(tree)
return tree
def getPycHeader():
mtime = time.time()
mtime = struct.pack('i', mtime)
return pycodegen.Module.MAGIC + mtime
def dump(fn,code):
f=open(fn,'wb')
f.write(getPycHeader())
marshal.dump(code,f)
f.close()
gen = pycodegen.ModuleCodeGenerator(_get_tree(tree,'synp.py'))
code = gen.getCode()
dump('synp.pyc',code)
end tp.py
The module synp.pyc fails with a traceback (as expected because there are no
global i,j), but p.py runs OK.
I assume that my attempt to compile the tree is broken (is missing some special
traverse etc) otherwise the code would end up the same (except for line
numbering which I have ignored).
--
Robin Becker
--
http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
In article mailman.3979.1239874929.11746.python-l...@python.org, Robin Becker ro...@reportlab.com wrote: Is the compiler package actually supposed to be equivalent to the parser module? Before I poke my nose into this, what versions of Python have you tried? -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ If you think it's expensive to hire a professional to do the job, wait until you hire an amateur. --Red Adair -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
Aahz wrote: In article mailman.3979.1239874929.11746.python-l...@python.org, Robin Becker ro...@reportlab.com wrote: Is the compiler package actually supposed to be equivalent to the parser module? Before I poke my nose into this, what versions of Python have you tried? I'm using 2.6. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
Robin Becker wrote: Aahz wrote: In article mailman.3979.1239874929.11746.python-l...@python.org, Robin Becker ro...@reportlab.com wrote: Is the compiler package actually supposed to be equivalent to the parser module? Before I poke my nose into this, what versions of Python have you tried? I'm using 2.6. I just checked and it's the same in 2.5. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
On 16 Apr., 11:41, Robin Becker ro...@reportlab.com wrote: Is the compiler package actually supposed to be equivalent to the parser module? No. The parser module creates a concrete parse tree ( CST ) whereas the compiler package transforms this CST into an AST for subsequent computations. In more recent versions those CST - AST transformations are performed by the runtime and the Python compiler uses those internally produced ASTs. The Python 2.6 API to ASTs is the ast module. -- http://mail.python.org/mailman/listinfo/python-list
Re: compiler package vs parser
Kay Schluehr wrote: On 16 Apr., 11:41, Robin Becker ro...@reportlab.com wrote: Is the compiler package actually supposed to be equivalent to the parser module? No. The parser module creates a concrete parse tree ( CST ) whereas the compiler package transforms this CST into an AST for subsequent computations. In more recent versions those CST - AST transformations are performed by the runtime and the Python compiler uses those internally produced ASTs. The Python 2.6 API to ASTs is the ast module. OK so I've been using the old compiler package to synthesize my trees, but presumably there are equivalents in the ast module so I could re-code my stuff to use the new module fairly easily. Presumably compiling the ast trees should produce identical code. On the other hand it seems as though there are at least three generations of AST lying about with compiler, ast (and _ast in 2.5) so multi version coding will be hard. -- Robin Becker -- http://mail.python.org/mailman/listinfo/python-list
