The attached patch is an attempt at extending the C code generator so
that it annotates the generated .c code with file/line information for
the corresponding Python sources.
Method:
I followed the instructions in pypy/doc/getting-started-dev.txt:
cd pypy
python bin/translatorshell.py
>>> t = Translation(snippet.is_perfect_number)
>>> t.annotate([int])
>>> f = t.compile_c()
then examine the generated .c code.
With this patch, the top of a generated .c function contains an inline
copy of the python source, like this:
bool_t pypy_g_ll_issubclass__object_vtablePtr_object_vtablePtr(struct
pypy_object_vtable0 *l_subcls_0, struct pypy_object_vtable0 *l_cls_0) {
bool_t l_v11; long l_v12; long l_v13; long l_v14;
/* Python source
'/home/david/coding/pypy-svn-anon/trunk-clean/pypy/rpython/lltypesystem/rclass.py'
* 665 : def ll_issubclass(subcls, cls):
* 666 : return llop.int_between(Bool, cls.subclassrange_min,
* 667 : subcls.subclassrange_min,
* 668 : cls.subclassrange_max)
*/
goto block0;
...and for every operation that has an offset, it calculates the
linenumber (using the "dis" module on the code object). For every
operation that changes the line number, it emits a comment containing
that Python source, so that you get code like this:
block0:
/* 666 : return llop.int_between(Bool, cls.subclassrange_min, */
l_v12 = RPyField(l_cls_0, ov_subclassrange_min);
/* 667 : subcls.subclassrange_min, */
l_v13 = RPyField(l_subcls_0, ov_subclassrange_min);
/* 668 : cls.subclassrange_max) */
l_v14 = RPyField(l_cls_0, ov_subclassrange_max);
OP_INT_BETWEEN(l_v12, l_v13, l_v14, l_v11);
goto block1;
Hopefully this makes the generated .c much more readable. (I hope to
work on the names of locals also; ideally they ought to embed the python
indentifier)
This is on a Fedora 13 x86_64 box, with cpython 2.6.5
Caveats:
- this is my first patch to PyPy (please be gentle!): I hope I've
correctly understood the various levels, but I suspect I may still be
missing things at the conceptual level
- haven't completed the full unit tests yet.
- the patch attempts to propagate the "offset" of flowspace
SpaceOperation instances in a couple of places where the offset was
being discarded:
- in BaseInliner.copy_operation
- when generating operations in rtyper: set each of the new
operations' "offset" to that of the high-level operation
- I added the offset to SpaceOperation.__repr__() to help track the
above down.
- I also pass the reference to the func object from the flowspace
FunctionGraph to the rtyper's copy (I wonder if this affect memory usage
during translation?)
- most functions appear correct, but there are some where I'm not at
all convinced by the output (see below)
- the patch is assuming that within any given generated C function,
there is a single python function, and that the offsets are indexes into
the bytecode for that function's code object. I suspect that this is an
oversimplification, and why I'm seeing the errors that I'm seeing.
- I haven't stripped some of my debugging python comments from
funcgen.py in the patch (this is a work in progress).
- I haven't tried doing this on a full build of the interpreter
(having highly readable generated .c for this is my objective [1]).
One other idea I had was to sort the blocks in the function by the
bytecode offset; currently the c blocks seem to "jump around" a fair bit
relative to the corresponding rpython code. Has any tuning been done on
the ordering of the blocks within the generated .c code? (or is it
assumed that the .c compiler will "flatten" these, for the case when a
node has a single successor node?)
(I wonder if similar work could be done on the JVM and CLI code
generators?)
As mentioned above, I think my patch is getting it wrong in a few
places. Here's an example:
struct pypy_rpy_string0 *pypy_g_mallocstr__Signed(long l_length_3) {
...snip...
/* Python source
'/home/david/coding/pypy-svn-anon/trunk-clean/pypy/rpython/lltypesystem/rstr.py'
* 35 : def mallocstr(length):
* 36 : ll_assert(length >= 0, "negative string length")
* 37 : r = malloc(TP, length)
* 38 : if not we_are_translated() or not malloc_zero_filled:
* 39 : r.hash = 0
* 40 : return r
*/
...snip...
block15:
/* 37 : r = malloc(TP, length) */
OP_ADR_SUB(l_v202, 0, l_v234);
l_v235 = (struct pypy_header0 *)l_v234;
l_v236 = RPyField(l_v235, h_refcount);
/* 38 : if not we_are_translated() or not malloc_zero_filled:
*/
OP_INT_ADD(l_v236, 1L, l_v237);
RPyField(l_v235, h_refcount) = l_v237;
goto block5;
The C code doesn't look anything like the .py code that my patch is
inserting in the above. (My current guess is that I need to be smarter
about inlined operations, though that's a guess at this stage)
Hope this is helpful
Dave
[1] http://fedoraproject.org/wiki/Features/PyPyStack is what I'm aiming
at
Index: objspace/flow/model.py
===================================================================
--- objspace/flow/model.py (revision 79962)
+++ objspace/flow/model.py (working copy)
@@ -352,8 +352,9 @@
return hash((self.opname,tuple(self.args),self.result))
def __repr__(self):
- return "%r = %s(%s)" % (self.result, self.opname,
- ", ".join(map(repr, self.args)))
+ return "%r = %s(%s) (offset %r)" % (self.result, self.opname,
+ ", ".join(map(repr, self.args)),
+ self.offset)
class Atom(object):
def __init__(self, name):
Index: translator/c/funcgen.py
===================================================================
--- translator/c/funcgen.py (revision 79962)
+++ translator/c/funcgen.py (working copy)
@@ -1,4 +1,6 @@
import sys
+import inspect
+import dis
from pypy.translator.c.support import USESLOTS # set to False if necessary while refactoring
from pypy.translator.c.support import cdecl
from pypy.translator.c.support import llvalue_from_constant, gen_assignments
@@ -22,6 +24,17 @@
KEEP_INLINED_GRAPHS = False
+def get_linenum_for_offset(linestarts, offset):
+ # Locate the source line number of a given bytecode offset
+ # linestarts is output of dis.findlinestarts
+ lastlineno = -1
+ for disoffset, lineno in linestarts:
+ if disoffset > offset:
+ return lastlineno
+ lastlineno = lineno
+ return lastlineno
+
+
class FunctionCodeGenerator(object):
"""
Collects information about a function which we have to generate
@@ -210,14 +223,53 @@
def cfunction_body(self):
graph = self.graph
+ #yield '/* graph: %r */' % graph
+ # Try to print python source code:
+ if hasattr(graph, 'func'):
+ filename = inspect.getfile(graph.func)
+ f = open(filename, 'r')
+ filelines = [line.rstrip() for line in f.readlines()]
+ f.close()
+ #yield '/* name: %r */' % filename
+ src, startline = inspect.getsourcelines(graph.func)
+ yield '/* Python source %r' % filename
+ for i, line in enumerate(src):
+ line = line.rstrip()
+ # FuncNode.funcgen_implementation treats lines ending in ':'
+ # as C blocks, which messes up the formatting.
+ # Work around this:
+ if line.endswith(':'):
+ line += ' '
+ yield ' * %4d : %s' % (startline + i, line)
+ yield ' */'
+
+ linestarts = list(dis.findlinestarts(graph.func.func_code))
+ #yield '/* %r */' % linestarts
+
yield 'goto block0;' # to avoid a warning "this label is not used"
# generate the body of each block
for block in graph.iterblocks():
+ cursrclineno = -1
myblocknum = self.blocknum[block]
yield ''
yield 'block%d:' % myblocknum
+ #yield "/* type(block): %r */" % (type(block), )
for i, op in enumerate(block.operations):
+ #yield "/* type(op): %r */" % (type(op), )
+ #yield "/* op.offset: %r */" % (op.offset, )
+ if op.offset != -1:
+ if hasattr(graph, 'func'):
+ # Get linenum for this offset; if it's changed, print src:
+ srclineno = get_linenum_for_offset(linestarts, op.offset)
+ #yield "/* linenum for offset %r = %r */" % (op.offset, srclineno)
+ if srclineno != cursrclineno:
+ cursrclineno = srclineno
+ try:
+ yield '/* %4d : %s */' % (srclineno, filelines[srclineno-1])
+ except IndexError:
+ yield '/* IndexError: %r */' % (srclineno, )
+
for line in self.gen_op(op):
yield line
if len(block.exits) == 0:
Index: translator/llsupport/wrapper.py
===================================================================
--- translator/llsupport/wrapper.py (revision 79962)
+++ translator/llsupport/wrapper.py (working copy)
@@ -59,6 +59,8 @@
# "return result"
block = Block(wrapper_inputargs)
wgraph = FunctionGraph('pyfn_' + (newname or func.func_name), block)
+ if hasattr(graph, 'func'):
+ wgraph.func = graph.func
translator.update_call_graph(wgraph, graph, object())
translator.graphs.append(wgraph)
block.operations[:] = newops
Index: translator/backendopt/inline.py
===================================================================
--- translator/backendopt/inline.py (revision 79962)
+++ translator/backendopt/inline.py (working copy)
@@ -297,7 +297,7 @@
def copy_operation(self, op):
args = [self.get_new_name(arg) for arg in op.args]
- result = SpaceOperation(op.opname, args, self.get_new_name(op.result))
+ result = SpaceOperation(op.opname, args, self.get_new_name(op.result), op.offset)
return result
def copy_block(self, block):
Index: rpython/rtyper.py
===================================================================
--- rpython/rtyper.py (revision 79962)
+++ rpython/rtyper.py (working copy)
@@ -800,7 +800,7 @@
return vars
def genop(self, opname, args_v, resulttype=None):
- return self.llops.genop(opname, args_v, resulttype)
+ return self.llops.genop(opname, args_v, resulttype, self.spaceop.offset)
def gendirectcall(self, ll_function, *args_v):
return self.llops.gendirectcall(ll_function, *args_v)
@@ -935,7 +935,7 @@
v.concretetype))
return v
- def genop(self, opname, args_v, resulttype=None):
+ def genop(self, opname, args_v, resulttype=None, offset=-1):
try:
for v in args_v:
v.concretetype
@@ -944,7 +944,7 @@
" and pass its result to genop(),"
" never hop.args_v directly.")
vresult = Variable()
- self.append(SpaceOperation(opname, args_v, vresult))
+ self.append(SpaceOperation(opname, args_v, vresult, offset))
if resulttype is None:
vresult.concretetype = Void
return None
_______________________________________________
[email protected]
http://codespeak.net/mailman/listinfo/pypy-dev
