[Python-Dev] Re: Helpers for dynamic bytecode generation
I think this probably belongs on python-list instead of python-dev because python-dev is for development _of_ python, not _with_ python. To answer your question though, there are a few tools that do this: - https://github.com/vstinner/bytecode - https://github.com/ll/codetransformer I am one of the authors of codetransformer, so I am more familiar with how to use that to generate bytecode, but the bytecode project may be easier for your use case. On Thu, Oct 24, 2019 at 8:59 PM Yonatan Zunger wrote: > Hi everyone, > > I've found myself recently writing Python code that dynamically generates > bytecode.¹ I now have yet another case where I'm having to do this, in > which my nice situation of being able to easily precompute all the jump > addresses no longer holds. So I'm starting to write a helper to make it > easy to write bytecode from Python, with its basic API calls being > write(opcode, arg) and nextLine(optional label). The argument can be an > int, name, local name, constant, label, etc., depending on the opcode, and > it maintains all the appropriate tables and finally dumps a code object at > the end. > > All of which is well and good and makes life much easier, but... I am > *not* looking forward to writing the logic that basically duplicates that > of assemble() in compile.c, of splitting all of this into basic blocks and > computing the correct jump positions and so on before finally dumping out > the bytecode. > > Has anyone already done this that people know of? (Searching the > Internetz didn't turn anything up) Failing that, to what extent is it > reasonable to either consider assemble() as some kind of sane API point > into compile.c, and/or add some new API in compile.h which implements all > of the stuff described above in C? > > (I'm fully expecting the answer to these latter questions to be "you have > got to be kidding me," but figured it was wiser to check than to reinvent > this particular wheel if it isn't necessary) > > Yonatan > > > > ¹ Not out of masochism, in case you're wondering; there was a real use > case. A storage system would receive a read request that specified a bunch > of (key, condition) pairs, where conditions where either return any value, > return an exact value, or return values in a range. It would then receive > between 1 and 1M (depending on the request parameters) candidate cells from > the underlying storage layers, each of which had a tuple of bytes as its > actual key values; it had to compare each of those tuples against the > request parameters, and yield the values which matched. Because it's an > inner loop and can easily be called 1M times, doing this in pure Python > slows things down by a lot. Because it's also only called once, doing some > really expensive overhead like synthesizing Python code and calling > compile() on it would also slow things down a lot. But converting a bunch > of (key, condition) pairs to a really efficient function from tuples of > bytes to bools was pretty easy. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/E57DTM65LFEROFZLHKRV442JPPFAWNJU/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TF5NLPS5DQVR3N75VHIXOGSK5APZMLGK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Helpers for dynamic bytecode generation
@jjevnik -- Oops! I meant to send it there and sent it here by mistake. But thank you, everyone, for the recommendations! I'll check them all out and (mercifully) will probably not try to write this from scratch. On Thu, Oct 24, 2019 at 11:05 PM Brandt Bucher wrote: > Interesting that you bring this up. Just earlier this week I published the > first version of a new package that lets you write compiled bytecode > instructions inline with pure-Python syntax. The code's still a bit messy, > being only a week old and all, but it works as advertised for CPython 3.6.2 > through 3.9.0a0, and even includes neat features like labeled jumps, unused > name/constant removal, stack size adjustments, etc... Perhaps it'll be > useful to you (or at least you'll find it interesting): > https://github.com/brandtbucher/hax > > Victor's Stinner's Bytecode package (already mentioned) is surely better > for *dynamic* generation... I've never used it personally, but it looks > great. > > Definitely not for the faint of heart, though! ;) > > Brandt > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/PAQUZZ4MHZ6TJFIBNCPZGZBW6DOZFUJG/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YOQT3QSXUY3TI3FXOSHLRY24GJBJHLWG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Helpers for dynamic bytecode generation
Interesting that you bring this up. Just earlier this week I published the first version of a new package that lets you write compiled bytecode instructions inline with pure-Python syntax. The code's still a bit messy, being only a week old and all, but it works as advertised for CPython 3.6.2 through 3.9.0a0, and even includes neat features like labeled jumps, unused name/constant removal, stack size adjustments, etc... Perhaps it'll be useful to you (or at least you'll find it interesting): https://github.com/brandtbucher/hax Victor's Stinner's Bytecode package (already mentioned) is surely better for *dynamic* generation... I've never used it personally, but it looks great. Definitely not for the faint of heart, though! ;) Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PAQUZZ4MHZ6TJFIBNCPZGZBW6DOZFUJG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Helpers for dynamic bytecode generation
Hi, On Fri, 25 Oct 2019 at 04:13, Jonathan Goble wrote: >> Has anyone already done this that people know of? (Searching the Internetz >> didn't turn anything up) Failing that, to what extent is it reasonable to >> either consider assemble() as some kind of sane API point into compile.c PyPy contains a complete rewrite of compile.c in Python, which should be relatively easy to extract. Here's the py3.6 version: https://bitbucket.org/pypy/pypy/src/py3.6/pypy/interpreter/astcompiler/ Independently, you may also want to benchmark your code on PyPy (*without* using any bytecode generation, just plain Python loops). A bientôt, Armin. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NUPFLTIO7GB2DNBFHKVBMKGYYZE4PPO4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Helpers for dynamic bytecode generation
On Thu, Oct 24, 2019, 9:05 PM Yonatan Zunger wrote: > Hi everyone, > > Has anyone already done this that people know of? (Searching the > Internetz didn't turn anything up) Failing that, to what extent is it > reasonable to either consider assemble() as some kind of sane API point > into compile.c, and/or add some new API in compile.h which implements all > of the stuff described above in C? > > (I'm fully expecting the answer to these latter questions to be "you have > got to be kidding me," but figured it was wiser to check than to reinvent > this particular wheel if it isn't necessary) > There is byteplay (https://github.com/tallforasmurf/byteplay), but it is broken on Python 3.6+ due to the change to wordcode. There is also a newer one called simply "bytecode" (https://github.com/vstinner/bytecode) that appears to work through at least 3.7 (though I haven't personally tried it. > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L7UWUOI22RNKJUUXDB6FV4XXBH3HP4QT/ Code of Conduct: http://python.org/psf/codeofconduct/
