Re: [Python-Dev] Unexpected bytecode difference
As a general rule, you should not expect the bytecode to be the same between different versions of CPython, including minor version changes. For example, the instructions for dictionary literals are different in 3.4, 3.5, and 3.6. On Fri, Jan 19, 2018 at 6:54 PM, Victor Stinner wrote: > Python bytecode format changed deeply in Python 3.6. It now uses > regular units of 2 bytes, instead of 1 or 3 bytes depending if the > instruction has an argument. > > See for example https://bugs.python.org/issue26647 "wordcode". > > But CALL_FUNCTION bytecode also evolved. > > Victor > > 2018-01-20 0:46 GMT+01:00 Alexander Belopolsky < > [email protected]>: > > I have encountered the following difference between Python 3 and 2: > > > > (py3) > compile('xxx', '<>', 'eval').co_code > > b'e\x00S\x00' > > > > (py2) > compile('xxx', '<>', 'eval').co_code > > 'e\x00\x00S' > > > > Note that 'S' (the code for RETURN_VALUE) and a zero byte are swapped > > in Python 2 compared to Python 3. Is this change documented > > somewhere? > > ___ > > Python-Dev mailing list > > [email protected] > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > victor.stinner%40gmail.com > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > joe%40quantopian.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] inconsistency in annotated assigned targets
Currently there are many ways to introduce variables in Python; however, only a few allow annotations. I was working on a toy language and chose to base my syntax on Python's when I noticed that I could not annotate a loop iteration variable. For example: for x: int in range(5): ... This led me to search for other places where new variables are introduced and I noticed that the `as` target of a context manager cannot have an annotation. In the case of a context manager, it would probably need parenthesis to avoid ambiguity with a single-line with statement, for example: with ctx as (variable: annotation): body Finally, you cannot annotate individual members of a destructuring assignment like: a: int, b: int, c: int = 1, 2, 3 Looking at the grammar, these appear to be `expr` or `exprlist` targets. One change may be to allow arbitrary expressions to have an annotation . This would be a small change to the grammar but would potentially have a large effect on the language or static analysis tools. I am posting on the mailing list to see if this is a real problem, and if so, is it worth investing any time to address it. I would be happy to attempt to fix this, but I don't want to start if people don't want the change. Also, I apologize if this should have gone to python-idea; this feels somewhere between a bug report and implementation question more than a new feature so I wasn't sure which list would be more appropriate. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] inconsistency in annotated assigned targets
Thank you for the clarification! I should have looked through the PEPs first. On Thu, Jan 25, 2018 at 10:14 PM, Guido van Rossum wrote: > PEP 526 has this in the "Rejected/Postponed Proposals" section: > > - **Allow annotations in** ``with`` **and** ``for`` **statement:** > This was rejected because in ``for`` it would make it hard to spot the > actual > iterable, and in ``with`` it would confuse the CPython's LL(1) parser. > > > On Thu, Jan 25, 2018 at 3:17 PM, Jelle Zijlstra > wrote: > >> >> >> 2018-01-25 15:00 GMT-08:00 Joe Jevnik via Python-Dev < >> [email protected]>: >> >>> Currently there are many ways to introduce variables in Python; however, >>> only a few allow annotations. I was working on a toy language and chose to >>> base my syntax on Python's when I noticed that I could not annotate a loop >>> iteration variable. For example: >>> >>> for x: int in range(5): >>> ... >>> >>> This led me to search for other places where new variables are >>> introduced and I noticed that the `as` target of a context manager cannot >>> have an annotation. In the case of a context manager, it would probably >>> need parenthesis to avoid ambiguity with a single-line with statement, for >>> example: >>> >>> with ctx as (variable: annotation): body >>> >>> Finally, you cannot annotate individual members of a destructuring >>> assignment like: >>> >>> a: int, b: int, c: int = 1, 2, 3 >>> >>> Looking at the grammar, these appear to be `expr` or `exprlist` targets. >>> One change may be to allow arbitrary expressions to have an annotation . >>> This would be a small change to the grammar but would potentially have a >>> large effect on the language or static analysis tools. >>> >>> I am posting on the mailing list to see if this is a real problem, and >>> if so, is it worth investing any time to address it. I would be happy to >>> attempt to fix this, but I don't want to start if people don't want the >>> change. Also, I apologize if this should have gone to python-idea; this >>> feels somewhere between a bug report and implementation question more than >>> a new feature so I wasn't sure which list would be more appropriate. >>> >> I have written a fair amount of code with variable annotations, and I >> don't remember ever wanting to add annotations in any of the three contexts >> you mention. In practice, variable annotations are usually needed for >> class/instance variables and for variables whose type the type checker >> can't infer. The types of loop iteration variables and context manager >> assignment targets can almost always be inferred trivially. >> >> >>> >>> ___ >>> Python-Dev mailing list >>> [email protected] >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: https://mail.python.org/mailma >>> n/options/python-dev/jelle.zijlstra%40gmail.com >>> >>> >> >> ___ >> Python-Dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% >> 40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Register-based VM [Was: Possible performance regression]
METH_FASTCALL passing arguments on the stack doesn't necessarily mean it will be slow. In x86 there are calling conventions that read all the arguments from the stack, but the rest of the machine is register based. Python could also look at ABI calling conventions for inspiration, like x86-64 where some arguments up to a fixed amount are passed on the stack and the rest are passed on the stack. One thing that I am wondering is would Python want to use a global set of registers and a global data stack, or continue to have a new data stack (and now registers) per call stack. If Python switched to a global stack and global registers we may be able to eliminate a lot of instructions that just shuffle data from the caller's stack to the callee's stack. On Tue, Feb 26, 2019 at 4:55 PM Victor Stinner wrote: > Hum, I read again my old REGISTERVM.txt that I wrote a few years ago. > > A little bit more context. In my "registervm" fork I also tried to > implement further optimizations like moving invariants out of the > loop. Some optimizations could change the Python semantics, like > remove "duplicated" LOAD_GLOBAL whereas the global might be modified > in the middle. I wanted to experiment such optimizations. Maybe it was > a bad idea to convert stack-based bytecode to register-based bytecode > and experiment these optimizations at the same time. > > Victor > > Le mar. 26 févr. 2019 à 22:42, Victor Stinner a > écrit : > > > > No, I wasn't aware of this project. My starting point was: > > > > http://static.usenix.org/events/vee05/full_papers/p153-yunhe.pdf > > Yunhe Shi, David Gregg, Andrew Beatty, M. Anton Ertl, 2005 > > > > See also my email to python-dev that I sent in 2012: > > https://mail.python.org/pipermail/python-dev/2012-November/122777.html > > > > Ah, my main issue was my implementation is that I started without > > taking care of clearing registers when the stack-based bytecode > > implicitly cleared a reference (decref), like "POP_TOP" operation. > > > > I added "CLEAR_REG" late in the development and it caused me troubles, > > and the "correct" register-based bytecode was less efficient than > > bytecode without CLEAR_REG. But my optimizer was very limited, too > > limited. > > > > Another implementation issue that I had was to understand some > > "implicit usage" of the stack like try/except which do black magic, > > whereas I wanted to make everything explicit for registers. I'm > > talking about things like "POP_BLOCK" and "SETUP_EXCEPT". In my > > implementation, I kept support for stack-based bytecode, and so I had > > some inefficient code and some corner cases. > > > > My approach was to convert stack-based bytecode to register-based > > bytecode on the fly. Having both in the same code allowed to me run > > some benchmarks. Maybe it wasn't the best approach, but I didn't feel > > able to write a real compiler (AST => bytecode). > > > > Victor > > > > Le mar. 26 févr. 2019 à 21:58, Neil Schemenauer > a écrit : > > > > > > On 2019-02-26, Victor Stinner wrote: > > > > I made an attempt once and it was faster: > > > > https://faster-cpython.readthedocs.io/registervm.html > > > > > > Interesting. I don't think I have seen that before. Were you aware > > > of "Rattlesnake" before you started on that? It seems your approach > > > is similar. Probably not because I don't think it is easy to find. > > > I uploaded a tarfile I had on my PC to my web site: > > > > > > http://python.ca/nas/python/rattlesnake20010813/ > > > > > > It seems his name doesn't appear in the readme or source but I think > > > Rattlesnake was Skip Montanaro's project. I suppose my idea of > > > unifying the local variables and the registers could have came from > > > Rattlesnake. Very little new in the world. ;-P > > > > > > Cheers, > > > > > > Neil > > > > > > > > -- > > Night gathers, and now my watch begins. It shall not end until my death. > > > > -- > Night gathers, and now my watch begins. It shall not end until my death. > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/joe%40quantopian.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inplace operations for PyLong objects
Is it true that checking for refcount == 1 is enough? What if a user wrote: args = (compute_integer(), 5) # give away args to someone int.__iadd__(*args) here `args[0]` still has refcount=1 because only `args` owns this integer. On Fri, Sep 1, 2017 at 4:19 PM, Jelle Zijlstra wrote: > > > 2017-09-01 13:05 GMT-07:00 Chris Barker : > >> On Thu, Aug 31, 2017 at 5:12 PM, Antoine Pitrou >> wrote: >> >>> I'm skeptical there are some programs out there that are limited by the >>> speed of PyLong inplace additions. >>> >> >> indeed, but that could be said about any number of operations. >> >> My question is -- how can the interpreter know if it can alter what is >> supposed to be an immutable in-place? If it's used only internally to a >> function, the it would be safe, but how to know that? >> > I believe Catalin's implementation checks if the object's refcount is 1. > If that is the case, it is safe to mutate it. > >> >> -CHB >> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R(206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> [email protected] >> >> ___ >> Python-Dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle. >> zijlstra%40gmail.com >> >> > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > joe%40quantopian.com > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inplace operations for PyLong objects
The string concat optimization happens in the interpreter dispatch for INPLACE_ADD On Fri, Sep 1, 2017 at 9:10 PM, Greg Ewing wrote: > Chris Angelico wrote: > >> This particular example is safe, because the arguments get passed >> individually - so 'args' has one reference, plus there's one more for >> the actual function call >> > > However, that's also true when you use the += operator, > so if the optimisation is to trigger at all in any useful > case, the refcount threshold needs to be set higher than > 1. > > Some experiments I did suggest that if you set it high > enough for x += y to trigger it, then it will also be > triggered in Joe's case. > > BTW, isn't there already a similar optimisation somewhere > for concatenating strings? Does it still exist? How does > it avoid this issue? > > -- > Greg > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/joe% > 40quantopian.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Helpers for dynamic bytecode generation
I think this probably belongs on python-list instead of python-dev because python-dev is for development _of_ python, not _with_ python. To answer your question though, there are a few tools that do this: - https://github.com/vstinner/bytecode - https://github.com/ll/codetransformer I am one of the authors of codetransformer, so I am more familiar with how to use that to generate bytecode, but the bytecode project may be easier for your use case. On Thu, Oct 24, 2019 at 8:59 PM Yonatan Zunger wrote: > Hi everyone, > > I've found myself recently writing Python code that dynamically generates > bytecode.¹ I now have yet another case where I'm having to do this, in > which my nice situation of being able to easily precompute all the jump > addresses no longer holds. So I'm starting to write a helper to make it > easy to write bytecode from Python, with its basic API calls being > write(opcode, arg) and nextLine(optional label). The argument can be an > int, name, local name, constant, label, etc., depending on the opcode, and > it maintains all the appropriate tables and finally dumps a code object at > the end. > > All of which is well and good and makes life much easier, but... I am > *not* looking forward to writing the logic that basically duplicates that > of assemble() in compile.c, of splitting all of this into basic blocks and > computing the correct jump positions and so on before finally dumping out > the bytecode. > > Has anyone already done this that people know of? (Searching the > Internetz didn't turn anything up) Failing that, to what extent is it > reasonable to either consider assemble() as some kind of sane API point > into compile.c, and/or add some new API in compile.h which implements all > of the stuff described above in C? > > (I'm fully expecting the answer to these latter questions to be "you have > got to be kidding me," but figured it was wiser to check than to reinvent > this particular wheel if it isn't necessary) > > Yonatan > > > > ¹ Not out of masochism, in case you're wondering; there was a real use > case. A storage system would receive a read request that specified a bunch > of (key, condition) pairs, where conditions where either return any value, > return an exact value, or return values in a range. It would then receive > between 1 and 1M (depending on the request parameters) candidate cells from > the underlying storage layers, each of which had a tuple of bytes as its > actual key values; it had to compare each of those tuples against the > request parameters, and yield the values which matched. Because it's an > inner loop and can easily be called 1M times, doing this in pure Python > slows things down by a lot. Because it's also only called once, doing some > really expensive overhead like synthesizing Python code and calling > compile() on it would also slow things down a lot. But converting a bunch > of (key, condition) pairs to a really efficient function from tuples of > bytes to bools was pretty easy. > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/E57DTM65LFEROFZLHKRV442JPPFAWNJU/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/TF5NLPS5DQVR3N75VHIXOGSK5APZMLGK/ Code of Conduct: http://python.org/psf/codeofconduct/
