Re: Byte code descriptions somewhere?

[Chris Angelico]

On Mon, Oct 3, 2016 at 9:58 AM, Cem Karan  wrote:
>> To execute your own code, look at types.FunctionType and
>> types.CodeType, particularly the latter's 'codestring' argument
>> (stored as the co_code attribute). Be careful: you can easily crash
>> CPython if you mess this stuff up :)
>
> Ah, but crashing things is how we learn! :)
>

I never said not to do it, just to be careful :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Byte code descriptions somewhere?

[Cem Karan]

On Oct 1, 2016, at 7:34 PM, breamore...@gmail.com wrote:

> On Saturday, October 1, 2016 at 11:57:17 PM UTC+1, Cem Karan wrote:
>> Hi all, I've all of a sudden gotten interested in the CPython interpreter, 
>> and started trying to understand how it ingests and runs byte code.  I found 
>> Include/opcode.h in the python sources, and I found some basic documentation 
>> on how to add in new opcodes online, but I haven't found the equivalent of 
>> an assembly manual like you might for x86, etc.  Is there something similar 
>> to a manual dedicated to python byte code?  Also, is there a manual for how 
>> the interpreter expects the stack, etc. to be setup so that all interactions 
>> go as expected (garbage collections works, exceptions work, etc.)?  
>> Basically, I want a manual similar to what Intel or AMD might put out for 
>> their chips so that all executables behave nicely with one another.
>> 
>> Thanks,
>> Cem Karan
> 
> Further to Ben Finney's answer this 
> https://docs.python.org/devguide/compiler.html should help.
> 
> Kindest regards.
> 
> Mark Lawrence.
> -- 
> https://mail.python.org/mailman/listinfo/python-list

Thank you!

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Byte code descriptions somewhere?

[Cem Karan]

On Oct 1, 2016, at 8:30 PM, Ned Batchelder  wrote:

> On Saturday, October 1, 2016 at 7:48:09 PM UTC-4, Cem Karan wrote:
>> Cool, thank you!  Quick experimentation suggests that I don't need to worry 
>> about marking anything for garbage collection, correct?  The next question 
>> is, how do I create a stream of byte codes that can be interpreted by 
>> CPython directly?  I don't mean 'use the compile module', I mean writing my 
>> own byte array with bytes that CPython can directly interpret.
> 
> In Python 2, you use new.code: 
> https://docs.python.org/2/library/new.html#new.code  It takes a bytestring of 
> byte codes as one of its
> twelve (!) arguments.
> 
> Something that might help (indirectly) with understanding bytecode:
> byterun (https://github.com/nedbat/byterun) is a pure-Python implementation
> of a Python bytecode VM.
> 
> --Ned.

byterun seems like the perfect project to work through to understand things.  
Thank you for pointing it out!

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Byte code descriptions somewhere?

[Cem Karan]

On Oct 1, 2016, at 7:56 PM, Chris Angelico  wrote:

> On Sun, Oct 2, 2016 at 10:47 AM, Cem Karan  wrote:
>> Cool, thank you!  Quick experimentation suggests that I don't need to worry 
>> about marking anything for garbage collection, correct?  The next question 
>> is, how do I create a stream of byte codes that can be interpreted by 
>> CPython directly?  I don't mean 'use the compile module', I mean writing my 
>> own byte array with bytes that CPython can directly interpret.
>> 
> 
> "Marking for garbage collection" in CPython is done by refcounts; the
> bytecode is at a higher level than that.
> 
 dis.dis("x = y*2")
>  1   0 LOAD_NAME0 (y)
>  3 LOAD_CONST   0 (2)
>  6 BINARY_MULTIPLY
>  7 STORE_NAME   1 (x)
> 10 LOAD_CONST   1 (None)
> 13 RETURN_VALUE
> 
> A LOAD operation will increase the refcount (a ref is on the stack),
> BINARY_MULTIPLY dereferences the multiplicands and adds a ref to the
> product, STORE will deref whatever previously was stored, etc.
> 
> To execute your own code, look at types.FunctionType and
> types.CodeType, particularly the latter's 'codestring' argument
> (stored as the co_code attribute). Be careful: you can easily crash
> CPython if you mess this stuff up :)

Ah, but crashing things is how we learn! :)  

That said, types.CodeType and types.FunctionType appear to be EXACTLY what I'm 
looking for!  Thank you!  Although I have to admit, the built-in docs for 
types.CodeType are concerning... "Create a code object.  Not for the faint of 
heart." Maybe that should be updated to "Here there be dragons"?  I'll poke 
through python's sources to get an idea of how to use codestring argument, but 
I'll probably be asking more questions on here about it.

Thanks,
Cem Karan
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Byte code descriptions somewhere?

[Cem Karan]

I kind of got the feeling that was so from reading the docs in the source code. 
 Too bad! :(

Cem

On Oct 1, 2016, at 7:53 PM, Paul Rubin  wrote:

> Cem Karan  writes:
>> how do I create a stream of byte codes that can be interpreted by
>> CPython directly?
> 
> Basically, study the already existing code and do something similar.
> The CPython bytecode isn't standardized like JVM bytecode.  It's
> designed for the interpreter's convenience, not officially documented,
> and (somewhat) subject to change between versions.
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Byte code descriptions somewhere?

[breamoreboy]

On Saturday, October 1, 2016 at 11:57:17 PM UTC+1, Cem Karan wrote:
> Hi all, I've all of a sudden gotten interested in the CPython interpreter, 
> and started trying to understand how it ingests and runs byte code.  I found 
> Include/opcode.h in the python sources, and I found some basic documentation 
> on how to add in new opcodes online, but I haven't found the equivalent of an 
> assembly manual like you might for x86, etc.  Is there something similar to a 
> manual dedicated to python byte code?  Also, is there a manual for how the 
> interpreter expects the stack, etc. to be setup so that all interactions go 
> as expected (garbage collections works, exceptions work, etc.)?  Basically, I 
> want a manual similar to what Intel or AMD might put out for their chips so 
> that all executables behave nicely with one another.
> 
> Thanks,
> Cem Karan

Further to Ben Finney's answer this 
https://docs.python.org/devguide/compiler.html should help.

Kindest regards.

Mark Lawrence.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Byte code descriptions somewhere?

[Ned Batchelder]

On Saturday, October 1, 2016 at 7:48:09 PM UTC-4, Cem Karan wrote:
> Cool, thank you!  Quick experimentation suggests that I don't need to worry 
> about marking anything for garbage collection, correct?  The next question 
> is, how do I create a stream of byte codes that can be interpreted by CPython 
> directly?  I don't mean 'use the compile module', I mean writing my own byte 
> array with bytes that CPython can directly interpret.

In Python 2, you use new.code: 
https://docs.python.org/2/library/new.html#new.code  It takes a bytestring of 
byte codes as one of its
twelve (!) arguments.

Something that might help (indirectly) with understanding bytecode:
byterun (https://github.com/nedbat/byterun) is a pure-Python implementation
of a Python bytecode VM.

--Ned.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Byte code descriptions somewhere?

[Chris Angelico]

On Sun, Oct 2, 2016 at 10:47 AM, Cem Karan  wrote:
> Cool, thank you!  Quick experimentation suggests that I don't need to worry 
> about marking anything for garbage collection, correct?  The next question 
> is, how do I create a stream of byte codes that can be interpreted by CPython 
> directly?  I don't mean 'use the compile module', I mean writing my own byte 
> array with bytes that CPython can directly interpret.
>

"Marking for garbage collection" in CPython is done by refcounts; the
bytecode is at a higher level than that.

>>> dis.dis("x = y*2")
  1   0 LOAD_NAME0 (y)
  3 LOAD_CONST   0 (2)
  6 BINARY_MULTIPLY
  7 STORE_NAME   1 (x)
 10 LOAD_CONST   1 (None)
 13 RETURN_VALUE

A LOAD operation will increase the refcount (a ref is on the stack),
BINARY_MULTIPLY dereferences the multiplicands and adds a ref to the
product, STORE will deref whatever previously was stored, etc.

To execute your own code, look at types.FunctionType and
types.CodeType, particularly the latter's 'codestring' argument
(stored as the co_code attribute). Be careful: you can easily crash
CPython if you mess this stuff up :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Byte code descriptions somewhere?

[Paul Rubin]

Cem Karan  writes:
> how do I create a stream of byte codes that can be interpreted by
> CPython directly?

Basically, study the already existing code and do something similar.
The CPython bytecode isn't standardized like JVM bytecode.  It's
designed for the interpreter's convenience, not officially documented,
and (somewhat) subject to change between versions.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Byte code descriptions somewhere?

[Cem Karan]

Cool, thank you!  Quick experimentation suggests that I don't need to worry 
about marking anything for garbage collection, correct?  The next question is, 
how do I create a stream of byte codes that can be interpreted by CPython 
directly?  I don't mean 'use the compile module', I mean writing my own byte 
array with bytes that CPython can directly interpret.

Thanks,
Cem Karan


On Oct 1, 2016, at 7:02 PM, Ben Finney  wrote:

> Cem Karan  writes:
> 
>> Hi all, I've all of a sudden gotten interested in the CPython
>> interpreter, and started trying to understand how it ingests and runs
>> byte code.
> 
> That sounds like fun!
> 
>> Is there something similar to a manual dedicated to python byte code?
> 
> The Python documentation for the ‘dis’ module shows not only how to use
> that module for dis-assembly of Python byte code, but also a reference
> for the byte code.
> 
>32.12. dis — Disassembler for Python bytecode
> 
>
> 
> -- 
> \ “Skepticism is the highest duty and blind faith the one |
>  `\   unpardonable sin.” —Thomas Henry Huxley, _Essays on |
> _o__)   Controversial Questions_, 1889 |
> Ben Finney
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Byte code descriptions somewhere?

[Ben Finney]

Cem Karan  writes:

> Hi all, I've all of a sudden gotten interested in the CPython
> interpreter, and started trying to understand how it ingests and runs
> byte code.

That sounds like fun!

> Is there something similar to a manual dedicated to python byte code?

The Python documentation for the ‘dis’ module shows not only how to use
that module for dis-assembly of Python byte code, but also a reference
for the byte code.

32.12. dis — Disassembler for Python bytecode



-- 
 \ “Skepticism is the highest duty and blind faith the one |
  `\   unpardonable sin.” —Thomas Henry Huxley, _Essays on |
_o__)   Controversial Questions_, 1889 |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list