[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Roundup Robot]

Roundup Robot added the comment:

New changeset bf997b22df06 by Victor Stinner in branch '3.5':
Fix BytecodeTestCase.assertNotInBytecode()
https://hg.python.org/cpython/rev/bf997b22df06

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan added the comment:

I checked in the missing file after I woke up this morning. Maybe I'll learn to 
use hg import instead of patch some day...

Sorry for the noise.

--
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Thomas Kluyver]

Thomas Kluyver added the comment:

Ping - the latest patches (dis_api3  test_peepholer) are ready for review when 
someone's got a moment. Thanks!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan added the comment:

I created issue 17916 after realising that the new OO API doesn't yet provide 
an equivalent to dis.distb that returns an appropriate Bytecode object.

(I don't think it makes sense to hold up this patch for that change)

--
assignee:  - ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


--
stage: patch review - commit review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan added the comment:

Good thing test_peepholer was moved out to a separate patch - a failure of that 
picked up a bug in the new disassembly output (unifying the handling of name 
and constant dereferences had changed the way constant strings were reported in 
the disassembly, and the error was consistent in both the new implementation 
and in the new tests due to the way the expected test results had been 
generated)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Roundup Robot]

Roundup Robot added the comment:

New changeset f65b867ce817 by Nick Coghlan in branch 'default':
Issue #11816: multiple improvements to the dis module
http://hg.python.org/cpython/rev/f65b867ce817

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Roundup Robot]

Roundup Robot added the comment:

New changeset d3fee4c64654 by Nick Coghlan in branch 'default':
Issue #11816: switch test_peepholer to bytecode_helper
http://hg.python.org/cpython/rev/d3fee4c64654

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan added the comment:

And two-and-a-bit years later, we're done - thanks all, any further feedback or 
problems can be filed as a new issue :)

--
resolution:  - fixed
stage: commit review - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Charles-François Natali]

Charles-François Natali added the comment:

test_dis is failing on some buildbots:

http://buildbot.python.org/all/builders/AMD64 Ubuntu LTS 
3.x/builds/1674/steps/test/logs/stdio

Re-running test 'test_dis' in verbose mode
test test_dis crashed -- Traceback (most recent call last):
  File /opt/python/3.x.langa-ubuntu/build/Lib/test/regrtest.py, line 1294, in 
runtest_inner
the_module = importlib.import_module(abstest)
  File /opt/python/3.x.langa-ubuntu/build/Lib/importlib/__init__.py, line 92, 
in import_module
return _bootstrap._gcd_import(name[level:], package, level)
  File frozen importlib._bootstrap, line 1603, in _gcd_import
  File frozen importlib._bootstrap, line 1584, in _find_and_load
  File frozen importlib._bootstrap, line 1551, in _find_and_load_unlocked
  File frozen importlib._bootstrap, line 591, in _check_name_wrapper
  File frozen importlib._bootstrap, line 1053, in load_module
  File frozen importlib._bootstrap, line 1034, in load_module
  File frozen importlib._bootstrap, line 567, in module_for_loader_wrapper
  File frozen importlib._bootstrap, line 901, in _load_module
  File frozen importlib._bootstrap, line 297, in _call_with_frames_removed
  File /opt/python/3.x.langa-ubuntu/build/Lib/test/test_dis.py, line 4, in 
module
from test.bytecode_helper import BytecodeTestCase
ImportError: No module named 'test.bytecode_helper'

--
nosy: +neologix
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Antoine Pitrou]

Antoine Pitrou added the comment:

Yes, this is bytecode_helper hasn't been added to the repository.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Antoine Pitrou]

Antoine Pitrou added the comment:

(this is *because*, sorry)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[STINNER Victor]

STINNER Victor added the comment:

Ping! The test is still failing.

--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Thomas Kluyver]

Thomas Kluyver added the comment:

bytecode_helper is there in dis_api3.diff - anyone with commit rights should be 
able to add it to the repository.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Roundup Robot]

Roundup Robot added the comment:

New changeset 84d1a0e32d3b by Nick Coghlan in branch 'default':
Issue #11816: Add missing test helper
http://hg.python.org/cpython/rev/84d1a0e32d3b

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Thomas Kluyver]

Thomas Kluyver added the comment:

I've added docs and tests, and split the changes to test_peepholer into a 
separate patch.

I haven't re-exposed details of the code object as attributes of Bytecode 
instances, because they're already available as e.g. bytecode.codeobj.co_names 
. I think it would be more confusing than useful to offer the same values in 
two places, though I'm open to discussion on this.

I've re-organised the dis module docs a bit. I've put Bytecode at the top, as I 
think it's a more intuitive API than the functions, which have somewhat 
counter-intuitive names due to the module's history.

--
Added file: http://bugs.python.org/file29694/dis_api3.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Thomas Kluyver]

Changes by Thomas Kluyver tak...@gmail.com:


Added file: http://bugs.python.org/file29695/test_peepholer.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Thomas Kluyver]

Thomas Kluyver added the comment:

Updated version of the patch.

Changed from review:
- Included test.bytecode_helper module used by some tests
- Updated docs to indicate that the changes are new in 3.4
- ByteCode - Bytecode
- Added meaningful repr for Bytecode

Still to do:
- ? Re-expose attributes from code object on Bytecode instance
- Tests  documentation for Bytecode class
- Split changes into multiple patches

--
Added file: http://bugs.python.org/file29114/dis_api2.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Thomas Kluyver]

Thomas Kluyver added the comment:

I've updated Nick's patch so that test_dis and test_peephole pass again, and 
added a prototype ByteCode class (without any docs or tests for now, to allow 
for API discussion).

The prototype ByteCode is instantiated with any of the objects that 
get_instructions already accepts (functions, methods, code strings  code 
objects). Iterating over it yields Instruction objects. It has info(), 
show_info() and display_code() methods, which correspond to the code_info(), 
show_code() and disassemble() functions.

I've tried to go for names that make sense, rather than names that fit the 
existing pattern, because the existing pattern feels a bit messy. E.g. the 
show_code() function doesn't actually show the code, so I've called its method 
equivalent show_info().

--
nosy: +takluyver
Added file: http://bugs.python.org/file29051/dis_api.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan added the comment:

Thanks Thomas! It's a promising start - a few more detailed comments in the 
patch review.

I like the idea of creating the initial version as an object-oriented wrapper 
around the existing APIs, rather than completely refactoring the module to make 
everything else a functional wrapper around an underlying object-oriented 
implementation.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan added the comment:

To clarify the vague allusion in my last comment, Ron's suggestion was along 
the lines of creating a dis.Bytecode object that encapsulated everything the 
dis module can figure out about a piece of compiled code.

That would mean exposing the kind of info reported in a string by 
dis.code_info() as attributes/properties, and have the proposed get_opinfo() 
be the __iter__ method on the disassembled Bytecode objects.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Yury Selivanov]

Changes by Yury Selivanov yselivanov...@gmail.com:


--
nosy: +yselivanov

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Given the imminent 3.3 beta 1 feature freeze and the fact I would like to 
explore Ron's suggestion of a higher level ByteCode object to encapsulate a 
sequence of instructions (along with additional information from the code 
object), postponing this one.

--
assignee: ncoghlan - 
versions: +Python 3.4 -Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


--
hgrepos:  -93

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


--
hgrepos: +94

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Added file: http://bugs.python.org/file23853/5ce60675e572.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

MvL pointed out I hadn't updated the Hg repo reference when I moved my sandbox 
over to BitBucket - the diff it was generating was from the last time I updated 
my pydotorg sandbox in order to try something on the buildbots.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Ron Adam]

Ron Adam ron3...@gmail.com added the comment:

Instead of a get_instructions() function, How about using a DisCode class that 
defines the API for accessing Opinfo tuples of a disassembled object.

So instead of...

for instr in dis.bytecode_instructions(thing):
process(instr)

You could use...

for instr in dis.DisCode(thing):
process(instr)

And I would like to be able to do...

assertEqual(DisCode(thing1), DisCode(thing2))


It could also have a .dis() method that returns formatted output that matches 
what dis() currently prints.  That would allow tests that use dis.dis() to get 
rid of capturing stdout with minimal changes.

 result = DisCode(func).dis()  # return a dis compatible string.

A DisCode object also offers a nice place to put documentation for the various 
pieces and an overall view of the new API without getting it confused with the 
current dis.dis() API.

It's easier for me to remember an object with methods than several separate but 
related functions. (YMMV)

This is very near a version of dis I did a while back where I removed the 
prints and returned a list of list object from dis.dis().  The returned object 
had __repr__ that formatted the data so it matched the current dis output.  
That made it work the same in a python shell. But I could still access and 
alter individual lines and fields by indexing or iterating it.  While it worked 
nicely, it wouldn't be backwards compatible.

--
nosy: +ron_adam

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Removed file: 
http://bugs.python.org/file23569/issue11816_get_opinfo_branch_20111031.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


--
hgrepos:  -17

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Removed file: http://bugs.python.org/file23773/9512712044a6.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


--
hgrepos: +93

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Added file: http://bugs.python.org/file23848/9512712044a6.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

OK, there's something crazy going on with Create Patch failing to pick up the 
latest changes. I've removed all the obsolete patches, and am doing another 
merge from default to see if I can get it to pick things up correctly.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Removed file: http://bugs.python.org/file23848/9512712044a6.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Grr, Create Patch insists on trying to produce a patch based on 
https://bitbucket.org/ncoghlan/cpython_sandbox/changesets/9512712044a6.

That checkin is from *September* and ignores all my recent changes :P

Relevant meta-tracker issue: 
http://psf.upfronthosting.co.za/roundup/meta/issue429

Manual patch upload coming shortly...

--
nosy: +loewis

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

OK, manual up-to-date patch attached.

--
Added file: 
http://bugs.python.org/file23849/issue11816_get_opinfo_branch_20111204.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

@Ron: Now that it has a reasonably clear signature, I could see my way clear to 
making the Instruction._disassemble() method public, which makes it easy for 
people to compose their own disassembly output.

For all the other display methods, I prefer Ryan Kelly's suggestion of 
supporting a file argument which is then passed through to the underlying 
print() calls.

This is inconsistent with what I originally did for the code_info() APIs (where 
I made a separate give me the string function), but it's the lowest impact 
change that avoids the need to capture stdout.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Removed file: http://bugs.python.org/file23773/9512712044a6.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Added file: http://bugs.python.org/file23773/9512712044a6.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

I have updated my BitBucket repo with the following changes:

- the main API is now dis.get_instructions()
- the info class is now dis.Instruction
- get_instructions() accepts a 'line_offset' argument that is added to any 
source code line numbers
- the printing of individual lines in all the disassembly operations is now 
handled by a private Instruction._disassemble() helper method
- assertBytecodeExactlyMatches now relies on the list comparison machinery in 
unittest, as well as the Instruction.__eq__ implementation provide by 
collections.namedtuple (this gives *much* better error messages when 
discrepancies are found)
- to further improve error messages reported by unittest (which can truncate 
representations), the order of the Instruction fields now has 'opname' first, 
then 'opcode', then the remaining fields.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Somewhat inevitably, the detailed disassembler tests broke when Antoine updated 
the code generation for function definitions (as part of PEP 3155). (At least, 
the tests broke, and PEP 3155 seems the most likely culprit).

I subsequently realised there's a potential opportunity here: if Instruction 
objects define an __eq__ method instead of relying on the custom 
assertBytecodeExactlyMatches helper method the new disassembly tests 
currently use, then the sequence diffing functionality in unittest could be 
very helpful in identifying and fixing discrepancies between actual output and 
expected output.

However, the assertBytecodeExactlyMatches() API has this concept of a 
line_offset which it uses to cope with some of the introspection fodder 
moving around in the files, so the question then becomes how to deal with that 
in the context of an __eq__ implementation.

Accordingly, my current plan is to offer line_offset as a keyword-only 
argument to get_instructions() itself. That way, the entire sequence of 
generated bytecode can easily be adjusted to (for example), be based around the 
first line being line 1, even if the actual code object being disassembled is 
located elsewhere in the original source file.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Added file: http://bugs.python.org/file23773/9512712044a6.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Meador's suggested name change has grown on me, so I plan to switch the name of 
the new API to get_instructions() and the new class to Instruction.

--
assignee: rhettinger - ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Bitbucket repo and attached patch updated relative to current tip.

--
Added file: 
http://bugs.python.org/file23569/issue11816_get_opinfo_branch_20111031.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Removed file: 
http://bugs.python.org/file23197/issue11816_get_opinfo_branch_20110920.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Removed file: 
http://bugs.python.org/file23095/issue11816_get_opinfo_branch_20110904.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Removed file: 
http://bugs.python.org/file23019/issue11816_get_opinfo_branch_20110824.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Removed file: http://bugs.python.org/file21599/dis.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

I agree that 'bytecode_instructions' is a long-winded.  FWIW, I
have worked on or with a fair amount instruction level things and
instruction or instr seem to be the established domain terminology.

Here are a few examples:

* Java ASM - http://asm.ow2.org/asm33/javadoc/user/index.html
   - Classes named 'InsnList', 'InsnNode'
* PyPy - 
https://bitbucket.org/pypy/pypy/src/default/pypy/interpreter/astcompiler/assemble.py
   - class named 'Instruction'
* Python - http://hg.python.org/cpython/file/a06ef7ab7321/Python/compile.c#l45
   - struct named 'instr'
* binutils - http://sourceware.org/git/?p=binutils.git;a=tree
   - structs, variables, etc ... with use of 'insn' and 'instruction'
* gcc - http://gcc.gnu.org/git/?p=gcc.git;a=tree
   - structs, variables, etc ... with use of 'insn' and 'instruction'

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

I took a quick look over the final patch (I will do a more thorough
review later).  I like the general idea a lot.  The first thing that
popped out at me are the names 'OpInfo' and 'get_opinfo'.

'OpInfo' makes it sound like information concerning only the opcode, but
these objects really represent bytecode instructions.  I see a lot
of code in the future like:

for opinfo in dis.get_opinfo(thing):
process(opinfo)

which seems vague.  The following seems clearer to me:

for instr in dis.bytecode_instructions(thing):
process(instr)

And instead of 'OpInfo' perhaps 'ByteCodeInstruction'.  Even the current
'dis' documentation uses the terminology Byte Code Instruction.

--
nosy: +meador.inge

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

'Op' is just an abbreviation of 'operation'. So 'operation code' becomes 
'opcode' and 'operation information' becomes 'opinfo'. The fact that it comes 
for the 'dis' module gives the context that the *kind* of operation we're 
talking about is a Python byte code instruction.

When people are hacking on bytecode in the future, they'll likely end up using 
get_opinfo() a fair bit, so swapping the succinct 'opinfo' for the verbose 
'bytecode_instruction' strikes me as a poor trade-off.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Attached patch should now be complete, including the documentation for the new 
keyword-only 'file' parameter on various dis module functions.

--
Added file: 
http://bugs.python.org/file23197/issue11816_get_opinfo_branch_20110920.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Meador Inge]

Changes by Meador Inge mead...@gmail.com:


--
stage:  - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Regenerated the get_opinfo patch against current 3.3 tip.

Still haven't fixed the missing doc updates mentioned in my last message, 
though.

--
Added file: 
http://bugs.python.org/file23095/issue11816_get_opinfo_branch_20110904.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Raymond Hettinger]

Raymond Hettinger raymond.hettin...@gmail.com added the comment:

Nick, I still want to work on this one.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Eugene Toder]

Changes by Eugene Toder elto...@gmail.com:


--
nosy:  -eltoder

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Added file: http://bugs.python.org/file23018/7cead6bf5832.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


Removed file: http://bugs.python.org/file23018/7cead6bf5832.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

The diff generator didn't work - I've uploaded the current patch manually to 
make it easier to review than it is in my bitbucket repo.

I just noticed there's a missing element in the docs patch at the moment - to 
make testing easier, Ryan added a 'file' argument to the various print-based 
dis functions so the output can easily be captured in a StringIO object. The 
docs updates don't currently reflect that, they only cover the OpInfo and 
get_opinfo additions (along with a clarification of the dis module's slightly 
odd use of the term 'free').

Aside from that, the core concept of the patch is pretty simple:
- add dis.OpInfo and dis.get_opinfo() to make it easier to walk the bytecode 
programmatically
- eliminate a lot of the logic duplication inside dis by refactoring more of 
the operations to internally rely on get_opinfo()
- add a new test.bytecode_helper.BytecodeTestCase with some convenient 
assertions for checking code generation
- update test_peepholer to be independent of the disassembly formatting details
- add tests for the new features to test_dis (but keep the old detailed 
formatting tests)

One potential criticism is the complexity of the 'expected output' for the new 
OpInfoTestCase, but it seemed worth it to vet the way the new code handles 
several cases. The programmatic nature makes the opcode sequences much easier 
to read and maintain than the corresponding formatted output tests would have 
been.

These new tests also cover an error that the previous incarnation of the test 
suite missed completely (I had a bug at one point where I had incorrectly 
omitted the second half of the list of cell names - there was no test to check 
that the disassembler handled references to such names correctly)

--
assignee: ncoghlan - rhettinger
Added file: 
http://bugs.python.org/file23019/issue11816_get_opinfo_branch_20110824.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Ryan Kelly]

Changes by Ryan Kelly r...@rfk.id.au:


--
nosy: +rfk

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


--
assignee: rhettinger - ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Jesús Cea Avión]

Changes by Jesús Cea Avión j...@jcea.es:


--
nosy: +jcea

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Jesús Cea Avión]

Jesús Cea Avión j...@jcea.es added the comment:

Do not forget to update docs too.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Alexander Belopolsky]

Changes by Alexander Belopolsky belopol...@users.sourceforge.net:


--
nosy: +belopolsky

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Changed issue title to cover ideas like get_opinfo().

--
title: Add functions to return disassembly as string - Refactor the dis module 
to provide better building blocks for bytecode analysis

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Oops, I forgot to edit my comment to match the OpInfo definition I used in the 
proof-of-concept:

OpInfo = collections.namedtuple(OpInfo,
opindex opcode opname oparg details starts_line is_jump_target)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Eugene Toder]

Eugene Toder elto...@gmail.com added the comment:

So in the near term, dis-based tests should continue to copy/paste sys.stdout 
redirection code?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

If we decide our long term goal is the use of the opcode stream for 
programmatic access, then yes.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Alex Gaynor]

Alex Gaynor alex.gay...@gmail.com added the comment:

FWIW in PyPy we have 
https://bitbucket.org/pypy/pypy/src/default/lib_pypy/disassembler.py which we 
use for some of our tools.

--
nosy: +alex

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11816] Refactor the dis module to provide better building blocks for bytecode analysis

[Raymond Hettinger]

Changes by Raymond Hettinger raymond.hettin...@gmail.com:


--
assignee:  - rhettinger

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11816
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com