[issue15368] bytecode generation is not deterministic
Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +eric.snow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Brett Cannon br...@python.org added the comment: You ran the benchmarks correctly, but I was more worried about compilation time than execution time (changing which index is for what really shouldn't make a difference in performance, and the benchmarks show that). If you want to test using the importlib benchmarks, run ``./python.exe -m importlib.test.benchmark`` and compare the output (run with -h to see all the options). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Meador Inge mead...@gmail.com added the comment: Oh, cool -- I didn't know about the importlib benchmarks. Thanks Brett. Here are the results (OS X 10.7.4, Dual 1.7GHz Core i5, 4 GB RAM): $ ./python.exe -m importlib.test.benchmark Measuring imports/second over 1 second, best out of 3 Entire benchmark run should take about 33 seconds Using function __import__ at 0x1063bdd40 as __import__ sys.modules [ 41831 41812 40914 ] best is 41,831 Built-in module [ 20327 20441 19775 ] best is 20,441 Source writing bytecode: small [ 2108 2119 2117 ] best is 2,119 Source w/o bytecode: small [ 5093 5075 5106 ] best is 5,106 Source w/ bytecode: small [ 6021 5915 5997 ] best is 6,021 Source writing bytecode: tabnanny [ 379 379 379 ] best is 379 Source w/o bytecode: tabnanny [ 448 453 456 ] best is 456 Source w/ bytecode: tabnanny [ 2270 2279 2279 ] best is 2,279 Source writing bytecode: decimal [ 28 27 28 ] best is 28 Source w/o bytecode: decimal [ 30 30 30 ] best is 30 Source w/ bytecode: decimal [ 256 254 259 ] best is 259 $ ./python-with-sorting.exe -m importlib.test.benchmark Measuring imports/second over 1 second, best out of 3 Entire benchmark run should take about 33 seconds Using function __import__ at 0x104b01d40 as __import__ sys.modules [ 42369 42203 42348 ] best is 42,369 Built-in module [ 17060 16358 20486 ] best is 20,486 Source writing bytecode: small [ 2118 2112 2127 ] best is 2,127 Source w/o bytecode: small [ 5121 5098 5125 ] best is 5,125 Source w/ bytecode: small [ 6015 5652 6046 ] best is 6,046 Source writing bytecode: tabnanny [ 367 368 366 ] best is 368 Source w/o bytecode: tabnanny [ 439 429 439 ] best is 439 Source w/ bytecode: tabnanny [ 2267 2274 2277 ] best is 2,277 Source writing bytecode: decimal [ 27 27 27 ] best is 27 Source w/o bytecode: decimal [ 28 28 28 ] best is 28 Source w/ bytecode: decimal [ 257 259 259 ] best is 259 Pretty similar results either way. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Brett Cannon br...@python.org added the comment: Yep, so I consider the performance worry dealt with. -- stage: needs patch - commit review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Nick Coghlan ncogh...@gmail.com added the comment: Agreed, so definite +1 from me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Brett Cannon br...@python.org added the comment: So I think that just leaves one other person to verify the patch works as expected and then commit it. I can hopefully later this week, but no sooner than Friday, so if someone can get to it faster that would be great. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Meador Inge mead...@gmail.com added the comment: I can commit today unless you think we need someone else to independently verify the patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Meador Inge mead...@gmail.com added the comment: BTW, I think this should be committed on 2.7 and 3.2 as well since the same issue can happen when throwing -R. Any objections? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Brett Cannon br...@python.org added the comment: Nah, you are a committer and thus qualify as the second review. =) As for backporting, it only affects regeneration of bytecode, so I don't see any major harm in it (might surprise some people, but there is a legitimate reason for fixing this). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 7eb0180c1734 by Meador Inge in branch '2.7': Issue #15368: make bytecode generation deterministic. http://hg.python.org/cpython/rev/7eb0180c1734 New changeset 79d54fba49b3 by Meador Inge in branch '3.2': Issue #15368: make bytecode generation deterministic. http://hg.python.org/cpython/rev/79d54fba49b3 New changeset 578066372485 by Meador Inge in branch 'default': Issue #15368: make bytecode generation deterministic. http://hg.python.org/cpython/rev/578066372485 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Meador Inge mead...@gmail.com added the comment: Committed. Thanks for the reviews y'all. -- resolution: - fixed stage: commit review - committed/rejected status: open - closed versions: +Python 2.7, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Serhiy Storchaka storch...@gmail.com added the comment: Meador, you have forgotten to fix the error PyList_GET_SIZE(src). src is not list, should be PyList_GET_SIZE(sorted_keys). -- nosy: +storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Meador Inge mead...@gmail.com added the comment: D'oh! Thanks for catching that Serhiy! Fixing now. Sorry about that. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 47e074f6ca26 by Meador Inge in branch '2.7': Issue #15368: fixing variable typo. http://hg.python.org/cpython/rev/47e074f6ca26 New changeset 1c46f2ede4cb by Meador Inge in branch '3.2': Issue #15368: fixing variable typo. http://hg.python.org/cpython/rev/1c46f2ede4cb New changeset 6502de91610d by Meador Inge in branch 'default': Issue #15368: fixing variable typo. http://hg.python.org/cpython/rev/6502de91610d -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Meador Inge mead...@gmail.com added the comment: New patch that addresses feedback from Brett's code review (thanks!). -- Added file: http://bugs.python.org/file26418/issue15368-v1.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
New submission from Meador Inge mead...@gmail.com: Consider this small example (you might have to run sample program multiple times to see a difference): $ cat dis-closure.py import dis def adder(a, b): def add(): return a + b return add print(dis.dis(adder(1, 2).__code__)) $ ./python.exe dis-closure.py 5 0 LOAD_DEREF 0 (a) 3 LOAD_DEREF 1 (b) 6 BINARY_ADD 7 RETURN_VALUE None $ ./python.exe dis-closure.py 5 0 LOAD_DEREF 1 (a) 3 LOAD_DEREF 0 (b) 6 BINARY_ADD 7 RETURN_VALUE None The order of 'co_cellvars' and 'co_freevars' can be different from compile to compile, thus the bytecode can be different from compile to compile. This is due to the fact that these variable sets are managed with hashes and the ordering may come out different when the names in the hashes are given indexes (via 'dictbytype' in 'compile.c'). I am not sure if these are the only areas that causes bytecode generation to be non-deterministic. I found this behavior surprising. -- components: Interpreter Core messages: 165596 nosy: meador.inge priority: normal severity: normal stage: needs patch status: open title: bytecode generation is not deterministic type: behavior versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Brett Cannon br...@python.org added the comment: I might come to regret asking this, but so what? Is this actually causing you issues, or are you literally just finding this behavior surprising and that's it? I mean we could simply sort the tuples, but I don't know what kind of performance hit that would cause. -- nosy: +brett.cannon ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Meador Inge mead...@gmail.com added the comment: On Mon, Jul 16, 2012 at 8:11 AM, Brett Cannon rep...@bugs.python.org wrote: I might come to regret asking this, but so what? Is this actually causing you issues, or are you literally just finding this behavior surprising and that's it? I noticed it in the context of working on issue15352. There are still a few issue in the build system where 'importlib.h' isn't regenerated when it needs to be (e.g. when the marshaling code is changed). When investigating that issue I noticed that trivial changes (e.g. `touch _bootstrap.py`) can cause differences in 'importlib.h'. Those differences are mainly due to the cell and free var indexes being reordered. Since the bytecode generation isn't deterministic, there can be some noisy diffs in your working copy when mucking around with the files related to the frozen importlib. Thus if we ever wanted to just always generate importlib.h because we can't be sure which changed source files should cause importlib.h to be regenerated, then the non-determinism would be a real annoyance. That is the only concrete issue I currently am aware of. This issue may not actually be worth addressing, but I felt it was worthy of discussion. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Changes by Mark Dickinson dicki...@gmail.com: -- nosy: +mark.dickinson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Changes by Florent Xicluna florent.xicl...@gmail.com: -- nosy: +flox ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Nick Coghlan ncogh...@gmail.com added the comment: At least we know the hash randomisation is working :) Spurious changes when freezing modules seems like a legitimate reason to fix it - the import benchmarks would probably give the compiler enough of a workout to highlight if the sorting is excessively time consuming. -- nosy: +ncoghlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Antoine Pitrou pit...@free.fr added the comment: Ditto. I think predictability of bytecode generation is useful, e.g. for make-like tools that examine content, or for unit testing. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Brett Cannon br...@python.org added the comment: OK, so it sounds like we need to do the equivalent of sorting those tuples when generating the bytecode. That would suggest that probably need to tweak Python/compile.c to make it deterministic. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15368] bytecode generation is not deterministic
Meador Inge mead...@gmail.com added the comment: I haven't done any benchmarking (yet), but here is a patch implementing the basic sorting approach. -- keywords: +patch Added file: http://bugs.python.org/file26409/issue15368-v0.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15368 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
