[issue25843] lambdas on the same line may incorrectly share code objects

Changes by Mark Dickinson : -- nosy: +mark.dickinson ___ Python tracker ___ ___

[issue25843] lambdas on the same line may incorrectly share code objects

STINNER Victor added the comment: Let me try to explain this issue again. "f1, f2 = lambda: 1, lambda: 1.0" is compiled to two MAKE_FUNCTION instructions, MAKE_FUNCTION takes a code object as parameter (and a name). The Python compiler merges constants which are seen as "equal", with

[issue25843] lambdas on the same line may incorrectly share code objects

STINNER Victor added the comment: FYI this issue is linked to the issue #26146 which allows to emit constants from an AST optimizer (see also the PEP 511). -- ___ Python tracker

[issue25843] lambdas on the same line may incorrectly share code objects

Changes by Mark Lawrence : -- nosy: -BreamoreBoy ___ Python tracker ___ ___

[issue25843] lambdas on the same line may incorrectly share code objects

Mark Lawrence added the comment: http://code.activestate.com/recipes/578353-code-to-source-and-back/ compares code objects as part of round trip testing. -- nosy: +BreamoreBoy ___ Python tracker

[issue25843] lambdas on the same line may incorrectly share code objects

Emanuel Barry added the comment: I see. Is there any to special-case those so that only closures use that? Maybe by checking on the function object itself - the function itself would be quite similar, as well. @Mark - I think you've misunderstood me (others did too, so I'm going to assume I

[issue25843] lambdas on the same line may incorrectly share code objects

Raymond Hettinger added the comment: [Emanuel Barry] > In which circumstances does comparing two code objects > (at function creation time, what's more) make any sense? It makes closures efficient: >>> def f(x): def g(y): return x + y return g

[issue25843] lambdas on the same line may incorrectly share code objects

Serhiy Storchaka added the comment: > It makes closures efficient: No, code objects are not compared here. The code object is constant, and f() returns different closure objects using the same code object. AFAIK the comparison of different code objects is used exclusively in testing.

[issue25843] lambdas on the same line may incorrectly share code objects

R. David Murray added the comment: Ah, haypo explained the code to me, and now I understand your comment Serhiy. So, the replacement happens because of (a) an optimizer general rule and (b) the fact that code object *can* be compared. So it sounds like Armin's suggestion of making an

[issue25843] lambdas on the same line may incorrectly share code objects

Serhiy Storchaka added the comment: Indeed. My answer actually is an answer to implicit question: could we make different code objects be always non-equal? Answer: no, because we use the comparison of different code objects to test compiler and marshaller. May be we can get rid of code

[issue25843] lambdas on the same line may incorrectly share code objects

STINNER Victor added the comment: compiler_add_o() uses an heuristic to compare and merge duplicated constants. It has special cases for float and complex numbers, but it's not designed to handle more types. Funny, I had the same isue last week why I added support for tuple and frozenset

[issue25843] lambdas on the same line may incorrectly share code objects

STINNER Victor added the comment: code_richcompare.patch: fix code_richcompare() to not consider that constants (1,) and (1.0,) are equal. The patch lacks an unit test. We may use the new _PyCode_ConstantKey() function to compare other code attributes? -- keywords: +patch Added

[issue25843] lambdas on the same line may incorrectly share code objects

STINNER Victor added the comment: > Would option (1) work if wrap 1 and 1.0 in a tuple? In a list? In a custom > collection? Right now, the Python compiler is quite limited. It only produces constants for integers and strings, not complex types. The peephole optimizers is responsible to

[issue25843] lambdas on the same line may incorrectly share code objects

STINNER Victor added the comment: > The frozenset are different are expected. Oh wait, in this example, it works, but not because of the expected reason. frozenset({1}) is equal to frozenset({1.0}) and code_richcompare() doesn't create a special key to make them different. The example only

[issue25843] lambdas on the same line may incorrectly share code objects

STINNER Victor added the comment: code_eq.py: code to test that frozenset() are inequal. The code hacks constants, don't run the code with new constants or it will crash! -- Added file: http://bugs.python.org/file41309/code_eq.py ___ Python tracker

[issue25843] lambdas on the same line may incorrectly share code objects

R. David Murray added the comment: Serhiy: that can't be entirely true, since we have here a bug where two lambdas on the same line are getting the same code object incorrectly, and that is not a testing context. -- ___ Python tracker

[issue25843] lambdas on the same line may incorrectly share code objects

Serhiy Storchaka added the comment: Would option (1) work if wrap 1 and 1.0 in a tuple? In a list? In a custom collection? -- ___ Python tracker ___

[issue25843] lambdas on the same line may incorrectly share code objects

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___

[issue25843] lambdas on the same line may incorrectly share code objects

STINNER Victor added the comment: code_richcompare-2.patch: Updated patch to handle also frozenset, the other constant type generated by the peephole optimizer. -- Added file: http://bugs.python.org/file41308/code_richcompare-2.patch ___ Python

[issue25843] lambdas on the same line may incorrectly share code objects

Serhiy Storchaka added the comment: If go this way, I would add explicit support of all types supported in marshal, and include id(obj) into the key of all other types. -- ___ Python tracker

[issue25843] lambdas on the same line may incorrectly share code objects

Serhiy Storchaka added the comment: > The lowest impact fix from a code change would be to add a type equivalence > check for constants as Raymond first suggested, as that only involves adding > an extra check to code_richcompare: >

[issue25843] lambdas on the same line may incorrectly share code objects

Serhiy Storchaka added the comment: > I think we should focus on fixing the spec for code object equivalents. > Perhaps the test can be simplified to use (co_firstlineno, co_firstrowno, > co_filename). This is not enough if the code was compiled from a string. >>> x = eval('lambda: 1') >>>

[issue25843] lambdas on the same line may incorrectly share code objects

David MacIver added the comment: Note that 3.x does not correctly handle -0.0, you just have to work a bit harder: >>> (lambda: (-0.0, 0.0), lambda: (0.0, -0.0))[1]() (-0.0, 0.0) -- nosy: +David MacIver ___ Python tracker

[issue25843] lambdas on the same line may incorrectly share code objects

Emanuel Barry added the comment: Nobody seems to have asked this, so I'll be that guy. In which circumstances does comparing two code objects (at function creation time, what's more) make any sense? I mean, I'm fine with being able to compare two code objects, but I don't think that's

[issue25843] lambdas on the same line may incorrectly share code objects

Armin Rigo added the comment: That's what I suggested ("compile.c:compiler_addop(): special-case code objects too, and stick their identity in the tuple 't'.") and how I fixed the same bug in PyPy (https://bitbucket.org/pypy/pypy/commits/7ec3e1d02197). --

[issue25843] lambdas on the same line may incorrectly share code objects

Serhiy Storchaka added the comment: I suppose this is needed above all for testing. test_compile, test_codeop, test_marshal, and few other tests compare code objects. -- ___ Python tracker

[issue25843] lambdas on the same line may incorrectly share code objects

Emanuel Barry added the comment: I'm not suggesting to get rid of the rich compare ability of code objects, which makes sense to me. What doesn't make sense to me, however, is when a function's code object is replaced by another one because it compares equal. I see no use case for this, and

[issue25843] lambdas on the same line may incorrectly share code objects

Armin Rigo added the comment: Other possible minimal fixes: * compile.c:compiler_addop(): special-case code objects too, and stick their identity in the tuple 't'. * or, in compile.c:makecode(), append the first row number to co_consts. -- ___

[issue25843] lambdas on the same line may incorrectly share code objects

Raymond Hettinger added the comment: FWIW, the bug is present in PyPy as well. -- ___ Python tracker ___ ___

[issue25843] lambdas on the same line may incorrectly share code objects

Nick Coghlan added the comment: >From what I can see: * checking constant types in addition to their values should be a two line change (plus tests) * checking the column number in addition to the line number would be a more comprehensive fix, but also a constructor API change (since

[issue25843] lambdas on the same line may incorrectly share code objects

Changes by Ismail Donmez : -- nosy: +donmez ___ Python tracker ___ ___ Python-bugs-list

[issue25843] lambdas on the same line may incorrectly share code objects

Changes by Ezio Melotti : -- nosy: +ezio.melotti ___ Python tracker ___ ___

[issue25843] lambdas on the same line may incorrectly share code objects

Raymond Hettinger added the comment: Interestingly, co_filename is not used as part of the equivalence criteria, so code object equivalence can be fooled across multiple input files. Fortunately in this case, the false equivalence isn't relied on by the code generator. $ cat a.py def

[issue25843] lambdas on the same line may incorrectly share code objects

Changes by Torsten Landschoff : -- nosy: +torsten ___ Python tracker ___ ___

[issue25843] lambdas on the same line may incorrectly share code objects

Raymond Hettinger added the comment: The equality of code objects is determined by the code_richcompare() logic in Objects/codeobject.c. Two code objects are equal if all of their attributes compare equal. That includes co_name, co_argcount, co_kwonlyargcount, co_nlocals, co_flags,

[issue25843] lambdas on the same line may incorrectly share code objects

Raymond Hettinger added the comment: Here's another variant (courtesy of Nick Coghlan): python3.5 -c "seq1 = [1.0 for x in range(5)]; seq2 = [True for x in range(5)]; print(seq1); print(seq2)" -- ___ Python tracker

[issue25843] lambdas on the same line may incorrectly share code objects

Kevin Shweh added the comment: A type-based check runs into problems with 0.0 vs -0.0. For example, on Python 2.7.11: >>> x, y = lambda: 0.0, lambda: -0.0 >>> y() 0.0 I wasn't able to reproduce the -0.0 problem with Python 3.4 on Ideone; y.__code__.co_consts seems to have an unused 0.0 in it

[issue25843] lambdas on the same line may incorrectly share code objects

Raymond Hettinger added the comment: One possible solution for all these variants is to let code objects track both the co.firstlineno and co.firstrowno. -- ___ Python tracker

[issue25843] lambdas on the same line may incorrectly share code objects

R. David Murray added the comment: Thanks for posting the bug. I added the aside because this is the third or fourth of these reference-only things I've seen in the past couple weeks and I finally figured out what bothered me about them. We should put something about this in the devguide on