[issue29514] Add a test case that prevents magic number changes in minor releases

Ned Deily added the comment: New changeset 828351716ec436bfa174cfdb33def7ee7c0d2e63 by Ned Deily in branch '3.7': bpo-29514: Make magic number test work for candidates https://github.com/python/cpython/commit/828351716ec436bfa174cfdb33def7ee7c0d2e63 --

[issue29514] Add a test case that prevents magic number changes in minor releases

Ned Deily added the comment: New changeset f79386f669c22199e3ce99cfba2e5945c73beb47 by Ned Deily in branch '2.7': bpo-29514: Make magic number test work for candidates (GH-7946) https://github.com/python/cpython/commit/f79386f669c22199e3ce99cfba2e5945c73beb47 --

[issue29514] Add a test case that prevents magic number changes in minor releases

Ned Deily added the comment: New changeset 57389dbaa897fadc97de78af67d5e555e28be60b by Ned Deily in branch '3.6': bpo-29514: Make magic number test work for candidates (GH-7945) https://github.com/python/cpython/commit/57389dbaa897fadc97de78af67d5e555e28be60b --

[issue29514] Add a test case that prevents magic number changes in minor releases

Change by Ned Deily : -- pull_requests: +7555 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29514] Add a test case that prevents magic number changes in minor releases

Change by Ned Deily : -- pull_requests: +7554 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue29514] Add a test case that prevents magic number changes in minor releases

Ned Deily added the comment: New changeset 8511b448a55929f15aa5730c9e30cbb654978ea4 by Ned Deily in branch '3.7': bpo-29514: Make magic number test work for candidates https://github.com/python/cpython/commit/8511b448a55929f15aa5730c9e30cbb654978ea4 --

[issue29514] Add a test case that prevents magic number changes in minor releases

Ned Deily added the comment: New changeset e7d3cccd8e0da93a15ca80f4b6aac2ca3435fd06 by Ned Deily in branch 'master': bpo-29514: Make magic number test work for candidates https://github.com/python/cpython/commit/e7d3cccd8e0da93a15ca80f4b6aac2ca3435fd06 -- nosy: +ned.deily

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: Serhiy: I believe that magic number was current when the test was written, and it hasn't been getting updated since the test is currently being skipped. Given how we designed the skip logic, I think that counts as "working as intended" - we don't really care

[issue29514] Add a test case that prevents magic number changes in minor releases

Serhiy Storchaka added the comment: Is it intentionally that old magic number is used in 3.7? The test will become failing just after reaching the RC stage. -- ___ Python tracker

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: This has now been backported to all active maintenance branches. For 2.7, I backported the test as a standalone one in Lib/test/test_import_magic.py, since there's only a partial importlib implementation in 2.7, and hence no test_importlib directory. Thanks

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: New changeset a51f12f154baef9e10698f43bd19b36d01af62f4 by Nick Coghlan in branch '2.7': [2.7] bpo-29514: Check magic number for bugfix releases https://github.com/python/cpython/commit/a51f12f154baef9e10698f43bd19b36d01af62f4 --

[issue29514] Add a test case that prevents magic number changes in minor releases

Changes by Nick Coghlan : -- pull_requests: +2211 ___ Python tracker ___ ___

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: New changeset 92e9e35292d4e2044c25970389f3cdb30cfc903a by Nick Coghlan in branch '3.6': [3.6] bpo-29514: Check magic number for bugfix releases (#2157) https://github.com/python/cpython/commit/92e9e35292d4e2044c25970389f3cdb30cfc903a --

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: New changeset d071a20f136547a490783fcfa969c6528025a6d2 by Nick Coghlan in branch '3.5': [3.5] bpo-29514: Check magic number for bugfix releases (#2158) https://github.com/python/cpython/commit/d071a20f136547a490783fcfa969c6528025a6d2 --

[issue29514] Add a test case that prevents magic number changes in minor releases

Changes by Nick Coghlan : -- pull_requests: +2209 ___ Python tracker ___ ___

[issue29514] Add a test case that prevents magic number changes in minor releases

Changes by Nick Coghlan : -- pull_requests: +2208 ___ Python tracker ___ ___

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: We don't use the alpha and beta phases for maintenance releases, so I think the checks are functionally equivalent. For example, the 3.6 branch currently reports itself as "3.6.1 final", will temporarily report itself as "3.6.2 release" during the rc period,

[issue29514] Add a test case that prevents magic number changes in minor releases

Eric Snow added the comment: Sorry I didn't see the just-landed PR earlier, but it may make sense to not skip that test under *any* release level for a micro version > 0: @unittest.skipUnless( (sys.version_info.micro > 0 or sys.version_info.releaselevel in ('final',

[issue29514] Add a test case that prevents magic number changes in minor releases

Changes by Mariatta Wijaya : -- stage: patch review -> backport needed ___ Python tracker ___

[issue29514] Add a test case that prevents magic number changes in minor releases

Łukasz Langa added the comment: New changeset d6d344d8330a5975fc102e8f275d47044294f1d1 by Łukasz Langa (Eric Appelt) in branch 'master': bpo-29514: Check magic number for bugfix release (#54) https://github.com/python/cpython/commit/d6d344d8330a5975fc102e8f275d47044294f1d1 --

[issue29514] Add a test case that prevents magic number changes in minor releases

Eric Appelt added the comment: Thank you for the review and suggestions Nick, Serhiy, and all. I believe that I have implemented the suggestions here and on github into a new commit on my pull request which moves the test into an existing module and removes the notion of a table in favor of a

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: I just merged https://github.com/python/cpython/commit/93602e3af70d3b9f98ae2da654b16b3382b68d50 as the fix for issue #29537, so 3.5.4+ should handle legacy bytecode files without any complaints (and redistributors can backport it to 3.5.3 as needed). For

[issue29514] Add a test case that prevents magic number changes in minor releases

Changes by Carson Lam : -- nosy: +Carson Lam ___ Python tracker ___ ___ Python-bugs-list

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: Serhiy posted a patch at http://bugs.python.org/issue29537 that addresses the original problem from http://bugs.python.org/issue27286 in a different way: it simply omits the function name in cases where the stack reference is ambiguous. So what I think we

[issue29514] Add a test case that prevents magic number changes in minor releases

Eric Snow added the comment: > These files *do not* belong to the CPython package, they belong to the > individual Python modules that depend on CPython, so messing with them > when installing a new version of CPython would be a significant breach > of distro policies. Thanks for clarifying,

[issue29514] Add a test case that prevents magic number changes in minor releases

Barry A. Warsaw added the comment: On Feb 12, 2017, at 05:42 PM, Brett Cannon wrote: >That comment is poorly worded. Pretty sure at one time it was accurately worded, but things have changed since PEP 3147 so the comment could use an update. -- ___

[issue29514] Add a test case that prevents magic number changes in minor releases

Petr Viktorin added the comment: In Fedora also want to stay close to upstream: every downstream patch is like an open issue, something we'd like to merge given enough time and people. But, for a problem that: - is fixed in the latest version (3.6) - is getting a test to not happen again -

[issue29514] Add a test case that prevents magic number changes in minor releases

Changes by Iryna : -- nosy: +ishcherb ___ Python tracker ___ ___ Python-bugs-list

[issue29514] Add a test case that prevents magic number changes in minor releases

Mark Dickinson added the comment: > it's a one-time hack intended to be removed and forgotten at the end of 3.5's > lifetime. Fair enough. If necessary, one of our options is to apply that patch or something like it to the Python executables that we distribute for Canopy. But in general,

[issue29514] Add a test case that prevents magic number changes in minor releases

Petr Viktorin added the comment: > perhaps Petr's patch to accept both magic numbers should be submitted > upstream so it's the default in 3.5.4+? I don't think it can be submitted in the current state; it's a one-time hack intended to be removed and forgotten at the end of 3.5's lifetime. I

[issue29514] Add a test case that prevents magic number changes in minor releases

Mark Dickinson added the comment: [Nick] > perhaps Petr's patch to accept both magic numbers should be submitted > upstream so it's the default in 3.5.4+? [...] Enthought's primary buildsystem maintainer is unfortunately out of contact for the week, so I can't speak with any authority on

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: Given some of the other folks encountering problems, perhaps Petr's patch to accept both magic numbers should be submitted upstream so it's the default in 3.5.4+? And then we ask Larry and the other release management folks nicely for an early 3.5.4 release?

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: On 12 Feb 2017 11:54 pm, "Serhiy Storchaka" wrote: Serhiy Storchaka added the comment: I don't understand how the table can make maintaining easier. You need to support multiple values in every branch even if the only one value is used.

[issue29514] Add a test case that prevents magic number changes in minor releases

Mark Dickinson added the comment: [Nick] > "Requires a mass rebuild of all the Python packages in the distro to get > things working properly again" is again a major concern for what's nominally > a low impact maintenance update. With my Enthought developer hat on, I'd like to second this

[issue29514] Add a test case that prevents magic number changes in minor releases

Changes by Mark Dickinson : -- nosy: +mark.dickinson ___ Python tracker ___ ___

[issue29514] Add a test case that prevents magic number changes in minor releases

Łukasz Langa added the comment: I commented on the original issue where the magic number was changed. This broke the world at work for me. Our distribution mechanism for Python programs is zipped bundles of .pyc and .so files, the optimized variant doesn't keep .py files around. So suddenly

[issue29514] Add a test case that prevents magic number changes in minor releases

Serhiy Storchaka added the comment: I don't understand how the table can make maintaining easier. You need to support multiple values in every branch even if the only one value is used. I'm sure unused values will became outdated pretty fast. "alpha" and "beta" stages are not related here.

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: Regarding the naming of the test file: I agree this can just be added to Lib/test/test_importlib/test_util:MagicNumberTests rather than being added as a new test file. -- ___ Python tracker

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: I think the table of expected magic numbers will be easier to maintain than a single value, as otherwise you either have to: - update it every time the magic number changes during the alpha/beta cycle (which would be a bad habit to encourage, since it runs

[issue29514] Add a test case that prevents magic number changes in minor releases

Serhiy Storchaka added the comment: What is the purpose of having a *table* of magic numbers? We need just one value. Different Python versions run different tests. And I don't think this test needs adding separate file. -- ___ Python tracker

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: Thanks Eric! I've left some detailed comments on the PR, with the main one being to include the "expected" table directly in the test case so it doesn't require any changes to importlib itself. For Serhiy - thanks for the explanation of the potential security

[issue29514] Add a test case that prevents magic number changes in minor releases

Serhiy Storchaka added the comment: The patch looks overcomplicated to me. The test could be as simple as self.assertEqual(importlib.util.MAGIC_NUMBER, ...) -- ___ Python tracker

[issue29514] Add a test case that prevents magic number changes in minor releases

Eric Appelt added the comment: I added PR #54 with a test as Nick describes for checking magic number changes and explaining to the developer the procedure to follow if this is required in a maintenance release. -- nosy: +Eric Appelt pull_requests: +49

[issue29514] Add a test case that prevents magic number changes in minor releases

Brett Cannon added the comment: That comment is poorly worded. All that is really necessary is that the tag differ when you want to have a .pyc file exist side-by-side with another .pyc file. So far we have only changed it when releasing a new version of Python, but for instance Python 3.5.2

[issue29514] Add a test case that prevents magic number changes in minor releases

Serhiy Storchaka added the comment: Now I think that it would be safer to omit the function name in error message rather than fixing the compilation and bumping the magic number. Nick, if you want I could write a patch that fixes the possible security issue for .pyc files with old magic

[issue29514] Add a test case that prevents magic number changes in minor releases

Serhiy Storchaka added the comment: Could anyone please explain to me the comment from _bootstrap_external.py? # Starting with the adoption of PEP 3147 in Python 3.2, every bump in magic # number also includes a new "magic tag", i.e. a human readable string used # to represent the magic number

[issue29514] Add a test case that prevents magic number changes in minor releases

Brett Cannon added the comment: OK, so it sounds like Serhiy had good reason to do what he did. If we do add a test then I would argue that the message should be very clear that the magic number can change in a bugfix release but that it should be discussed with python-dev first so that

[issue29514] Add a test case that prevents magic number changes in minor releases

Serhiy Storchaka added the comment: My apologies for breaking the world. But I considered this bug as security issue. Unlikely, but not impossible. The BUILD_MAP_UNPACK_WITH_CALL opcode is rarely used, but if it is used, it can cause reading arbitrary memory and either crashing or even

[issue29514] Add a test case that prevents magic number changes in minor releases

Changes by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: Providing some additional background: - 3.5.3 was the *first time ever* in more than two decades that CPython changed the magic number in a maintenance release. It *wasn't* necessary to do so - it could have been left alone (as 3.5.0, 3.5.1, and 3.5.2 can read

[issue29514] Add a test case that prevents magic number changes in minor releases

Eric Snow added the comment: This is strictly a problem for the system Python, right? In that case, can't the dist package clear __pycache__ under the system site-packages directory (and any other user-read-only dirs) during install of the updated Python? Is the concern that upgrading Python

[issue29514] Add a test case that prevents magic number changes in minor releases

Petr Viktorin added the comment: According to issue27286, Ubuntu 16.04LTS has the fix and uses the new magic number. (And this is one reason undoing the change in CPython is out of the question.) The patch Nick mentioned is brewing at

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: Ah, interesting. The install-time generation case is even trickier, since you presumably can't force in-place regeneration of existing pyc files on systems that previously had 3.5.2 and upgraded to 3.5.3. When I last spoke to Peter about this yesterday, he was

[issue29514] Add a test case that prevents magic number changes in minor releases

Petr Viktorin added the comment: > How would you propose to fix broken bytecodes in a bugfix be fixed if you > can't bump the magic number? That would depend on the situation. I can imagine that if the bug is severe enough, the number could be bumped, after careful discussion, and with the

[issue29514] Add a test case that prevents magic number changes in minor releases

Barry A. Warsaw added the comment: On Feb 10, 2017, at 05:46 PM, Nick Coghlan wrote: >Note that Fedora doesn't even rebuild all the extension modules when bumping >CPython to a new maintenance release, let alone rebuilding and re-releasing >all the pure Python ones. (RPM supports doing that

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: Note that Fedora doesn't even rebuild all the extension modules when bumping CPython to a new maintenance release, let alone rebuilding and re-releasing all the pure Python ones. (RPM supports doing that just fine, but it would mean shipping thousands of

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: We *do* ship the bytecode in the built RPMs - the issue is that Python maintenance releases shouldn't be triggering rebuilds of thousands of system components (cf http://fedora.portingdb.xyz/ ) and potentially breaking otherwise working end user applications.

[issue29514] Add a test case that prevents magic number changes in minor releases

Brett Cannon added the comment: OK, but how would you propose to fix broken bytecodes in a bugfix be fixed if you can't bump the magic number? And why aren't you regenerating the bytecode when updating Python or shipping the bytecode with the install (and I bet the answer is "RPM doesn't

[issue29514] Add a test case that prevents magic number changes in minor releases

Nick Coghlan added the comment: There are a couple of pieces to it: - first, there's the problem of breaking anyone that actually is doing pyc-only distribution, even though nothing in Fedora itself does that. While that aspect doesn't break Fedora per se, it does break Fedora's compatibility

[issue29514] Add a test case that prevents magic number changes in minor releases

Brett Cannon added the comment: Why shouldn't the magic number be bumped when a bug in the bytecode is fixed in a bugfix release? How did this break Fedora? Are you not shipping source code but only .pyc files? I view the whole point of having the magic number is to cover things like this,

[issue29514] Add a test case that prevents magic number changes in minor releases

Petr Viktorin added the comment: For the record, the magic number was changed in issue27286 -- ___ Python tracker ___

[issue29514] Add a test case that prevents magic number changes in minor releases

New submission from Nick Coghlan: The magic number change in 3.5.3 broke the world when Fedora attempted to rebase from 3.5.2, but upstream CPython developers don't currently get an automated notification that it isn't permitted to change the magic number in a maintenance release - instead,