[Python-Dev] importlib hooray!
I have become a fan of the new python 3.3 importlib in the last few days. It has allowed me to write a ModuleMapper which I put into sys.metapath (in sitecustomize.py, for Python 3.3). This mapper currently does rename modules like 'Queue' or '_winreg' to the Python3 modules 'queue' or 'winreg' without the need to change my 2.7 sourcecode (changing the sourcecode is required when using six.moves to write 2.x/3.x compatible code). The six.moves approach has another problem with freeze tools (py2exe for example): ModuleFinder does not find the moved modules because it cannot track __import__(). I have also started a new modulefinder which uses importlib to find modules; this was quite easy since I could copy a lot of code from importlib._bootstrap. The great thing is that this new modulefinder is much better than the old one: It finds modules in zipped eggs or other zip-archives which are on sys.path; also it finds the renamed modules correctly that my ModuleMapper has mapped. The only thing that I am missing is that it is a bit of guesswork to find out the type of the module that importlib.find_loader() has found. Previously imp.find_module returned a tuple containing the type of module found (like imp.PY_SOURCE), I have to poke around in some attributes of the result of importlib.find_loader() to guess the type. Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze
On Sun, Feb 3, 2013 at 11:40 AM, Chris Jerdonek chris.jerdo...@gmail.com wrote: On Sun, Feb 3, 2013 at 10:33 AM, Éric Araujo mer...@netwok.org wrote: Le 03/02/2013 07:48, Antoine Pitrou a écrit : I vote for removing the distutils is frozen principle. I’ve also been thinking about that. There have been two exceptions to the freeze, for ABI flags in extension module names and for pycache directories. When the stable ABI was added and MvL wanted to change distutils (I don’t know to do what exactly), Tarek stood firm on the freeze and asked for any improvement to go into distutils2, and after MvL said that he would not contibute to an outside project, we merged d2 into the stdlib. Namespace packages did not impact distutils either. Now that we’ve removed packaging from the stdlib, we have two Python features that are not supported in the standard packaging system, and I agree that it is a bad thing for our users. I’d like to propose a reformulation of the freeze: This could be common knowledge, but is the current formulation of the freeze spelled out somewhere? I asked this earlier, but didn't see a response. Is the freeze stated somewhere like in a PEP? If not, can someone state it precisely (e.g. what's allowed to change and what's not)? Thanks, --Chris ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build
Am 05.02.2013 07:13, schrieb Terry Reedy: On 2/4/2013 3:04 PM, Matthias Klose wrote: - the 2.7 branch is the only branch which doesn't have expected release dates on the calendar. Which calendar? I see 2.7.4, 3.2.4 (its final release), and 3.3.1 on the Release Schecule at python.org. maybe these are now updated. until recently 3.3.1 was targeted for Nov/Dec 2012, and 2.7.4 wasn't found on the calendar. - there were way too may regressions checked in on at least the 2.7 branch. Regressions? That normally means adding a bug as part of a patch, especially one that was previously fixed. We continually add tests to try to prevent that. What period of time do you mean 'were' to refer to? - http://bugs.python.org/issue9374 - http://bugs.python.org/issue15906 - http://bugs.python.org/issue16012 - http://bugs.python.org/issue10182 - http://bugs.python.org/issue16828 not a complete list, all these are past the 2.7.3 release. Is it just our vcs merge model that we first have to check in on the branches, and then merge to the trunk? Mercurial works best if merges are done in the forward direction. However, this is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The repository is a two-headed monster ;=). so it should be possible to delay patches for 2.7 until they don't show issues in 3.2 or 3.3. Afaics python is the only project to have such an approach. Others have trunk first, maybe with immediate backport, maybe with later backport. If a patch is first commited to tip (for 3.4) and then backported to 3.3, then when the backport is pushed, a null merge is needed to avoid making a third head, and the dag looks messier. (I believe 'null merge' is the right term, but I have never done this.) My impression is that most 3.x bugfix patches merge forward with little or no change. so is it only this technical limitation, why the bugfix process is defined this way? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] importlib hooray!
On Wed, Feb 6, 2013 at 4:26 AM, Thomas Heller thel...@ctypes.org wrote: I have become a fan of the new python 3.3 importlib in the last few days. Glad it's working out for you! It has allowed me to write a ModuleMapper which I put into sys.metapath (in sitecustomize.py, for Python 3.3). This mapper currently does rename modules like 'Queue' or '_winreg' to the Python3 modules 'queue' or 'winreg' without the need to change my 2.7 sourcecode (changing the sourcecode is required when using six.moves to write 2.x/3.x compatible code). The six.moves approach has another problem with freeze tools (py2exe for example): ModuleFinder does not find the moved modules because it cannot track __import__(). I have also started a new modulefinder which uses importlib to find modules; this was quite easy since I could copy a lot of code from importlib._bootstrap. The great thing is that this new modulefinder is much better than the old one: It finds modules in zipped eggs or other zip-archives which are on sys.path; also it finds the renamed modules correctly that my ModuleMapper has mapped. The only thing that I am missing is that it is a bit of guesswork to find out the type of the module that importlib.find_loader() has found. Previously imp.find_module returned a tuple containing the type of module found (like imp.PY_SOURCE), I have to poke around in some attributes of the result of importlib.find_loader() to guess the type. You should be able to tell with an isinstance check on the returned loader and not have to look at any attributes (e.g. source will subclass SourceLoader, frozen will subclass FrozenImporter, etc.). If it continues to be an issue just let me know and we can see what we can do (this is obviously a niche need and so I would prefer not to change the loader API to explicitly accommodate this, but it can be done). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyCon Tickets Almost Sold Out
Since the Language Summit is held at PyCon I think this counts as on-topic... If you're interested in going to the conference, there are under 50 tickets remaining: https://us.pycon.org/2013/registration/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze
Le 06/02/2013 05:03, Chris Jerdonek a écrit : I asked this earlier, but didn't see a response. Is the freeze stated somewhere like in a PEP? It’s part in notes from the PyCon 2010 Language Summit, part in unwritten policy in the heads of people involved in distutils bugs these last years. If not, can someone state it precisely (e.g. what's allowed to change and what's not)? Bug fixes are possible; changes to keep working with Python (i.e. pyc files are generated in pycache directories after 3.2); changes to be able to build on current OSes (e.g. following Mac dev tools location change, introduction of Debian multiarch, etc.). Some bugs have been here for so long that everybody depends on them or work around them, or they would be extremely painful to fix (e.g. in the option parsing code) for little benefit, so they are wontfix. Cleanups, refactorings and improvements were banned by the feature freeze. Regards ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build
On Feb 06, 2013, at 02:38 PM, Antoine Pitrou wrote: Le Mon, 04 Feb 2013 21:04:39 +0100, Matthias Klose d...@ubuntu.com a écrit : So what I do understand, build-related issues like an arm64 or mingw32 port are ok for 2.7, if they are stable on the trunk, and communicated on python-dev? Making Python build on a new platform is, AFAICT, considered a new feature, not a bugfix. For example, we support new MSVC versions in the main development branch, not in bugfix branches. Except that 2.7 is an exception to that since it's the last of the Python 2 series, and has a much longer lifespan than normal releases. I'm pretty sure we agreed that there would be some exceptions for issues like new platforms for 2.7. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build
On 2/6/2013 8:16 AM, Matthias Klose wrote: Am 05.02.2013 07:13, schrieb Terry Reedy: On 2/4/2013 3:04 PM, Matthias Klose wrote: - there were way too may regressions checked in on at least the 2.7 branch. I think you are using 'regression' too freely. But that aside, I think you are pointing to a legitimate issue, in the first and maybe second example of how to handle 2.7. Regressions? That normally means adding a bug as part of a patch, especially one that was previously fixed. We continually add tests to try to prevent that. What period of time do you mean 'were' to refer to? - http://bugs.python.org/issue9374 From what I read (but it was a long discussion), this patch fixed behavior that was agreed to be a bug. The disagreement was whether the bug was clear or obscure. Since the bug was long-standing, and since much code had accommodated to the bug, and even depended on it, the bugfix broke more code than usual. This is not a regression, but is a concern, and if I understand correctly, the fix was removed or revised for existing versions. - http://bugs.python.org/issue15906 I did not understand the issue, except to note that it affected all versions, involved apparent ambiguity in the doc, and was caught by ubuntu testing. - http://bugs.python.org/issue16012 This reported a clear regression, on all versions, not just 2.7, that was caught before release. As the person responsible for the regression said Too bad it [the regressed behavior] wasn't tested for :(. We still need more patches to improve test coverage. - http://bugs.python.org/issue10182 This fixed an all-versions bug in _sre. The initial patch was fine for 3.x but not sufficiently adjusted for 2.7. Py_BuildValue needed to be replaced in certain places with PyInt_FromSsize_t rather than PyLong_FromSsize_t to maintain the external API. This was not caught because by the stdlib test but was by a third party who tests their package with tip. Yay for them. The process worked. - http://bugs.python.org/issue16828 Again, an all-versions issue, not 2.7 specific. http://bugs.python.org/issue14398 disabled compression of '' in the process of enabling compression of 4GB strings. Not a bad tradeoff, really, but not best. Again, this was not caught initially because there *was* no stdlib test for ''. But there was in a 2.7 ubuntu package. not a complete list, all these are past the 2.7.3 release. In only 1 of the 5 was there a 2.7-specific regression. I am not sure what specific change you would like. In the last three examples, would you want 2.7 left with the bug? Is it just our vcs merge model that we first have to check in on the branches, and then merge to the trunk? Mercurial works best if merges are done in the forward direction. However, this is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The repository is a two-headed monster ;=). so it should be possible to delay patches for 2.7 until they don't show issues in 3.2 or 3.3. Yes, however 1. Later may mean never. 2. If the regression is 2.7 specific, as with #10182, then not showing an issue for 3.x is irrelevant. 3. If the regression is general, then *at present*, the 2.7 regression is the most likely to be caught by external testing (mostly by unbuntu). That says to me that bug fixes for 2.7 *should* go into tip along with the others. I could even argue that the 2.7 fix should go in first. 4. If bugfixes are not applied to 2.7, that increases the distance between 2.7 and 3.x and possibly increases the difficulty of writing 23 code. If we do leave a fixed-in-3.x bug in 2.7, then perhaps, if it is possible to detect that the bug is being depended upon, there should be a deprecation warning in the code. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Moderating the severity of the distutils freeze
On Thu, Feb 7, 2013 at 6:45 AM, Éric Araujo mer...@netwok.org wrote: Bug fixes are possible; changes to keep working with Python (i.e. pyc files are generated in pycache directories after 3.2); changes to be able to build on current OSes (e.g. following Mac dev tools location change, introduction of Debian multiarch, etc.). Some bugs have been here for so long that everybody depends on them or work around them, or they would be extremely painful to fix (e.g. in the option parsing code) for little benefit, so they are wontfix. Cleanups, refactorings and improvements were banned by the feature freeze. I think the points to be clarified are that *new* internal APIs (including new commands, like bdist_wheel) are legitimate in a feature release, and internal changes which don't affect APIs (like some of the tweaks Dave Malcolm and I made to try to make building Python 3 RPMs on RHEL 6 with bdist_rpm less broken) are legitimate in maintenance releases. distutils is going to be around for a long time for backwards compatibility reasons, so if people want to improve it within those constraints they should be allowed to - we just need to be extra careful with maintenance releases, by treating even nominally internal APIs as if they were public. The current interpretation of the situation as a complete no few features freeze is overkill, when the setuptools/distribute developers will have plenty of time to send up warning flares during the alpha/beta/rc cycle. An appropriate rule would be more like no major refactorings of distutils internals (which may indeed mean distutils never supports metadata versions past 1.1). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
