Besides the "normal" pending update of the python version for the unstable distribution, there will be more changes around python packaging, including the introduction of python-3.x and addressing some packaging issues.
Python versions --------------- - 2.4 is still used by zope-2.x and depending packages like plone. There was a SoC project which intended to update zope-2.x to use a newer python version, but afaik this didn't reach a stable state yet. It doesn't make sense to provide a full set of third party python modules and extensions for 2.4. If it is forseeable that zope-2.x still requires 2.4 when the freeze for the next Debian release approaches, maybe package only 2.4 modules for zope-2.x's and plone's dependencies. An alternative might be not to distribute zope-2.x and plone with the next stable release. - 2.5 is superseded by 2.6; currently there doesn't seem to be a reason to ship 2.5 and modules for 2.5 with the next stable release. The upstream 2.5 maintainance branch doesn't see bug fixes anymore, only security releases will be made from this branch. - 2.6 is the current stable release which should enter unstable soon and become the default after extension packages supporting more than one python version are built and migrated to testing. - I do not want to speculate about a 2.7 release and the upstream version freeze for the next Debian release. If the 2.7 release is well ahead of the Debian freeze, then there should be no problem to include it. - 3.0/3.1: I do not plan to upload 3.0 to unstable or experimental, but will prepare 3.1 packages for experimental and upload those to unstable with the final release or a late release candidate. The 3.1 release is planned for April 2009. Packaging for Python 3.x ------------------------ Python 3.x is not upward compatible to 2.x and requires changes to the sources. Almost no file can be used unmodified. A large part of the changes can be done automatically with the help of the 2to3 utility. Porting resources can be found at - http://docs.python.org/3.0/whatsnew/3.0.html#porting-to-python-3-0 - http://docs.python.org/3.0/howto/cporting.html#cporting-howto - http://docs.python.org/3.0/library/2to3.html Upstream does expect Python-2.x to stay around for a while; Python-3.x will need some time until extensions and modules are ported. For this reason I do plan to have both Python-2.x and Python-3.x in the next stable Debian release. Some third party modules and extensions may be released in a form where the code for 3.x can be auto-generated from the 2.x code with the 2to3 utility, some upstreams may decide to stop 2.x support with one version and provide 3.x support with another version. Debian has to support both approaches. The currently used packaging methods only allow packaging of one version for all Python version (installing in a shared area and providing this version for each Python version). Binary packages will double in size when providing both 2.x and 3.x compatible code, so it does make sense to provide separate binary packages for 2.x and 3.x. These new packages should be prefixed with `python3-' instead of `python-'. It's up to the package maintainer if these packages are built from one or two source packages. There will be a `python3' package and similiar packages built out of a python3-defaults, which will provide the `python3' binary. Local installation path ----------------------- When installing Python modules using distutils, the resulting files end up in the same location wether they are installed by a Debian package, or by a local user or administrator, unless the installation path is overwritten on the command line. Compare this with most software based on autoconf, where an explicit prefix has to be provided for the packaging, while the default install installs into /usr/local. For new Python versions packaged in Debian this will change so that an installation into /usr (not /usr/local) requires an extra option to distutils install command (--install-layout=deb). To avoid breaking the packaging of existing code the distutils install command for 2.4 and 2.5 will just accept this option and ignore it. For the majority of packages we won't see changes in the packaging, provided that the python packaging helpers can find the files in the right location and move it to the expected target path. A second issue raised by developers was the clash of modules and extensions installed by a local python installation (with default prefix /usr/local) with the modules provided by Debian packages (/usr/local/lib/pythonX.Y/site-packages shared by the patched "system" python and the locally installed python. To avoid this clash the directory `site-packages' should be renamed to `dist-packages' in both locations: - /usr/lib/pythonX.Y/dist-packages (installation location for code packaged for Debian) - /usr/local/lib/pythonX.Y/dist-packages (installation location for locally installed code using distutils install without options). The path /usr/lib/pythonX.Y/site-packages is not found on sys.path anymore. About the name: Discussed this with Barry Warsaw and Martin v. Loewis, and we came to the conclusion that using the same directory name for both locations would be the most consistent way. This change should make the request to conditionalize the inclusion of /usr/local/lib/pythonX.Y/site-packages into sys.path obsolete. If needed we can provide a symlink /usr/lib/pythonX.Y/site-packages pointing to dist-packages. Common installation location for shared code -------------------------------------------- The source files for code (public modules) which is shared between Python versions are currently located in different paths: - /usr/share/pycentral/<binary package name> - /usr/share/pysupport/<binary package name> - /usr/share/pyshared With this approach the Debian packaging tools (dpkg) cannot handle file conflicts and replaces themself anymore. Errors are delayed to the python packaging tools which either ignore these kind of errors or have to replicate the conflicts/replaces checks. The use of a common location was proposed in 2006, but didn't get any feedback at this time. Last summer I did choose a name for this location, and a large part of packages already install into this location `/usr/share/pyshared'. For code shared between Python-3.x installations I propose to use `/usr/share/py3shared'. Use of exactly one location in sys.path --------------------------------------- Depending on the python packaging helper, public modules are available in different locations on sys.path. Having parts of python packages or upstream plugin systems installed into different locations may break the python package or the plugin system. To avoid this situation we do have the policy (section 2.1, second paragraph) that such packages must use the same location (means: the same python packaging helper). This part of the policy tends to be ignored. - The import order is changed; this issue was raised earlier. See http://lists.debian.org/debian-python/2006/01/msg00065.html The use of relative imports (introduced in 2.5) make this more obvious. - Plugin systems derive the location for plugins from their installation location. It is not an upstream bug if Debian packages different plugins with different python packaging helpers. The choice of packaging helper should not add additional constraints on upstream software. To avoid these problems in the future, exactly one location for public python modules shold be used. Avoid runtime dependencies on python packaging helpers ------------------------------------------------------ During upgrades or distribution upgrades, public python modules are not available for use, or they may be in an inconsistent state. This is the time between removal of old version / unpacking new version and the postinst configuration, when the python packaging helpers are run and the public modules made available in sys.path. There are currently work arounds implemented to shorten the window of unavailability: - Not removing symlinked files and not removing byte-compiled files during upgrades. This only works reliable if the new version does neither remove nor add files, or else an inconsistent set of files in a package may be imported during an upgrade. - Already create the symlinked files in the preinst step, because the time between running the preinst and unpacking the new version is much shorter. Ideally I would like to see public python modules available after unpack without the help of any helper application. This is possible by creating the symlinks while creating the package and including these in the package instead of creating these at installation time. The only configuration needed at installation time would be byte-compilation. The advantage would be a more robust upgrade path. The disadvantage of this approach is the limitation to a specific set of python versions (the package will have a dependency of the form "python (<< X.Y)", which we already have for architecture dependent packages containing extensions. This would add this kind of dependency to all architecture independent python-* modules and then requires an upload of these packages for each python transition as well. Afaics this would not affect a transition or a set of transitions to testing, because no new dependencies on new versions of shared libraries are introduced during these uploads. This approach certainly has an impact on the ftp archive and its mirrors, but I can't say if this would be an issue or not. If this cannot be done for all packages, this should be done for a subset of "core" modules and extensions. There is still the issue of handling name space packages based on setuptools. Ideally existing techniques like diversions should be used for this, even if it looks loke overkill to divert the same empty __init__.py file. Various ------- There are other things which may be worth a look. - The current policy advises packages to byte-compile only the files which a package owns itself. This is not done by several packages, with the result that an upgrade caused by a byte- compilation error may be attributed to the wrong package (makes the upgrade of an otherwise fine package fail). - The removal of a python version will cause the need for massive rebuilds. because many python extensions currently have dependencies of the form pythonX.Y-foo. There is nothing what can be done now for the upcoming removal, but those dependencies should not be there by default. This is 2.4 of the python policy, but many packages tend to ignore that. I will start uploading python2.6 and related packages, then proceed with python3.x in the next weeks. Matthias -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org