On Sat, 30 Jan 2010 19:00:05 -0500, Barry Warsaw <ba...@python.org> wrote:
> Linux distributions such as Ubuntu [2]_ and Debian [3]_ provide more
> than one Python version at the same time to their users. For example,
> Ubuntu 9.10 Karmic Koala can install Python 2.5, 2.6, and 3.1, with
> Python 2.6 being the default.
>
> In order to ease the burden on operating system packagers for these
> distributions, the distribution packages do not contain Python version
> numbers [4]_; they are shared across all Python versions installed on
> the system. Putting Python version numbers in the packages would be a
> maintenance nightmare, since all the packages - *and their
> dependencies* - would have to be updated every time a new Python
> release was added or removed from the distribution. Because of the
> sheer number of packages available, this amount of work is infeasible.
As a non-Debian user (I'm a Gentoo user), the above doesn't enlighten me,
even after skimming the referenced document. Perhaps an example would
be helpful?
I'm also not sure how it motivates the PEP. How would putting the Python
version numbers in the package names be an alternate solution to the
problem the PEP is trying to address?
Since Ubuntu is based on Debian, some discussion of what challenges
non-Debian distributions perceive, and what partial solutions they've
crafted, if any, would be a good addition to the motivation section.
(FYI, Gentoo just installs the pyc files into each of the installed
Python's site-packages that is supported by the package in question...disk
space is relatively cheap.)
> extension 'pyr'. The pyr directory contains individual files with the
> cached byte compilation of the source code, identical to current pyc
> and pyo files. The files inside the pyr directory retain their file
> extensions, but the base name is replaced by the hexlified [10]_ magic
> number of the Python version the byte code is compatible with.
[...]
> For example, a module `foo` with source code in `foo.py` and byte
> compiled with Python 2.5, Python 2.6, Python 2.6 `-O`, Python 2.6
> `-U`, and Python 3.1 would have the following file system layout::
>
> foo.py
> foo.pyr/
> f2b30a0d.pyc # Python 2.5
> f2d10a0d.pyc # Python 2.6
> f2d10a0d.pyo # Python 2.6 -O
> f2d20a0d.pyc # Python 2.6 -U
> 0c4f0a0d.pyc # Python 3.1
This may be a bit silly, but I find the hexified magic numbers ugly
and uninformative. I'd rather see version numbers in there. I realize
there isn't a one to one correspondence, though, so perhaps there isn't
a more readable alternative.
> Effects on non-conforming Python versions
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> Python implementations which don't know anything about `pyr`
> directories will ignore them. This means that they will read and
> write `pyc` files as usual. A conforming implementation will still
> prefer any existing `foo.pyr/<magic>.pyc` file over an existing
> sibling `pyc` file.
>
> The one possible conflicting state is where a sibling `pyc` file
> exists, but its magic number does not match.
>
> In the default case, when Python finds a `pyc` file with a
> non-matching magic number, it simply overwrites the `pyc` file with
> the new byte code and magic number. In the absence of the `-R` flag,
> this remains unchanged. When the `-R` flag was given, the
> non-matching sibling `pyc` file is ignored - it is neither removed nor
> overwritten - and a `foo.pyr/<magic>.pyc` file is written instead.
Shouldn't most of this discussion be in the section on the general
algorithm that is to be implemented? The discussion of the effect on
non-conforming versions should then talk about what happens on a system
that has both non-conforming and conforming versions under various
scenarios, which I don't think the section as written really makes clear.
> The implementation of this PEP would have to ensure that the same
> directory level is returned from `__file__` as it does without the
> `pyr` directory, so that the common idiom above continues to work::
>
> >>> import foo
> >>> foo.__file__
> 'foo.pyr'
> # baz is a package
> >>> import baz
> >>> baz.__file__
> 'baz/__init__.pyr'
This requirement should be in the specification portion, not in the
"alternatives" section.
> Note that some existing Python code only checks for `.py` and `.pyc`
> file extensions (and possibly `.pyo`). These would have to be
> extended to also check for `.pyr` extensions.
Some of that code is in the stdlib :)
Perhaps it is time to add a library function that returns all the
extensions that compiled python files might take, regardless of whether
or not this PEP gets accepted.
> Multiple file extensions
> ------------------------
>
> The PEP author also considered an approach where multiple thin byte
> compiled files lived in the same place, but used different file
> extensions to designate the Python version. E.g. foo.pyc25,
> foo.pyc26, foo.pyc31 etc. This was rejected because of the clutter
> involved in writing so many different files. The multiple extension
> approach makes it more difficult (and an ongoing task) to update any
> tools that are dependent on the file extension.
I'm inclined to agree with the other posters who suggested foo.xxx.pyc.
Since we already have pyc and pyo files for each pyc, what's a few more :)
> * What about `py` source files that are compatible with most but not
> all installed Python versions. We might need a way to say "this py
> file should be hidden from Python versions X.Y or earlier". There
> are three options:
This is a critical concern, IMO, that needs to be sorted out before
this PEP can make progress.
> - Use file system tricks to only share py files that are actually
> sharable in all installed Python versions (e.g. different search
> directories for Python X.Y and Python X.Z).
I'd like to see a discussion included in the PEP of how exactly you
see the multiple-bytecode-version support being used. Currently stuff
normally gets installed in the site-packages directory of each particular
Python version. So it seems like in a non-vendor-packaged install,
-R would be useful only for a user's local code? I would like to see
this discussed in the PEP.
As for these file system tricks, you'd have to have one directory for
each possible combination of supported python versions, and install
packages to the directory corresponding to the set they support,
since Python versions can get added and subtracted from the installed
Python versions. And then I presume you mean "different search *paths*
for...", or, more specifically, different default sys.path setups for
each installed Python version, where the path would point to all the
directories that included that Python version as one of the supported set.
> - Introduce Python syntax that is legal before __future__ imports
> and is evaluated to determine if the py file is compatible,
> raising an `ImportError('no module named foo')` if not.
It seems to me that 'no module named foo' would be absolutely the wrong
error message to raise, since the user would look and say, "but, there's
a foo.py file there, I can see it, what do you mean there's no module
named foo?" Also, how would you address non-conformant versions under
this scenario?
> - Add an optional metadata file co-located with the py file that
> declares which Python versions it is compatible with.
Couldn't this actually go at the package level (which could be a
single module), and thus be covered by the distutils meta-data work?
In other words, Python would look for the metdata file associated with
the package/module in the directory that's actually on sys.path. If so,
it sounds like the most sensible option, although it would require a
more complicated implementation in the import machinery.
> * Would a moratorium on byte code changes, similar to the language
> moratorium described in PEP 3003 [16]_ be a better approach to
> pursue, and would that solve the problem for vendors? At the time
> of this writing, PEP 3003 is silent on the issue.
Unless the bytecode change moratorium was permanent (unlikely), how would
this solve the vendor issues? Having multiple python versions installed
is a problem that will never go away as long as the language is allowed
to evolve at all.
--
R. David Murray www.bitdance.com
Business Process Automation - Network/Server Management - Routers/Firewalls
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
