Re: version independent pythin packages: ?
Hi, Donovan Baarda wrote:
It seems each individual package should be responsible for compiling
it's own *.py's with an appropriate version of python, even in
/usr/lib/python. We can't have the python package handle it directly.
Hmm. Correct. See below.
Try the following set of dependencies;
OK, for one, docutils isn't supported for python 2.2, so all those long
lines get a bit shorter.
I think the 'dependency package' approach works best.
Package: python-docutils
Architecture: all
Depends: python2.3-docutils | python2.2-docutils
Description: Utilities for the documentation of Python modules
The purpose of the Docutils project is to create a set of tools for
processing plaintext documentation into useful formats, such as HTML,
XML, and TeX.
.
The package includes the reStructuredText parser and the Python
Docstring Processing System project.
Package: python2.3-docutils
Architecture: all
Depends: python2.3, python-docutils, python-roman
Description: Dependency package for python-docutils with Python 2.3
This package is a dependency package. It represents the requirements of
the python-docutils package when used with Python 2.3.
Package: python2.2-docutils
Architecture: all
Depends: python2.2, python-docutils, python2.2-xmlbase, python-roman,
python-textwrap
Description: Dependency package for python-docutils with Python 2.2
This package is a dependency package. It represents the requirements of
the python-docutils package when used with Python 2.2.
If there's no objection, the next version will look like this.
(Due out shortly, as I need to package upstream's 0.3 as well as fix a
packaging bug.)
Package: python-xmlbase
Depends: python (2.3) | python2.2 | python2.1 | python2.0 | python1.6
| python1.5
Provides: python2.2-xmlbase, python2.1-xmlbase, python2.0-xmlbase,
python1.6-xmlbase, python1.5-xmlbase
Duh? xmlbase doesn't seem to be supported for Python 2.1.
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | [EMAIL PROTECTED]
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
--
I could not love thee, dear, so much, loved I not honor more.
-- Richard Lovelace
Re: version independent pythin packages: ?
On Thu, Aug 07, 2003 at 11:34:27AM +0100, Ricardo Javier Cardenes Medina wrote:
Thinking on this problem a bit further (not feasible with current
implementation), wouldn't it be nice if the user could enable Python
(via environment or command line switch) to use some local repository
(~/.python/X.Y/) to store .py{c,o} files if (s)he hasn't write access to
the system files?
You always can copy the .py[co] files in
/usr/lib/python2.X/site-packages at install time. Since these directory
come before /usr/lib/site-python in the default sys.path, the compiled
modules will be used on import.
Now this has some unpleasant consequences:
* python no longer has a way of seeing if the .pyc is up to date
* I think this screws up the exception reporting routines in python
Maybe this could be handled with symlinks?
--
Alexandre Fayolle
LOGILAB, Paris (France).
http://www.logilab.com http://www.logilab.fr http://www.logilab.org
Développement logiciel avancé - Intelligence Artificielle - Formations
Re: version independent pythin packages: ?
Ricardo Javier Cardenes Medina writes: Of course, all this require manual handling from the user. What I was proposing would require a whole PEP and some reasonable design and implementation, etc, so Python itself could map those .pyc to their original file, only resorting to them if the original .py/.pyc are not synced; deleting them if they're not usefult any more, etc. PEP304
Re: version independent pythin packages: ?
On Fri, 2003-08-08 at 12:50, Donovan Baarda wrote: On Thu, 2003-08-07 at 18:44, Alexandre Fayolle wrote: On Thu, Aug 07, 2003 at 12:58:25PM +1000, Donovan Baarda wrote: [...] Python applications using the default Python with their own modules not in /usr/lib/site-python... not an issue? Actually... I think I prefer /usr/lib/python/site-packages rather than /usr/lib/site-python because that way the directory structure mirrors that for the /usr/lib/pythonX.Y specific version tree. Having just looked at a few packages, it seems /usr/lib/site-python is already well on the way to becoming the default location for version independent modules. Please forget I said the above :-) -- Donovan Baarda [EMAIL PROTECTED]
Re: version independent pythin packages: ?
Hi,
Donovan Baarda wrote:
If there's no objection, the next version will look like this.
(Due out shortly, as I need to package upstream's 0.3 as well as fix a
packaging bug.)
Um... I have a few problems with this. It doesn't really follow the
current Python Policy.
True. But then IMHO the policy might be amenable to changing. This is why I
posted here...
After writing this I had a look at the current docutils package and
think I know what you are getting at. python-docutils puts all the
modules in /usr/lib/site-python and the other guys just make sure the
dependencies are met.
Right...
For some reason I feel a bit uncomfortable with it. The python-docutils
package doesn't support the default python. Installing
python-docutils, python2.3-docutils, and python (2.2) will meet all
dependencies but result in a docutils that doesnt work for
/usr/bin/python.
You're right. *Sigh*.
I'm not sure what the best solution is. My gut instinct is to have
python-docutils just depend on python-xmlbase and python-textwrap, and
have python (2.3) Provides them. That way you could have;
The problem I have with that is that there are quite a few modules which have
been canonized in 2.2, and more in 2.3, and next year there'll be more in
2.4. You can't back-change sarge's python2.2 package to include yet another
Provides: python-foo which is already included in python 2.3 but which
somebody has separated out in order to support python-foobar on older Python
installations.
It would be nice if you could specify dependencies as follows;
Depends: (python2.2, python2.2-xmlbase, python-textwrap) | (python2.3),
python-roman
Hmm. You can, just distribute the stuff out (standard Boole algebra):
Depends: python2.3 | python2.2, python2.2-xmlbase | python2.3,
python-textwrap | python2.3, python-roman
It looks ugly as hell, though, and I'll have to check whether it really does
the right thing when you have python2.2 installed and pull in
python-docutils.
It gets even worse if I decide to support Python versions 2.3, but that
doesn't seem to be helpable.
I was going to suggest a solution, but I'm tired and I can't think of
one. docutils is a tough one :-(
Yes, it's an interesting testcase for changing policy. ;-)
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | [EMAIL PROTECTED]
Disclaimer: Das Zitat wurde zufllig ausgewhlt. | http://smurf.noris.de
--
Frage an Radio Eriwan:
Was ist der Unterschied zwischen einem Optimisten und einem Pessimisten?
Antwort: Der Optimist lernt Englisch, der Pessimist chinesisch.
Re: version independent pythin packages: ?
On Fri, Aug 08, 2003 at 01:52:40PM +0200, Matthias Urlichs wrote: Hi, Donovan Baarda wrote: It would be nice if you could specify dependencies as follows; Depends: (python2.2, python2.2-xmlbase, python-textwrap) | (python2.3), python-roman Hmm. You can, just distribute the stuff out (standard Boole algebra): Depends: python2.3 | python2.2, python2.2-xmlbase | python2.3, python-textwrap | python2.3, python-roman It looks ugly as hell, though, and I'll have to check whether it really does the right thing when you have python2.2 installed and pull in python-docutils. I'm not sure this will work if I try to use docutils as a library in another package supporting only python2.2. Initially installed packages: python2.3 I apt-get install python2.2-so-and-so which Depends: python2.2, python-docutils python2.2 will be installed, so is python-docutils, but not python-textwrap nor python2.2-xmlbase because python2.3 is there. The result is a malfunctionning python2.2-so-and-so, unless the packager manually adds python-textwrap and python2.2-xml dependencies, but this feels wrong to me. -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org Développement logiciel avancé - Intelligence Artificielle - Formations
Re: version independent pythin packages: ?
On Fri, Aug 08, 2003 at 11:05:09AM +0200, Matthias Klose wrote: Ricardo Javier Cardenes Medina writes: Of course, all this require manual handling from the user. What I was proposing would require a whole PEP and some reasonable design and implementation, etc, so Python itself could map those .pyc to their original file, only resorting to them if the original .py/.pyc are not synced; deleting them if they're not usefult any more, etc. PEP304 Heh... Deep in my heart I knew someone had been thought this before. :-) It's just of common sense :D
Re: version independent pythin packages: ?
On Thu, Aug 07, 2003 at 12:58:25PM +1000, Donovan Baarda wrote: The only problem is when someone with write access to /usr/lib/site-python uses a non-default python... the pyc's will be updated for the non-default python. After testing, it seems that there is no way to prevent root from updating the pyc's when using a non-default python. However, even if Yes, even chmoding the .pyc file won't work. this does occur, everything will still work, it will just be slightly less optimal when using the default python. The only solution is root should never use the non-default python when importing modules from /usr/lib/site-python... which may or may not be OK. For most programs this should not really be a big issue. Most of the time, people tend to use mostly one python version on production machines. The worst case I've seen with missing/inapropriate .pyc files was on a cgi based web application, where the performance increase we got when fixing the problem was noticeable by end users. For long running processes, the import is done only once, so the loss of time is evened out in the long run. Given the complexity of the alternatives, this is a simple solution that fixes the problem with only minor issues. I'd be tempted to make this the solution and put it into the Python policy. +1 Anyone else agree? Other things to think about; Python applications using the default Python with their own modules not in /usr/lib/site-python... not an issue? They are expected to configure their own PYTHONPATH, are they not? Making sure all the pyc's are re-compiled when the default python is updated. Yes. Should not be too difficult. A brute force approach is to have the python packages post-inst run dpkg-reconfigure over every package that depends on python, and require that packages recompile their pyc files on a dpkg-reconfigure. This has the advantage of notifying all these packages when the default python has changed so they can do other stuff if they need to. This would be nice, but packages will take some time to catch up. Another thing we should take care of is packages which used to be pure python in version N, and include C extensions in version N+1. The packages for N+1 should be versioned, but how can we provide a nice upgrade path? In order to be sure not to break anything, we would have to make the new python-foobar package depend on python2.X-foobar for X in (1,2,3). The last point is how to write dependencies on packages which are ot available on all python versions. For instance python-docutils needs python-xmlbase and python-difflib. python-xmlbase exists for python2.1 and 2.2 but not 2.3, and difflib exists for 2.1, but not 2.2 or 2.3. How should the dependencies be written ? -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org Développement logiciel avancé - Intelligence Artificielle - Formations
Re: version independent pythin packages: ?
On Thu, Aug 07, 2003 at 12:58:25PM +1000, Donovan Baarda wrote:
Anyone else agree?
I see no problem, aside from performance for those users not using the
default version, but since this is only a load time problem, it
shouldn't be more than a little annoyance. I suppose that really picky
users could add some local repository to their PYTHONPATH and copy there
the load-time-critical modules if really in a need, or even ask the
sysadmin to set a version-specific optimized repository.
We can't do better with current Python behaviour, and I think most pure
Python packages would benefit from this change to Policy.
Thinking on this problem a bit further (not feasible with current
implementation), wouldn't it be nice if the user could enable Python
(via environment or command line switch) to use some local repository
(~/.python/X.Y/) to store .py{c,o} files if (s)he hasn't write access to
the system files?
Re: version independent pythin packages: ?
On Thu, 2003-08-07 at 18:44, Alexandre Fayolle wrote: On Thu, Aug 07, 2003 at 12:58:25PM +1000, Donovan Baarda wrote: [...] Python applications using the default Python with their own modules not in /usr/lib/site-python... not an issue? Actually... I think I prefer /usr/lib/python/site-packages rather than /usr/lib/site-python because that way the directory structure mirrors that for the /usr/lib/pythonX.Y specific version tree. They are expected to configure their own PYTHONPATH, are they not? Yeah, the issues with the current policy and this is that there is no way to re-compile these modules when the default python changes. Making sure all the pyc's are re-compiled when the default python is updated. Yes. Should not be too difficult. There are tricks here that make it a little harder than you would expect. Because deb's don't support triggers, the python package will have to find all the *.py's that need compiling itself. There are two ways I can see to handle this; 1) have all packages put .py's in a suitable place (/usr/lib/python) so they can be re-compiled. But this breaks Python programs that have their own modules in some other place. 2) Have the python package identify all the packages with py's that need to be recompiled, then tell them to recompile themselves by executing something that triggers the package's recompile scripts (dpkg-reconfigure or the .post-inst, or something). Identifying the .py's that need recompiling is actually pretty tricky too.. you need to identify all packages that depend on python with *.py files. A brute force approach is to have the python packages post-inst run dpkg-reconfigure over every package that depends on python, and require that packages recompile their pyc files on a dpkg-reconfigure. This has the advantage of notifying all these packages when the default python has changed so they can do other stuff if they need to. This would be nice, but packages will take some time to catch up. I think a dpkg-reconfigure re-runs the post-inst scripts which should already compile stuff. We could call the post-inst scripts directly, but dpkg-reconfigure might do other stuff that is more package friendly. I don't think any packages apart from python will need changing to support this (hence why I suggested it, despite it being a bit of overkill) Another thing we should take care of is packages which used to be pure python in version N, and include C extensions in version N+1. The packages for N+1 should be versioned, but how can we provide a nice upgrade path? In order to be sure not to break anything, we would have to make the new python-foobar package depend on python2.X-foobar for X in (1,2,3). Example use case; Up to and including python (2.1) we had; python-foo (1.1) with files in /usr/lib/python Depends: python Now we have python (2.2) and a new upstream release of foo python2.2-foo (2.1) has a C exension in /usr/lib/python2.2 Depends: python2.2 We need to make the following packages; python2.1-foo (1.1) with files in /usr/lib/python Depends: python1.5 | python1.6 | python2.0 | python2.1 Provides: python1.5-foo, python1.6-foo, python2.0-foo Replaces: python-foo (=1.1) python-foo (2.1) as a wrapper for python2.2-foo Depends: python (=2.2), python (2.3), python2.2-foo the almost version idependent python2.1-foo provides support for python1.5, python1.6, python2.0, and python2.1. The python2.2-foo provides support for python2.2. The new python-foo wrapper provides support for the default python (2.2). Because the python path for python2.2 lists /usr/lib/python2.2 before /usr/lib/python, python2.2 will grab the correct C extension version, but other versions of python will use the old pure python packages in /usr/lib/python. Hmmm... this has implications that affect what version the .pyc's for python2.1-foo in /usr/lib/python should be compiled for. They should _not_ be compiled against the default python, so we can't just compile-all in /usr/lib/python when the default python package changes. It seems each individual package should be responsible for compiling it's own *.py's with an appropriate version of python, even in /usr/lib/python. We can't have the python package handle it directly. Here's a suggestion; Each package that depends on any version of python or pythonX.Y must compile it's own *.py's in its postinst and remove its own *.pyc's in its postrm using a suitable version of python. Packages that support the default python should compile using /usr/bin/python (provided the version is OK). Packages that support multiple pythonX.Y's should compile using that highest installed version of pythonX.Y that they support. The python package's postinst and postrm scripts should find all the installed packages that depend on python and get them to re-run their postinst scripts, either by calling the postinst scripts directly or running dpkg-reconfigure. Each pythonX.Y package must do
Re: version independent pythin packages: ?
On Thu, 2003-08-07 at 03:22, Matthias Urlichs wrote: Hi, Python policy states: 2. A single package for all versions (NOT YET SUPPORTED!) You have a version independent Python module. Create a single package `python-foo' that has a dependency Depends: python It should install modules somewhere inside `/usr/lib/python/site-packages/' and use `#!/usr/bin/python' for programs. The `postinst' script should create symlinks in all `/usr/lib/pythonX.Y/site-packages/' directories that point to its `/usr/lib/python/site-packages/' files and compile them. This may be a dumb question, but why not use /usr/lib/site-python? Since that's in the PYTHONPATH for all Python versions Debian ships with, there should be no requirement to do any post-installation of symlinks. Some .pyc files might possibly be out-of-date, but for reasonably-small packages that won't be much of a problem, IMHO. This is an interesting idea... Python will re-generate pyc files if they are needed, and save them if it can. What this means is the pyc files will be updated every time they are used by a different version of python when the user has write access. If the user doesn't have write access, then everything will still work, but python will re-compile them every time a different version of python is used. Ideally the pyc files in /usr/lib/site-python should be compiled for the default version of Python, and write access to this directory should be limited to minimise overwriting them. This means users without write access to /usr/lib/site-python will use the pyc's when running the default python, and will re-compile them every time they use a non-default python, but everything will still work. The only problem is when someone with write access to /usr/lib/site-python uses a non-default python... the pyc's will be updated for the non-default python. After testing, it seems that there is no way to prevent root from updating the pyc's when using a non-default python. However, even if this does occur, everything will still work, it will just be slightly less optimal when using the default python. The only solution is root should never use the non-default python when importing modules from /usr/lib/site-python... which may or may not be OK. Given the complexity of the alternatives, this is a simple solution that fixes the problem with only minor issues. I'd be tempted to make this the solution and put it into the Python policy. Anyone else agree? Other things to think about; Python applications using the default Python with their own modules not in /usr/lib/site-python... not an issue? Making sure all the pyc's are re-compiled when the default python is updated. A brute force approach is to have the python packages post-inst run dpkg-reconfigure over every package that depends on python, and require that packages recompile their pyc files on a dpkg-reconfigure. This has the advantage of notifying all these packages when the default python has changed so they can do other stuff if they need to. -- Donovan Baarda [EMAIL PROTECTED]
