Re: On robustness of maintainer scripts
* Jakub Wilk jw...@debian.org, 2013-05-10, 00:41: prerm = [...] Real-world failure: #706758 Also #708831. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130521120144.ga4...@jwilk.net
Re: On robustness of maintainer scripts
Am 13.05.2013 19:21, schrieb Steve Langasek:
On Sat, May 11, 2013 at 12:29:07AM +0200, Piotr Ożarowski wrote:
[Jakub Wilk, 2013-05-10]
postinst
[...]
Proposed solution: 1) Wait until #671711 is fixed. 3) Make pycompile a
shell script that does only two simple things: - write options it was
called with to a file, say, /var/lib/python/pyX.Ycompile.todo; - use
dpkg-trigger to trigger pythonX.Y.
why not generate maintainer scripts without pycompile at all? I wanted to
delegate as much as possible to interpreter packages, but your idea with
temp. files is even better - maintainer scripts can look like this:
| touch /usr/lib/pythonX.Y/dist-packages/namespace/__init__.py¹ | touch
/usr/lib/pythonX.Z/dist-packages/namespace/__init__.py¹
namespace packages aren't an issue anymore in python3, and for python2 it
would be better to ship these files into separate binary packages. these
aren't that many, and it would be much better for the robustness. nothing to
do anymore about these in maintainer scripts.
| dpkg -L package | grep \.py$ | while read file | do |echo ${file}
/var/lib/python/pyX.Ycompile.todo | echo ${file}
/var/lib/python/pyX.Zcompile.todo | done
The disadvantage is that the more logic is included directly in the
maintainer scripts, the harder it is to fix any bugs with that logic
because every package that includes the buggy behavior needs to be fixed.
Even if that's only a simple rebuild with an updated version of dh_python2,
it's quite costly to do that over all the affected packages in the
archive.
So from a robustness standpoint, it would be preferable if this logic
could be encapsulated, if not in pycompile (for the reasons described),
then at least in a trigger.
If it needs to be encapsulated, then it should be in the pythonx.y
packages shipping the interpreters, even if it does mean to duplicate these
scripts up to two times.
My hope was that most of the code for pycompile would end up in py_compile.py,
and the pycompile script would rely on that. But that didn't happen.
Depending on the complexity of the resulting pycompile, it could be inlined in
maintainer scripts, but it doesn't have to be.
For my part, I'm having a hard time understanding why any of the proposed
changes here are warranted. Yes, there are corner cases where the current
maintainer scripts will fail, but the real-world failures included in
Jakub's original mail seem to reduce to bug #680930 (which I think it was
reasonable to fix by changing the interpreter package's linkage) and bug
#671711 (which is an obviously unacceptable bug in dpkg that needs to be
fixed). The other bugs, such as bug #706758, seem to map to these
directly (i.e., it's fixable by applying the same change for bug #680930 to
the python2.6 package in wheezy - currently this fix has only been applied
for python2.7); and I don't think that any problem that requires a command
like this in a minimal reproducer case:
# dpkg --force-depends -r libssl0.9.8
... should seriously be considered a high priority, and be allowed to
dictate the design of the interpreter handling.
Relying on packages to be usable when unpacked-but-not-configured is
fragile. But I am not convinced that the proposed cures are not worse
than the disease.
Certainly if we want to continue using pythonX.Y from maintainer scripts
the way we do currently, great care would need to be taken to ensure
they're usable when unpacked - which means a committment to either not add
new library dependencies to the interpreter or ensure new library deps are
listed in pre-depends instead of depends. Yet I think this should be
entertained as an alternative to increasing the amount of code duplicated
across hundreds of maintainer scripts.
python2.7-minimal shouldn't get any more dependencies on libraries. Now the
version in unstable depends on libc6, libgcc1, zlib1g. The libgcc1 dependency
comes from a linker bug when built with -flto and is dropped once linked with
ld 2.23.x. zlib is needed in the interpreter for the zip file support. From
what I can see, it would be hard to drop. libc6 is interesting in that the
update to 2.17 causes #708831, which I now worked around by making libc6 a
pre-dependency. I don't think this is specific for python2.7-minimal, and will
be exposed by other packages as well, as long as some essential packages like
coreutils and perl are rebuilt against the new libc6.
Afaicr, adding trigger support in some helpers did introduce its own set of
upgrade issues, but this might be safe, if no helpers use triggers to generate
and update symlink farms.
I don't share the opinion on the severity of #709198, and how it should be
fixed, if we are going this route of keeping the python interpreter working
during upgrades.
The dependency of python2.7-minimal on libpython2.7-minimal looks safe to me,
I don't think it needs to be raised to a pre-dependency.
Matthias
--
To
Re: On robustness of maintainer scripts
* Matthias Klose d...@debian.org, 2013-05-21, 17:55: The dependency of python2.7-minimal on libpython2.7-minimal looks safe to me, At least in theory, it's not safe. If python2.7-minimal is unpacked, but libpython2.7-minimal is not (yet) unpacked, then all pyclean can do is: Could not find platform independent libraries prefix Consider setting $PYTHONHOME to prefix[:exec_prefix] ImportError: No module named site dpkg: error while cleaning up: subprocess installed post-installation script returned error exit status 1 -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130521173556.ga5...@jwilk.net
Re: On robustness of maintainer scripts
* Matthias Klose d...@debian.org, 2013-05-21, 17:55: namespace packages aren't an issue anymore in python3, and for python2 it would be better to ship these files into separate binary packages. these aren't that many, and it would be much better for the robustness. nothing to do anymore about these in maintainer scripts. We have about ~60 packages which ship files in /usr/share/python/ns/, which is quite a bit. That said, most of them don't need __init__.py at all, because the namespace would be created at runtime via *-nspkg.pth files. The only namespaces for which appropriate *-nspkg.pth don't exist are: * flaskext (python-flaksext.wtf): flaskext/__init__.py is already shipped by python-flask, so not a problem. * r2 (python-radare2): no other users of the namespace AFAICS, so it could ship __init__.py directly in .deb. * dummy (python-van.pydeb) * zope.app (python-van.pydeb) - these two look like a bug in dh_python2. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130521180705.ga8...@jwilk.net
Re: On robustness of maintainer scripts
On Tue, May 21, 2013 at 05:55:58PM +0200, Matthias Klose wrote:
| dpkg -L package | grep \.py$ | while read file | do | echo ${file}
/var/lib/python/pyX.Ycompile.todo | echo ${file}
/var/lib/python/pyX.Zcompile.todo | done
The disadvantage is that the more logic is included directly in the
maintainer scripts, the harder it is to fix any bugs with that logic
because every package that includes the buggy behavior needs to be fixed.
Even if that's only a simple rebuild with an updated version of dh_python2,
it's quite costly to do that over all the affected packages in the
archive.
So from a robustness standpoint, it would be preferable if this logic
could be encapsulated, if not in pycompile (for the reasons described),
then at least in a trigger.
If it needs to be encapsulated, then it should be in the pythonx.y
packages shipping the interpreters, even if it does mean to duplicate these
scripts up to two times.
That makes sense to me. Would that code be able to live in a trigger?
(Bearing in mind that a trigger can't doesn't get told *where* the new files
are)
# dpkg --force-depends -r libssl0.9.8
... should seriously be considered a high priority, and be allowed to
dictate the design of the interpreter handling.
Relying on packages to be usable when unpacked-but-not-configured is
fragile. But I am not convinced that the proposed cures are not worse
than the disease.
Certainly if we want to continue using pythonX.Y from maintainer scripts
the way we do currently, great care would need to be taken to ensure
they're usable when unpacked - which means a committment to either not add
new library dependencies to the interpreter or ensure new library deps are
listed in pre-depends instead of depends. Yet I think this should be
entertained as an alternative to increasing the amount of code duplicated
across hundreds of maintainer scripts.
python2.7-minimal shouldn't get any more dependencies on libraries. Now
the version in unstable depends on libc6, libgcc1, zlib1g. The libgcc1
dependency comes from a linker bug when built with -flto and is dropped
once linked with ld 2.23.x. zlib is needed in the interpreter for the zip
file support. From what I can see, it would be hard to drop. libc6 is
interesting in that the update to 2.17 causes #708831, which I now worked
around by making libc6 a pre-dependency. I don't think this is specific
for python2.7-minimal, and will be exposed by other packages as well, as
long as some essential packages like coreutils and perl are rebuilt
against the new libc6.
Essential packages are *required* to deal with library dependencies as
pre-dependencies. So this is specific to python, in the sense that python
is being called from maintainer scripts before the package has been
configured.
I think the pre-depends is the right change here; but as Aurelien points
out, pre-dependencies are supposed to be discussed on debian-devel because
they can have system-wide repercussions. Can you please kick off a thread
to discuss this there?
I don't share the opinion on the severity of #709198, and how it should be
fixed, if we are going this route of keeping the python interpreter working
during upgrades.
Having the debconf package break during upgrades because it tries to call
python and fails looks like a pretty serious bug to me. I'm not sure about
what the right way to *fix* it is, but we definitely don't want users'
upgrades to blow up...
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
signature.asc
Description: Digital signature
Re: On robustness of maintainer scripts
[Jakub Wilk, 2013-05-12]
* Piotr Ożarowski pi...@debian.org, 2013-05-11, 00:29:
why not generate maintainer scripts without pycompile at all?
- There are hundreds of packages that use py{compile,clean} in the
maintainer script. The script needs to continue to exist at least
until jessie is released.
true, but we will get rid of it in jessie+1, not in jessie
- Fixing the problem in py{compile,clean} means that even the
packages that haven't been rebuilt can benefit from the fix.
that's why I want to keep it simple in maintainer scripts. dh_python2
knows for which versions given directory has to be bytecompiled, so why
not hardcode these directories in maintainer scripts (and use dpkg -L
package | grep dir to fill todo files).
- If you take all corner cases into cosideration, the required gets
pretty complicated. IMO too complicated to risk putting the code
directly into maintainer scripts.
if we get rid of exclude patterns (and not a single package in unstable
provides a file in /usr/share/python/bcep/ directory, so it's not really
needed apparently) there are not that many features in pycompile...
Maintainer scripts can simply create missing namespaces and then let
interpreter packages know which files they should bytecompile (by
creating todo files for X.Y or default interpreter).
pycompileX.Y (provided by interpreter package and invoked by dpkg
trigger) can read such files and bytecompile all files listed there
(rt* scripts can do similar thing)
[...]
Here you need to check with versions are supported, and which of them
are installed^Wunpacked.
Oh, and the call to dpkg-trigger is missing. /o\
shebang is missing too! ... you know it was just an example to
demonstrate the idea and not the final code, right? :)
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130513093816.gd24...@p1otr.com
Re: On robustness of maintainer scripts
On Sat, May 11, 2013 at 12:29:07AM +0200, Piotr Ożarowski wrote:
[Jakub Wilk, 2013-05-10]
postinst
[...]
Proposed solution:
1) Wait until #671711 is fixed.
3) Make pycompile a shell script that does only two simple things:
- write options it was called with to a file, say,
/var/lib/python/pyX.Ycompile.todo;
- use dpkg-trigger to trigger pythonX.Y.
why not generate maintainer scripts without pycompile at all?
I wanted to delegate as much as possible to interpreter packages, but
your idea with temp. files is even better - maintainer scripts can
look like this:
| touch /usr/lib/pythonX.Y/dist-packages/namespace/__init__.py¹
| touch /usr/lib/pythonX.Z/dist-packages/namespace/__init__.py¹
| dpkg -L package | grep \.py$ | while read file
| do
| echo ${file} /var/lib/python/pyX.Ycompile.todo
| echo ${file} /var/lib/python/pyX.Zcompile.todo
| done
The disadvantage is that the more logic is included directly in the
maintainer scripts, the harder it is to fix any bugs with that logic because
every package that includes the buggy behavior needs to be fixed. Even if
that's only a simple rebuild with an updated version of dh_python2, it's
quite costly to do that over all the affected packages in the archive.
So from a robustness standpoint, it would be preferable if this logic could
be encapsulated, if not in pycompile (for the reasons described), then at
least in a trigger.
For my part, I'm having a hard time understanding why any of the proposed
changes here are warranted. Yes, there are corner cases where the current
maintainer scripts will fail, but the real-world failures included in
Jakub's original mail seem to reduce to bug #680930 (which I think it was
reasonable to fix by changing the interpreter package's linkage) and bug
#671711 (which is an obviously unacceptable bug in dpkg that needs to be
fixed). The other bugs, such as bug #706758, seem to map to these directly
(i.e., it's fixable by applying the same change for bug #680930 to the
python2.6 package in wheezy - currently this fix has only been applied for
python2.7); and I don't think that any problem that requires a command like
this in a minimal reproducer case:
# dpkg --force-depends -r libssl0.9.8
... should seriously be considered a high priority, and be allowed to
dictate the design of the interpreter handling.
Relying on packages to be usable when unpacked-but-not-configured is
fragile. But I am not convinced that the proposed cures are not worse than
the disease.
Certainly if we want to continue using pythonX.Y from maintainer scripts the
way we do currently, great care would need to be taken to ensure they're
usable when unpacked - which means a committment to either not add new
library dependencies to the interpreter or ensure new library deps are
listed in pre-depends instead of depends. Yet I think this should be
entertained as an alternative to increasing the amount of code duplicated
across hundreds of maintainer scripts.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
signature.asc
Description: Digital signature
Re: On robustness of maintainer scripts
* Jakub Wilk jw...@debian.org, 2013-05-10, 00:41: postinst [...] Test-case (in a squeeze chroot, with wheezy added to sources.lists): # dpkg --unpack cpuset_1.5.6-2_all.deb # apt-get install -t squeeze python2.6 # apt-get install -t wheezy python gcc-4.4 # dpkg --force-depends -r libssl0.9.8 # dpkg -i python-pkg-resources_0.6.24-1_all.deb Blargh, I suck at copyingpasting: g/cpuset/d prerm = [...] Test case (in wheezy chroot): # apt-get install -q python-pkg-resources # dpkg --force-depends -r libssl0.9.8 # dpkg -r python-ipaddr s/wheezy/squeeze/ s/ipaddr/pkgresources/ A different test case (in squeeze chroot): # apt-get install python-pkg-resources # dpkg --unpack --auto-deconfigure python-minimal_2.7.3-4_all.deb # dpkg -r python-pkg-resources -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130513223139.ga8...@jwilk.net
Re: On robustness of maintainer scripts
* Steve Langasek vor...@debian.org, 2013-05-13, 12:21:
why not generate maintainer scripts without pycompile at all? I wanted
to delegate as much as possible to interpreter packages, but your idea
with temp. files is even better - maintainer scripts can look like this:
| touch /usr/lib/pythonX.Y/dist-packages/namespace/__init__.py¹
| touch /usr/lib/pythonX.Z/dist-packages/namespace/__init__.py¹
| dpkg -L package | grep \.py$ | while read file
| do
| echo ${file} /var/lib/python/pyX.Ycompile.todo
| echo ${file} /var/lib/python/pyX.Zcompile.todo
| done
The disadvantage is that the more logic is included directly in the
maintainer scripts, the harder it is to fix any bugs with that logic
because every package that includes the buggy behavior needs to be
fixed. Even if that's only a simple rebuild with an updated version of
dh_python2, it's quite costly to do that over all the affected packages
in the archive.
Hear, hear. We had a Python helper that put too much logic directly into
maintainer scripts once: it was python-central in lenny. Don't let such
a disaster happen again. Thanks. :)
--
Jakub Wilk
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130513223338.gb8...@jwilk.net
RE : On robustness of maintainer scripts
Hello Jackub prerm = ... Real-world failure: #706758 should I reassign this bug to python ? If I understand well, all this, python should be fixed then a rebuild of all python packages should be done. right ? which bug of python should I track to know when a rebuild of the package should fix the issue ? Fred -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/a2a20ec3b8560d408356cac2fc148e5358e90...@sun-dag1.synchrotron-soleil.fr
Re: On robustness of maintainer scripts
* Piotr Ożarowski pi...@debian.org, 2013-05-11, 00:29:
postinst
[...]
Proposed solution:
1) Wait until #671711 is fixed.
3) Make pycompile a shell script that does only two simple things:
- write options it was called with to a file, say,
/var/lib/python/pyX.Ycompile.todo;
- use dpkg-trigger to trigger pythonX.Y.
why not generate maintainer scripts without pycompile at all?
- There are hundreds of packages that use py{compile,clean} in the
maintainer script. The script needs to continue to exist at least until
jessie is released.
- Fixing the problem in py{compile,clean} means that even the packages
that haven't been rebuilt can benefit from the fix.
- If you take all corner cases into cosideration, the required gets
pretty complicated. IMO too complicated to risk putting the code
directly into maintainer scripts.
I wanted to delegate as much as possible to interpreter packages, but
your idea with temp. files is even better - maintainer scripts can
look like this:
| touch /usr/lib/pythonX.Y/dist-packages/namespace/__init__.py¹
| touch /usr/lib/pythonX.Z/dist-packages/namespace/__init__.py¹
| dpkg -L package | grep \.py$ | while read file
| do
| echo ${file} /var/lib/python/pyX.Ycompile.todo
| echo ${file} /var/lib/python/pyX.Zcompile.todo
| done
Here you need to check with versions are supported, and which of them
are installed^Wunpacked.
Oh, and the call to dpkg-trigger is missing. /o\
[¹] namespace has to be created here as other packages might want to
use given library in maintainer scipts, even without .pyc files
ACK
(test -f ... || is probably missing here)
Yes, it is missing. :)
prerm
=
[...]
Proposed solution:
Get rid of /usr/bin/pyclean, so that the fallback code is activated.
\o/ Just kidding. :) Not only that would almost certainly have
unintended consequences, but also the fallback code doesn't look
complete: it doesn't take care of namespaces.
Actual proposed solution:
Rewrite /usr/bin/pyclean in shell.
again, removing /usr/bin/pyclean is not that bad idea. I like it and
will try to provide example scripts for interpreter packages (and then
generate prerm like in postinst example)
You need to remove namespaces, remove files from *.todo...
I wouldn't dare to put it directly to prerm.
--
Jakub Wilk
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130512205130.ga2...@jwilk.net
Re: On robustness of maintainer scripts
* PICCA Frédéric-Emmanuel frederic-emmanuel.pi...@synchrotron-soleil.fr, 2013-05-12, 08:22: prerm = ... Real-world failure: #706758 should I reassign this bug to python ? I think so. (But maybe there's flaw a in my reasoning. Please shout you if this is the case. :P) -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130512205508.gb2...@jwilk.net
Re: On robustness of maintainer scripts
[Jakub Wilk, 2013-05-10]
postinst
[...]
Proposed solution:
1) Wait until #671711 is fixed.
3) Make pycompile a shell script that does only two simple things:
- write options it was called with to a file, say,
/var/lib/python/pyX.Ycompile.todo;
- use dpkg-trigger to trigger pythonX.Y.
why not generate maintainer scripts without pycompile at all?
I wanted to delegate as much as possible to interpreter packages, but
your idea with temp. files is even better - maintainer scripts can
look like this:
| touch /usr/lib/pythonX.Y/dist-packages/namespace/__init__.py¹
| touch /usr/lib/pythonX.Z/dist-packages/namespace/__init__.py¹
| dpkg -L package | grep \.py$ | while read file
| do
| echo ${file} /var/lib/python/pyX.Ycompile.todo
| echo ${file} /var/lib/python/pyX.Zcompile.todo
| done
[¹] namespace has to be created here as other packages might want to use
given library in maintainer scipts, even without .pyc files
(test -f ... || is probably missing here)
(with some extra code to handle private directories, including
rt* scripts, or any other logic from pycompile script, probably without
exclude patterns or other features that are not really used)
2) Add code to pythonX.Y's postinst that'll process stuff from
/var/lib/python/pyX.Ycompile.todo. Note that you don't need to move
any complicated logic to the interpreter packages. All that the
postinst would have to do is to call some script shipped in
python-minimal.
prerm
=
[...]
Proposed solution:
Get rid of /usr/bin/pyclean, so that the fallback code is activated. \o/
Just kidding. :) Not only that would almost certainly have unintended
consequences, but also the fallback code doesn't look complete: it
doesn't take care of namespaces.
Actual proposed solution:
Rewrite /usr/bin/pyclean in shell.
again, removing /usr/bin/pyclean is not that bad idea. I like it and
will try to provide example scripts for interpreter packages
(and then generate prerm like in postinst example)
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130510222907.gb24...@p1otr.com
Re: On robustness of maintainer scripts
[Piotr Ożarowski, 2013-05-11]
| dpkg -L package | grep \.py$ | while read file
| do
| echo ${file} /var/lib/python/pyX.Ycompile.todo
| echo ${file} /var/lib/python/pyX.Zcompile.todo
| done
with some grep .../pythonX.Y/dist-packages of course
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130510223413.gc24...@p1otr.com
