Re: [gentoo-dev] Thoughts about broken package handling
2011-06-26 13:48:02 Thomas Sachau napisał(a):
In case of slotted dependencies (like python, ruby or php), this would also
allow the user to define
per package, if he wants support for one or more slots of e.g. python.
You can set USE_PYTHON in /etc/portage/env/${CATEGORY}/${PN} or
/etc/portage/env/${CATEGORY}/${PN}:${SLOT}.
--
Arfrever Frehtes Taifersar Arahesis
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-dev] Thoughts about broken package handling
On 26 June 2011 14:59, Stuart Longland redhat...@gentoo.org wrote:
Hi all,
I've been busy for the past month or two, busy updating some of my
systems. In particular, the Yeeloong I have, hasn't seen attention in a
very long time. Soon as I update one part however, I find some swath of
packages break because of a soname change, anything Python-related stops
working because of a move from Python 2.6 to 2.7, or Perl gets updated.
I have a system I use/developed which tries to improve consistency,
but it greatly increases the amount of compile/test runs that get done
in the process, but for my cases that's ok because I *want* things to
be rebuilt needlessly just to make sure they can still be built.
Massive rant follows
I make some assumptions:
Take a timestamp.
Record all packages that need to be updated or reinstalled due to USE changes.
Assume, that there is a chance that any direct dependent of those
packages may become broken by this update, either at install time, or
during runtime ( ie: .so breakages, ABI breakages, $LANGUAGE breakages
etc )
Consider that all these packages were installed before that timestamp.
You can then mostly assume all packages installed after that timestamp
are built with the consideration of the changes that have occurred in
their dependencies.
For the most part, this principle appears to cover a very large range
of scenarios, and is somewhat like a proactive revdep-rebuild that
assumes that every new install/upgrade breaks everything that uses it.
To this end, I have conjured a very Naïve toolset which does what I want it to.
https://github.com/kentfredric/treebuilder
Note a lot of the paths are hardcoded in the source, and its only
really on Github for educational value.
Workflow proceeds as follows: ( nb: I /do/ use paludis, so I'll be
using its terms for the sake of accuracy )
./mk_timestamp.sh # this sets our timestamp values.
echo | rebuild.txt # start over the list of things that need rebuilding
Then I sync portage and overlays. ( I have a custom script that calls
cave sync as well as doing a few other tasks )
{
then I do a deep update of system including new uses but not execute it.
cave resolve -c system
And I record each updated/reinstalled package line-by-line in rebuild.txt
Then perform it:
cave resolve -c system --continue-on-failure if-independent -x
}
and then repeat that with 'world'.
Then I run ./sync.sh , which works out all the Still old packages,
computes some simple regexps' and emits 'rebuilds.out', which is a
list of packages that Might be broken which I'll reinstall just in
case.
VERY Often this is completely needless, ie: -r bumps to kdelibs
triggers me rebuilding everything in KDE, -r bumps to perl causes me
rebuilding every perl module in existence, and everything that uses
perl ( including everything in KDE incidentally , as well as GHC and a
few other very large nasties ).
Once this list is complete, there are 2 approaches I generally take,
1. If the list is small enough, I'll pass the whole thing to cave/paludis.
cave resolve -c -1 $( cat /root/rebuilder/rebuilds.out )
and note record any significant changes ( ie: new uses/updates of
dependendents for things that are orphans for whatever reason )
and then
cave resolve -c -1 $( cat /root/rebuilder/rebuilds.out ) -x
--continue-on-failure if-independent
or
2. If this list looks a bit large, I'll pass the things to reinstall
randomly in groups
dd if=/dev/urandom of=/tmp/rand count=1024 bs=1024 # generate
random file for 'shuf' to produce random but repeatable sort.
cave resolve -c -1 $( shuf --random-source=/tmp/rand -n 200
/root/rebuilder/rebuilds.out )
again, noting updates/etc
cave resolve -c -1 $( shuf --random-source=/tmp/rand -n 200
/root/rebuilder/rebuilds.out ) -x --continue-on-failure
if-independent
After each build run, sync.sh is re-run, updating the list of things
which this code still considers broken, and then I continue the
build/build/build/sync pattern until the build contains all items in
rebuilds.out, and they are all either failing or skipped.
At this point, all the files still listed in rebuilds.out are deemed
somewhat broken. This list is then concatentated with brokens.out
and the process is repeated until all results fail/skip.
then brokens.out and rebuilds.out are concatentated together and
replace broken.txt , which is a list of things to check later.
at this point I consider that merging things are as good as its going
to get, and the entire process starts over, update the timestamp,
sync portage.
Over time, broken.txt adapts itself, growing and shrinking as new
things start being broken due to dependencies, or being resolved due
to fixes entering portage.
Long story short, all of the above is mostly insanity, but its
reasonably successfull. And after all, I am an insane kinda guy =)
--
Kent
perl -e print substr( \edrgmaM SPA NOcomil.ic\\@tfrken\, \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );
Re: [gentoo-dev] Thoughts about broken package handling
chrome://messenger/locale/messengercompose/composeMsgs.properties: Hi all, I've been busy for the past month or two, busy updating some of my systems. In particular, the Yeeloong I have, hasn't seen attention in a very long time. Soon as I update one part however, I find some swath of packages break because of a soname change, anything Python-related stops working because of a move from Python 2.6 to 2.7, or Perl gets updated. Currently we have three packages that handle this separately: - revdep-rebuild (handles packages broken by soname changes, etc) - python-updater (handles Python module rebuilds after upgrading Python) - perl-cleaner (handles Perl module rebuilds after upgrading Perl) I am thinking about a solution for those similar to current ruby idea and already implemented for cross-compilation in my multilib-portage branch of portage. The very short version: Set the needed details in the ebuilds, where needed, in case of revdep-rebuild, either adjust the SLOT var for each change requiring a rebuild of depending packages or using some new var, e.g. API_SLOT for this. Ebuilds depending on packages like python or perl should define the range of versions they support. Now portage generates a (use_expanded) list of USE flags for depending packages, e.g. for a package depending on python-2.6 and 2.7 it adds something like PYTHON_DEPEND=pyhon26 python27 to the list of USE flags. If there is only one dependency installed (like perl or changing libs), this could be a hidden USE flag. When the dependency is now updated, the USE flags will change, so in case of portage, a --newuse will catch those changes and shows those packages in the list of packages, that need to be emerged again. In case of slotted dependencies (like python, ruby or php), this would also allow the user to define per package, if he wants support for one or more slots of e.g. python. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Thoughts about broken package handling
On 06/26/11 15:44, Benedikt Böhm wrote: On Sun, Jun 26, 2011 at 4:59 AM, Stuart Longland redhat...@gentoo.org wrote: - revdep-rebuild (handles packages broken by soname changes, etc) solved by preserved-libs in portage-2.2 Hmmm, except that portage-2.2 isn't stable yet... indeed it isn't even out of alpha yet. Not going to unleash that on my production systems. The reason why some of these things get out of date to this extent is because I'm in the it ain't broke, don't fix it camp much of the time, and so outside the kernel and a few other applications, I just keep things plodding along as they are. I then get bitten a little when I go to update. Unstable portage is kinda an iffy area when dealing with production machines... experimental boxes, no worries... but not on machines I depend on for work. - python-updater (handles Python module rebuilds after upgrading Python) - perl-cleaner (handles Perl module rebuilds after upgrading Perl) these just exist because python and perl ebuilds are horribly broken. take a look at RUBY_TARGETS or PHP_TARGETS for an example of how to do it right. this would also fix all the failures that python and perl introduce to binary packages. Perhaps there is room for improving things there... that's a more long-term solution however. This will require some careful forethought. Modifying the above tools though, to spit out a list of packages, shouldn't be much of a change... and then making a tool that can collate this information and merge it (revdep-rebuild has this code already) shouldn't be that much of a burden to maintain in the short term. -- Stuart Longland (aka Redhatter, VK4MSL) .'''. Gentoo Linux/MIPS Cobalt and Docs Developer '.'` : . . . . . . . . . . . . . . . . . . . . . . .'.' http://dev.gentoo.org/~redhatter :.' I haven't lost my mind... ...it's backed up on a tape somewhere.
Re: [gentoo-dev] Thoughts about broken package handling
On 06/26/11 21:48, Thomas Sachau wrote: I am thinking about a solution for those similar to current ruby idea and already implemented for cross-compilation in my multilib-portage branch of portage. The very short version: Set the needed details in the ebuilds, where needed, in case of revdep-rebuild, either adjust the SLOT var for each change requiring a rebuild of depending packages or using some new var, e.g. API_SLOT for this. Ebuilds depending on packages like python or perl should define the range of versions they support. Now portage generates a (use_expanded) list of USE flags for depending packages, e.g. for a package depending on python-2.6 and 2.7 it adds something like PYTHON_DEPEND=pyhon26 python27 to the list of USE flags. If there is only one dependency installed (like perl or changing libs), this could be a hidden USE flag. When the dependency is now updated, the USE flags will change, so in case of portage, a --newuse will catch those changes and shows those packages in the list of packages, that need to be emerged again. In case of slotted dependencies (like python, ruby or php), this would also allow the user to define per package, if he wants support for one or more slots of e.g. python. This sounds pretty good on the surface... the devil as always is in the details, and I'll have to have a look. I take it though, this would be exclusively tackling the domains of Python and Perl modules? Or does it also tackle ABI breakage of other packages? Sounds like the SLOT variable could get quite unwieldy where several SLOT-ed packages contribute to a package. -- Stuart Longland (aka Redhatter, VK4MSL) .'''. Gentoo Linux/MIPS Cobalt and Docs Developer '.'` : . . . . . . . . . . . . . . . . . . . . . . .'.' http://dev.gentoo.org/~redhatter :.' I haven't lost my mind... ...it's backed up on a tape somewhere.
Re: [gentoo-dev] Thoughts about broken package handling
On 06/26/11 14:27, Stuart Longland wrote: On 06/26/11 15:44, Benedikt Böhm wrote: On Sun, Jun 26, 2011 at 4:59 AM, Stuart Longland redhat...@gentoo.org wrote: - revdep-rebuild (handles packages broken by soname changes, etc) solved by preserved-libs in portage-2.2 Hmmm, except that portage-2.2 isn't stable yet... indeed it isn't even out of alpha yet. Not going to unleash that on my production systems. Here's a little secret: The only code differences between portage 2.1 and 2.2 appear to be one file managing features, so 2.1 just has preserved-rebuild and some of the set support disabled. But if you trust 2.1 enough to use it you also shouldn't have a problem with 2.2. There's still a few silly bugs with preserved-rebuild (corner cases like downgrades and stupid build systems), but I've not had any problematic behaviour in quite a long time ... -- Patrick Lauer http://service.gentooexperimental.org Gentoo Council Member and Evangelist Part of Gentoo Benchmarks, Forensics, PostgreSQL, KDE herds
Re: [gentoo-dev] Thoughts about broken package handling
110626 Stuart Longland wrote re his concept: Tool will be written in separate modules to handle: - ELF soname change breakage - Python module updates - Perl module updates - other checks that can cause broken packages... Each check is run in order, generating a list of packages that should be rebuilt. Having generated this list, it is then evaluated to sort the candidate packages into a suitable order for rebuilding. This is then passed to the package manager... three modes for rebuilds: - All-in-one-hit rebuild: What the tools presently do now. - One-by-one rebuild: For each package in the list, build each one individually... useful if Portage coughs up an error otherwise - Dump the list: allows people to handle it with their own tools From a long-time user : +1 . -- ,, SUPPORT ___//___, Philip Webb ELECTRIC /] [] [] [] [] []| Cities Centre, University of Toronto TRANSIT`-O--O---' purslowatchassdotutorontodotca
Re: [gentoo-dev] Thoughts about broken package handling
Hi, yes, it's no fun to update oldish gentoo Systems - especially ones that should only receive security updates. My bugbear at the moment, is often a package is broken for more than one reason in my situation, and I find myself having to manhandle the package lists generated by the above three, building each package one-by-one, until I manage to rebuild them all. Been there, done that... :/ Concept: Tool will be written in separate modules to handle: - ELF soname change breakage - Python module updates - Perl module updates - other checks that can cause broken packages... Each check is run in order, generating a list of packages that should be rebuilt. If you do a complete update, you will probably update the kernel (maybe only for security reasons), too. In that case, sys-kernel/module-rebuild, might be of interest here, too. It stores a list of packages that compile kernel modules, and those will need recompilation after a kernel update. Best regards, Craig
Re: [gentoo-dev] Thoughts about broken package handling
On Sun, Jun 26, 2011 at 4:59 AM, Stuart Longland redhat...@gentoo.org wrote: - revdep-rebuild (handles packages broken by soname changes, etc) solved by preserved-libs in portage-2.2 - python-updater (handles Python module rebuilds after upgrading Python) - perl-cleaner (handles Perl module rebuilds after upgrading Perl) these just exist because python and perl ebuilds are horribly broken. take a look at RUBY_TARGETS or PHP_TARGETS for an example of how to do it right. this would also fix all the failures that python and perl introduce to binary packages. -Bene
Re: [gentoo-dev] Thoughts about broken package handling
On 26 June 2011 17:44, Benedikt Böhm hol...@gentoo.org wrote: these just exist because python and perl ebuilds are horribly broken. take a look at RUBY_TARGETS or PHP_TARGETS for an example of how to do it right. this would also fix all the failures that python and perl introduce to binary packages. Perl doesn't slot and is presently far too complex to slot it reliably. Ideally we could have Perl slotted, but the effort involved is huge at present. If you're willing to contribute patches to solve this problem you're welcome, but until then, perl-cleaner is really the best we can do. -- Kent perl -e print substr( \edrgmaM SPA NOcomil.ic\\@tfrken\, \$_ * 3, 3 ) for ( 9,8,0,7,1,6,5,4,3,2 ); http://kent-fredric.fox.geek.nz
