RE: [gentoo-user] python, my nemesis
-Original Message- From: Michael Sent: Monday, September 20, 2021 8:21 AM To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] python, my nemesis On Monday, 20 September 2021 15:52:03 BST Gerrit Kuehn wrote: >> >> Just extracting stage3 over everything that is already there? >No, I move out of the way the config/data files I want to keep and move them >back in after untarring the stage 3. A less traumatic place to start I find is to unpack the stage3 into /tmp or wherever it's out of the way and then chroot into it and use quickpkg to bundle up newer versions of blocked core utilities. Not needing build deps can quite often simplify the upgrade path enough to get past certain things. You can also (with a lot of annoyance) use the new portage that's in the tarball to get past eapi restrictions and whatnot and tell it to install the stuff it builds into your original system's folders. I've even used that to salvage systems that got moved to new hardware with an incompatible set of processor flags. Not straightforward since you have to manage both the build environment in the stage3 and the install environment in the original system, but there's very little you can't fix with this approach. Just don't let your system build binary packages for virtual/* under any circumstances. That never ends well. I really need to write up a request to have portage blacklist those by default when buildpkg is enabled... LMP
RE: [gentoo-user] python, my nemesis
-Original Message- From: Neil Bothwick Sent: Monday, September 20, 2021 8:17 AM To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] python, my nemesis On Mon, 20 Sep 2021 15:56:46 +0200, Gerrit Kuehn wrote: >> > > ~ # cat /etc/portage/package.use/py >> > > */* PYTHON_TARGETS: -* python3_9 python3_8 >> > > */* PYTHON_SINGLE_TARGET: -* python3_9 >> > > --- >> >> > You should probably not mess with these variables until after your >> > system is 100% updated and consistent. And even then, probably not. >> > >> > With our package manager written in python, you often need old >> > python stuff to build the new python stuff, and disabling the old >> > python stuff will throw a wrench into that. Even in situations where >> > technically some upgrade path exists, the complexity of the python >> > dependencies often means that the package manager will give up >> > before it finds the solution unless the solution is obvious. By >> > tweaking those variables, you make the solution less obvious to it. >> >> Well, this was the suggested way to go, see >> https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html > >That news item is about going from 3.8 to 3.9, you are on 3.7. I'd try >removing the -* items are trying again. > > When upgrading really old systems I find it useful to run: export PYTHON_TARGETS="$(eix -I | grep -o "python[23]_[0-9]*" | sort -u)" This will temporarily set the system (for the duration of the current session) to build for every version of python that is either available for new builds or currently in use. The dependency resolver seems to have an easier time finding a path when it doesn't think it has to throw away all the currently installed stuff first. Note that this will end up rebuilding a lot more than is technically necessary, and then you'll need to rebuild again with the python versions you actually want and depclean the extras. Personally though I rather prefer to have the computer do more work instead of spending hours using *my* brain to find the minimum sized rebuild set to upgrade the system. LMP
Re: [gentoo-user] python, my nemesis
On Mon, 20 Sep 2021 16:20:58 +0100 Michael wrote: > > Well, yeah, your mileage may vary. > Quite, if you can get your existing installation to only run a > minimal number of rebuilds to arrive at an upgraded toolchain, then > the benefit of reinstalling wouldn't be there. This could have been > the case on an average year, when python deprecations didn't > accelerate and EAPI didn't change. If however you end up tying up > yourself in knots with subsequent python upgrades and difficult to > resolve conflicts, then the pain Vs gain calculus changes. I was already wondering why so many things happened during the (comparatively) short time I didn't watch. Looks like people have been in lockdown with plenty of time to come around with new things. Fortunately, the base installation I have to update doesn't contain many applications (e.g., no X involved). Everything sits on an NFS server with ZFS below it, so it is quite easy to do snaphots and go back and forth between them, and we have plenty to CPU power to rebuild things. OTOH, the orchestration to have the setup re-done in a fully automated way is still under development, so I definitely wanted to try updating before really starting over. cu Gerrit
Re: [gentoo-user] python, my nemesis
On Mon, 20 Sep 2021 16:17:17 +0100 Neil Bothwick wrote: > > Well, this was the suggested way to go, see > > https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html > That news item is about going from 3.8 to 3.9, you are on 3.7. Yes and no. It started with --- Display-If-Installed: dev-lang/python:3.7 Display-If-Installed: dev-lang/python:3.8 --- So my impression was I might find myself in a similar situation with 3.7. > I'd try > removing the -* items are trying again. Same issue (actually, I tried that first and only found the news item after this didn't work). But I think I fixed most issues now (see my previous posts). Thank you for taking time to look into this. cu Gerrit
Re: [gentoo-user] python, my nemesis
On Monday, 20 September 2021 15:52:03 BST Gerrit Kuehn wrote: > On Mon, 20 Sep 2021 15:29:30 +0100 > > Michael wrote: > > Personally, I'd back up /home /etc and world file, plus any databases > > or websites if stored under /var/, then untar the latest stage 3 > > tarball and update @system and @world. > > Just extracting stage3 over everything that is already there? No, I move out of the way the config/data files I want to keep and move them back in after untarring the stage 3. > > Overall it should be a *much* > > faster approach to allow you to bring your installation up to date. > > Well, yeah, your mileage may vary. Quite, if you can get your existing installation to only run a minimal number of rebuilds to arrive at an upgraded toolchain, then the benefit of reinstalling wouldn't be there. This could have been the case on an average year, when python deprecations didn't accelerate and EAPI didn't change. If however you end up tying up yourself in knots with subsequent python upgrades and difficult to resolve conflicts, then the pain Vs gain calculus changes. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] python, my nemesis
On Mon, 20 Sep 2021 15:56:46 +0200, Gerrit Kuehn wrote: > > > ~ # cat /etc/portage/package.use/py > > > */* PYTHON_TARGETS: -* python3_9 python3_8 > > > */* PYTHON_SINGLE_TARGET: -* python3_9 > > > --- > > > You should probably not mess with these variables until after your > > system is 100% updated and consistent. And even then, probably not. > > > > With our package manager written in python, you often need old python > > stuff to build the new python stuff, and disabling the old python > > stuff will throw a wrench into that. Even in situations where > > technically some upgrade path exists, the complexity of the python > > dependencies often means that the package manager will give up before > > it finds the solution unless the solution is obvious. By tweaking > > those variables, you make the solution less obvious to it. > > Well, this was the suggested way to go, see > https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html That news item is about going from 3.8 to 3.9, you are on 3.7. I'd try removing the -* items are trying again. -- Neil Bothwick "We demand rigidly defined areas of doubt and uncertainty!" pgpd48PCoW9g5.pgp Description: OpenPGP digital signature
Re: [gentoo-user] python, my nemesis
On Mon, 20 Sep 2021 10:34:16 -0400 Michael Orlitzky wrote: > > Well, this was the suggested way to go, see > > https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html > It was only suggested for a period of about three weeks: Uh, yes, another thing I overlooked. I should get more sleep, I guess. :) Thanks for pointing this out! cu Gerrit
Re: [gentoo-user] python, my nemesis
On Mon, 20 Sep 2021 16:41:17 +0200 Arve Barsnes wrote: > This is because your gentoolkit installed version has no support for > python 3.9. Yes, I must have been overlooking this all the time. That's why I came here first (more/better eyes on the issue :-). > You might be able to include gentoolkit in your emerge > command, or you might need to build portage with python 3.8 support as > well as 3.9. This type of problem might crop up for several other > packages as you try to get python updated. emerge -p1vUD @world lists about 200 ports to rebuild now, without showing any other issues ahead. I'll go for it (and keep my fingers crossed ;-). Thank you for your support. cu Gerrit
Re: [gentoo-user] python, my nemesis
On Mon, 20 Sep 2021 15:29:30 +0100 Michael wrote: > With a year old system you should question if reinstalling your > system after a back up of configuration and data files would be a > smarter approach. If you *must* upgrade your current installation > for learning or as an experiment, then this is something which has > been done before. I know. I'm a happy Gentoo user for more than 20 years now. However, this is the master installation for a diskless setup providing the baseline for more than a dozen servers. Of course, I can still reinstall but this would be more work than just one machine + a few config files. As one year is not too terribly old (imho) and this looked like something that might be easily solved, I decided to ask for advice here first. > Personally, I'd back up /home /etc and world file, plus any databases > or websites if stored under /var/, then untar the latest stage 3 > tarball and update @system and @world. Just extracting stage3 over everything that is already there? > Overall it should be a *much* > faster approach to allow you to bring your installation up to date. Well, yeah, your mileage may vary. cu Gerrit
Re: [gentoo-user] python, my nemesis
On Mon, 20 Sept 2021 at 16:32, Gerrit Kuehn wrote: > (sys-apps/portage-3.0.4-r1-3:0/0::gentoo, installed) USE="(ipc) > native-extensions rsync-verify xattr -apidoc -build -doc -gentoo-dev > (-selinux) -test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3) > -python3_6 -python3_8 -python3_9" pulled in by > > sys-apps/portage[python_targets_python3_7(-),-python_single_target_pypy3(-),-python_single_target_python3_6(-),-python_single_target_python3_7(-),-python_single_target_python3_8(-)] > required by (app-portage/gentoolkit-0.5.0:0/0::gentoo, installed) > USE="-test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3) > -python3_6 -python3_8" > --- > > > This looks like portage is blocked by itself... so how to solve this > one? This is because your gentoolkit installed version has no support for python 3.9. You might be able to include gentoolkit in your emerge command, or you might need to build portage with python 3.8 support as well as 3.9. This type of problem might crop up for several other packages as you try to get python updated. Cheers, Arve
Re: [gentoo-user] python, my nemesis
On Mon, 20 Sep 2021 16:31:59 +0200 Gerrit Kuehn wrote: > This looks like portage is blocked by itself... so how to solve this > one? Well, I simply installed the new virtual packages manually and used "--nodeps" on portage itself afterwards... and here we are. Looks like new portage will update the rest of my system without any issue now (at least it says so when checking with -p), so here we go. Still no idea why this was blocking so hard, though. cu Gerrit
Re: [gentoo-user] python, my nemesis
On Mon, 2021-09-20 at 15:56 +0200, Gerrit Kuehn wrote: > > Well, this was the suggested way to go, see > https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html It was only suggested for a period of about three weeks: We are planning to switch the default Python target of Gentoo systems on 2021-06-01, from Python 3.8 to Python 3.9. If you have not changed the values of PYTHON_TARGETS or PYTHON_SINGLE_TARGET, the change will have immediate effect on your system and the package manager will try to switch automatically on the next upgrade following the change. > > But also when trying "emerge -1vUD @world" (be it with or without the > package.use settings), I get stuck in conflicts (mostly on perl and > setuptools). perl issues would probably resolve once I have EAPI8 > support, i.e. get new portage. > > You can try --backtrack=200 (or a larger value) to fix that; it was a problem for a lot of people. Even though the end result of both approaches is "it doesn't work," I promise you you're better off not messing with the python stuff until the system is fully upgraded. Upgrading a year-old system is hard enough, and the introduction of EAPI=8 just made it worse.
Re: [gentoo-user] python, my nemesis
On Mon, 20 Sep 2021 09:18:23 -0400 Michael Orlitzky wrote: > With our package manager written in python, you often need old python > stuff to build the new python stuff, and disabling the old python > stuff will throw a wrench into that. Even in situations where > technically some upgrade path exists, the complexity of the python > dependencies often means that the package manager will give up before > it finds the solution unless the solution is obvious. By tweaking > those variables, you make the solution less obvious to it. I took out the extra settings and solved a few conflicts manually. I'm down to this now: --- ~ # emerge -p --oneshot sys-apps/portage --verbose-conflicts These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild N ] acct-group/portage-0 [ebuild N ] acct-user/portage-0 [ebuild U ] sys-apps/portage-3.0.20-r6 [3.0.4-r1] PYTHON_TARGETS="python3_9* (-python3_10)" !!! Multiple package instances within a single package slot have been pulled !!! into the dependency graph, resulting in a slot conflict: sys-apps/portage:0 (sys-apps/portage-3.0.20-r6:0/0::gentoo, ebuild scheduled for merge) USE="(ipc) native-extensions rsync-verify xattr -apidoc -build -doc -gentoo-dev (-selinux) -test" ABI_X86="(64)" PYTHON_TARGETS="python3_9 (-pypy3) (-python3_10) -python3_8" pulled in by sys-apps/portage (Argument) (sys-apps/portage-3.0.4-r1-3:0/0::gentoo, installed) USE="(ipc) native-extensions rsync-verify xattr -apidoc -build -doc -gentoo-dev (-selinux) -test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3) -python3_6 -python3_8 -python3_9" pulled in by sys-apps/portage[python_targets_python3_7(-),-python_single_target_pypy3(-),-python_single_target_python3_6(-),-python_single_target_python3_7(-),-python_single_target_python3_8(-)] required by (app-portage/gentoolkit-0.5.0:0/0::gentoo, installed) USE="-test" ABI_X86="(64)" PYTHON_TARGETS="python3_7 (-pypy3) -python3_6 -python3_8" --- This looks like portage is blocked by itself... so how to solve this one? cu Gerrit
Re: [gentoo-user] python, my nemesis
On Monday, 20 September 2021 14:56:46 BST Gerrit Kuehn wrote: > Well, this was the suggested way to go, see > https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html > > But also when trying "emerge -1vUD @world" (be it with or without the > package.use settings), I get stuck in conflicts (mostly on perl and > setuptools). perl issues would probably resolve once I have EAPI8 > support, i.e. get new portage. > > > cu > Gerrit With a year old system you should question if reinstalling your system after a back up of configuration and data files would be a smarter approach. If you *must* upgrade your current installation for learning or as an experiment, then this is something which has been done before. You will need to walk through all the major upgrades of the last year, by using git to gradually catch up with the latest stable versions. This means a lot of packages will have to be updated a number of times. Personally, I'd back up /home /etc and world file, plus any databases or websites if stored under /var/, then untar the latest stage 3 tarball and update @system and @world. Overall it should be a *much* faster approach to allow you to bring your installation up to date. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] python, my nemesis
On Mon, 20 Sep 2021 09:18:23 -0400 Michael Orlitzky wrote: > > --- > > ~ # cat /etc/portage/package.use/py > > */* PYTHON_TARGETS: -* python3_9 python3_8 > > */* PYTHON_SINGLE_TARGET: -* python3_9 > > --- > You should probably not mess with these variables until after your > system is 100% updated and consistent. And even then, probably not. > > With our package manager written in python, you often need old python > stuff to build the new python stuff, and disabling the old python > stuff will throw a wrench into that. Even in situations where > technically some upgrade path exists, the complexity of the python > dependencies often means that the package manager will give up before > it finds the solution unless the solution is obvious. By tweaking > those variables, you make the solution less obvious to it. Well, this was the suggested way to go, see https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html But also when trying "emerge -1vUD @world" (be it with or without the package.use settings), I get stuck in conflicts (mostly on perl and setuptools). perl issues would probably resolve once I have EAPI8 support, i.e. get new portage. cu Gerrit
Re: [gentoo-user] python, my nemesis
On Mon, 2021-09-20 at 14:50 +0200, Gerrit Kuehn wrote: > Hi, > > I'd like to update a system that is about 1 year old. I have python3.7 > and python 3.8 installed, 3.7 is the default. After updating the repo, > it was suggested to update portage first (probably useful to support > new EAPI versions). > I read about updating to python 3.9, so I created > > --- > ~ # cat /etc/portage/package.use/py > */* PYTHON_TARGETS: -* python3_9 python3_8 > */* PYTHON_SINGLE_TARGET: -* python3_9 > --- You should probably not mess with these variables until after your system is 100% updated and consistent. And even then, probably not. With our package manager written in python, you often need old python stuff to build the new python stuff, and disabling the old python stuff will throw a wrench into that. Even in situations where technically some upgrade path exists, the complexity of the python dependencies often means that the package manager will give up before it finds the solution unless the solution is obvious. By tweaking those variables, you make the solution less obvious to it.