Re: [gentoo-user] Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 3/8/21 7:30 PM, John Covici wrote: At least I didn't have to change profiles and gcc versions several times. I didn't /change/ the profile. As in it was 17.0 when I started and still is 17.0. I did have to update the make.profile link to point to the same profile in the alternate portage directory. This wouldn't have been an issue if I just re-used the same portage directory. I guess different situations require different methods. Indeed. -- Grant. . . . unix || die
Re: [gentoo-user] Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On Mon, 08 Mar 2021 18:37:21 -0500, Grant Taylor wrote: > > On 3/8/21 4:03 PM, Grant Edwards wrote: > > How do you feel it compares to just installing from scratch > > while preserving whatever config and user data you care about? > > I've done that quite a few times and it usually takes about 2-3 > > hours for the initial install and then overnight to build a > > desktop environment (if one is needed). > > I feel like installing from scratch misses a lot of things. Even > wholesale overwriting the new /etc with the old /etc is > questionable. You'd have to make sure that all the same software > was installed. > > Aside: I've spent too much time around other SAs that would > ""recover a down server by doing fresh installs in hours and then > spending weeks to get everything back to the way that it needed > to be verses spending ~18 hours to restore from tape and have > things work the way they were 24 hours prior. I also never cared > for in place upgrades (installing over top of itself) in the > Windows world. > > I feel *MUCH* /more/ comfortable with what I did than other > solutions. I trust that this is the same install with patches > applied. I couldn't and wouldn't say the same for an > installation over the top or fresh installation. > > Don't get me wrong. I believe there are places for fresh > installations. They usually happen coordinate with new machines > and / or new drives for me. > > After all, I effectively have the same thing that I would have if > I had done updates over the last year like they should have been > done. > > hmmm, I had to do a sort of fresh install, I did it on my running system and I did it because portage and other stuff was messed up in some way I could not fix, so I did the install, copied parts of my /etc, emerged part of my world file -- that needed cleaning up too -- and kept doing this till I was done and then copied from my chroot to my main system. I don't necessarily recomend this, but it did get things cleaned up and working again. At least I didn't have to change profiles and gcc versions several times. I guess different situations require different methods. -- Your life is like a penny. You're going to lose it. The question is: How do you spend it? John Covici wb2una cov...@ccs.covici.com
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 08/03/2021 23:16, Neil Bothwick wrote: I don't remember what it was at the start, probably 8. or 9.. I did see 9.3 somewhere along the way. gcc -v says that 10.2.0 is currently installed. It means you probably spent a lot of time compile gcc versions only to carry on using the old version, but as you said, this wasn't about efficiency. You were going to emerge -e @world at the end anyway, which would get everything built with the latest toolchain. As I remember, you always had to use eselect to switch versions ... and witness all the chaos with python at the moment ... If you leave things "at the default", doesn't that screw you over when python/kernel/gcc etc upgrade and a depclean deletes your original default version? Or is that now fixed so you can't mess things up that way? Cheers, Wol
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 3/8/21 5:35 PM, Neil Bothwick wrote: Not if you went up a slot, then the old version would still continue to be used until you ran gcc-config. However, if you were depcleaning at each step, that would remove the previous slot and you would stay current. So my overall method, which included depclean, did allow subsequent runs to use the new updated GCC. Thank you for clarifying. I tend to keep old copies of gcc around until I'm sure things play nicely with the new version: % gcc-config -l [1] x86_64-pc-linux-gnu-9.3.0 [2] x86_64-pc-linux-gnu-10.2.0 * [1] aarch64-unknown-linux-gnu-9.3.0 * (cyan *) [2] x86_64-pc-linux-gnu-10.2.0 * (green *) The aarch64* came in as part of @openwrt-prerequisites. I should probably remove that as I no longer need it. Thank you for your input Neil. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On Mon, 8 Mar 2021 16:44:35 -0700, Grant Taylor wrote: > > It means you probably spent a lot of time compile gcc versions only > > to carry on using the old version, but as you said, this wasn't about > > efficiency. > > Wouldn't the next execution of gcc, post Emerge & Installation use the > newly emerged binary? > > If not next package in a given emerge run didn't use the new gcc, I > would fully expect that subsequent emerges would use the new gcc. Not if you went up a slot, then the old version would still continue to be used until you ran gcc-config. However, if you were depcleaning at each step, that would remove the previous slot and you would stay current. I tend to keep old copies of gcc around until I'm sure things play nicely with the new version: % gcc-config -l [1] x86_64-pc-linux-gnu-9.3.0 [2] x86_64-pc-linux-gnu-10.2.0 * -- Neil Bothwick Psychiatrists say that 1 of 4 people are mentally ill. Check three friends. If they're OK, you're it. pgp1liLQnwr6t.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 3/8/21 4:16 PM, Neil Bothwick wrote: It would have to be done before the first update, when the repo was set to a date just after the last update. Yes and no. It really could have been done at any point along the way. Also, with the git version of the portage repo, I could switch back to the branch from any time I wanted to. You can rephrase that as "I left it at the default", which is an acceptable answer :) *nod* It means you probably spent a lot of time compile gcc versions only to carry on using the old version, but as you said, this wasn't about efficiency. Wouldn't the next execution of gcc, post Emerge & Installation use the newly emerged binary? If not next package in a given emerge run didn't use the new gcc, I would fully expect that subsequent emerges would use the new gcc. You were going to emerge -e @world at the end anyway, which would get everything built with the latest toolchain. Yes. I have initiated a full system backup. I'll start an `emerge -e @world` after that finishes. I'll actually do the full suite: 1) emerge -e @world 2) emerge --depclean --verbose n 3) emerge @preserved-rebuild 4) revdep-rebuild I expect that #3 should be a NoOp and just burn CPU cycles. I don't know anything else that can be done to make a Gentoo box happier (from a software standpoint). Most of the effort for you was developing the procedure. All the real effort was left to the computer. Exactly! Well, developing the method /and/ establishing trust therein. I was thinking of a week max. I suspect that would be quite safe. -- Grant. . . . unix || die
Re: [gentoo-user] Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 3/8/21 4:03 PM, Grant Edwards wrote: How do you feel it compares to just installing from scratch while preserving whatever config and user data you care about? I've done that quite a few times and it usually takes about 2-3 hours for the initial install and then overnight to build a desktop environment (if one is needed). I feel like installing from scratch misses a lot of things. Even wholesale overwriting the new /etc with the old /etc is questionable. You'd have to make sure that all the same software was installed. Aside: I've spent too much time around other SAs that would ""recover a down server by doing fresh installs in hours and then spending weeks to get everything back to the way that it needed to be verses spending ~18 hours to restore from tape and have things work the way they were 24 hours prior. I also never cared for in place upgrades (installing over top of itself) in the Windows world. I feel *MUCH* /more/ comfortable with what I did than other solutions. I trust that this is the same install with patches applied. I couldn't and wouldn't say the same for an installation over the top or fresh installation. Don't get me wrong. I believe there are places for fresh installations. They usually happen coordinate with new machines and / or new drives for me. After all, I effectively have the same thing that I would have if I had done updates over the last year like they should have been done. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On Mon, 8 Mar 2021 15:44:38 -0700, Grant Taylor wrote: > On 3/8/21 3:29 PM, Neil Bothwick wrote: > > With hindsight, removing firefox, thunderbird and libreoffice and > > replacing them with their -bin counterparts at the start of the > > process would have saved much time. You could switch back to the > > source options once the system is up to date. > > You're probably correct. > > However I don't think I would do that even if I had known / thought > about doing so. Partially because changing things was questionable at > / near the start and partially because this was about possibility, not > efficiency.It would ve to be done It would have to be done before the first update, when the repo was set to a date just after the last update. > > How did you manage gcc upgrades, did you run gcc-config manually > > whenever gcc was updated? > > Is "I ignored them and let emerge deal with it" count? I did see gcc > upgrades along the way. You can rephrase that as "I left it at the default", which is an acceptable answer :) > I don't remember what it was at the start, probably 8. or > 9.. I did see 9.3 somewhere along the way. gcc -v says > that 10.2.0 is currently installed. It means you probably spent a lot of time compile gcc versions only to carry on using the old version, but as you said, this wasn't about efficiency. You were going to emerge -e @world at the end anyway, which would get everything built with the latest toolchain. > > Do you feel it was worth the effort of updating for every day of the > > git history? > > I don't know if it was worth the effort or not. I initially did one > day at a time while testing the waters and going from theory to some > practical experience of the method. > > Very quickly I used a different version of e (1 or 2) that took the > date as a parameter. My command line was calling e with the date > derived from the d variable and then decrementing the d variable after > the e function finished. I.e. > > e $(date +%Y-%m-%d -d "$d days ago"); $((d=$d-1)) > > I would let that run, deal with any results, then hit the up arrow and > enter. > > I just let that process continue for a while. Then at some point I > optimized it into e3 and ran that for a while. Then I optimized that > into the while e3; do true; done loop. Most of the effort for you was developing the procedure. All the real effort was left to the computer. > But I stuck with single day steps mostly from inertia. It was working. > So ... stick with it. > > > Would a larger increment have saved time, or did you think minimising > > the number of issues to deal with after each emerge was more > > important? > > Maybe. If anything, it would have saved the time for emerge to process > all of it's meta data. Much like an initial emerge vs an emerge > --resume. But again, this was about the viability of the process, not > the efficiency thereof. > > I probably could have gone with a week at a time. I don't know if that > would have helped or not. I don't think I would go with more than a > week with a largely automated process. I was thinking of a week max. -- Neil Bothwick Distrust any enterprise that requires new clothes. - Henry David Thoreau (1817-1862) pgp7BOgh2uzc5.pgp Description: OpenPGP digital signature
[gentoo-user] Re: What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 2021-03-08, Grant Taylor wrote: > On 2/25/21 5:31 PM, Grant Taylor wrote: >> 10 have git switch to the next day >> 20 emerge -aDUN @world >> 30 assess / deal with masked packages >> 40 goto 10 >> >> It /looks/ like things are working. > > *TL;DR* > > DenverCoder9: DEAR PEOPLE FROM THE FUTURE ... > > This method /does/ work. I have successfully brought the problem system > from ~1 year old to ~current (Gentoo portage repo < 24 hours old). How do you feel it compares to just installing from scratch while preserving whatever config and user data you care about? I've done that quite a few times and it usually takes about 2-3 hours for the initial install and then overnight to build a desktop environment (if one is needed). -- The other Grant
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 3/8/21 3:29 PM, Neil Bothwick wrote: With hindsight, removing firefox, thunderbird and libreoffice and replacing them with their -bin counterparts at the start of the process would have saved much time. You could switch back to the source options once the system is up to date. You're probably correct. However I don't think I would do that even if I had known / thought about doing so. Partially because changing things was questionable at / near the start and partially because this was about possibility, not efficiency. How did you manage gcc upgrades, did you run gcc-config manually whenever gcc was updated? Is "I ignored them and let emerge deal with it" count? I did see gcc upgrades along the way. I don't remember what it was at the start, probably 8. or 9.. I did see 9.3 somewhere along the way. gcc -v says that 10.2.0 is currently installed. Do you feel it was worth the effort of updating for every day of the git history? I don't know if it was worth the effort or not. I initially did one day at a time while testing the waters and going from theory to some practical experience of the method. Very quickly I used a different version of e (1 or 2) that took the date as a parameter. My command line was calling e with the date derived from the d variable and then decrementing the d variable after the e function finished. I.e. e $(date +%Y-%m-%d -d "$d days ago"); $((d=$d-1)) I would let that run, deal with any results, then hit the up arrow and enter. I just let that process continue for a while. Then at some point I optimized it into e3 and ran that for a while. Then I optimized that into the while e3; do true; done loop. But I stuck with single day steps mostly from inertia. It was working. So ... stick with it. Would a larger increment have saved time, or did you think minimising the number of issues to deal with after each emerge was more important? Maybe. If anything, it would have saved the time for emerge to process all of it's meta data. Much like an initial emerge vs an emerge --resume. But again, this was about the viability of the process, not the efficiency thereof. I probably could have gone with a week at a time. I don't know if that would have helped or not. I don't think I would go with more than a week with a largely automated process. I think one month increments probably would be pushing the envelope. I feel like some of the Python changes were 2 or maybe 3 months apart. So with two combined with how you landed, you might cross Python 3.6 w/o 3.7, to both 3.6 and 3.7, to w/o 3.6 and w/ 3.7 barrier. That probably wouldn't be pretty. Anyway, glad it worked for you - it's more or less how I would have approached it but never had to, so thanks for doing the legwork :) You're welcome. Hence the DenverCoder9 comment, for people searching ~> reading the mailing list archive in the future. -- Grant. . . . unix || die
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On Mon, 8 Mar 2021 15:06:01 -0700, Grant Taylor wrote: > The following packages take what seems like F O R E V E R to emerge: > > - gcc > - rust > - Firefox > - Thunderbird With hindsight, removing firefox, thunderbird and libreoffice and replacing them with their -bin counterparts at the start of the process would have saved much time. You could switch back to the source options once the system is up to date. How did you manage gcc upgrades, did you run gcc-config manually whenever gcc was updated? Do you feel it was worth the effort of updating for every day of the git history? Would a larger increment have saved time, or did you think minimising the number of issues to deal with after each emerge was more important? Anyway, glad it worked for you - it's more or less how I would have approached it but never had to, so thanks for doing the legwork :) -- Neil Bothwick WinErr 013: Unexpected error - Huh ? pgpPxSJHUiuBX.pgp Description: OpenPGP digital signature
Re: [gentoo-user] What is the best way forward? - Update 2 - SUCCESS! - CURRENT!!!
On 2/25/21 5:31 PM, Grant Taylor wrote: 10 have git switch to the next day 20 emerge -aDUN @world 30 assess / deal with masked packages 40 goto 10 It /looks/ like things are working. *TL;DR* DenverCoder9: DEAR PEOPLE FROM THE FUTURE ... This method /does/ work. I have successfully brought the problem system from ~1 year old to ~current (Gentoo portage repo < 24 hours old). *Speed Bumps* These were the four things that caused the biggest slow down in this process. 1) Source packages / ebuilds no longer available. - I found and downloaded files to DISTDIR. - I copied some ebuilds from older versions of portage to my local repo. 2) make.profile not using PORTDIR definition in make.conf. - I ran into this while working on October ~ November '20 updates. 3) PYTHON_TARGETS & PYTHON_SINGLE_TARGET - I ran into this after fixing #2. - I had to add the following to pull Python 3.6 back in so that things would work to add Python 3.7, before allowing the system to remove Python 3.6 (again). PYTHON_TARGETS="python2_7 python3_6 python3_7" PYTHON_SINGLE_TARGET="python3_7" 4) Firefox & Thunderbird 68 disliking rust ≈ 1.48. - I had to give up on retaining version 68 of Firefox and Thunderbird. - The loss of some important extensions still really hurts. *How* The high level process that I used is a very close superset of what I hypothesized. 10 have git switch to the next day 20 emerge -DUN @world 21 emerge --depclean --verbose n 22 emerge @preserved-rebuild 23 revdep-rebuild 30 assess / deal with output from steps 20-23 40 goto 10 Steps 21-23 added mid-stream to make comparison to previous message simpler. All of these steps were in a function, `e3` (see attached file), which relied on one variable, `d`, the count of how many days to go backwards and set the date (`D`) to that everything should act on. Aside: The next version of e3 would probably store `d` in a file and subsequently re-load it from said file on each invocation. Thus eliminating the reliance on the environment variable. I would probably store this file in /var/tmp as /tmp and /dev/shm are cleared on boot. After gaining enough trust in the overall process, I ended up running the following while loop: while e3; true; done This allowed the system to stay busy emerging things up to the point that something failed and needed attention. *Setup* I did a `git clone` of the Gentoo portage repo. Currently ~6 GB. I then created the branches in the git repo with the following command (from inside of the git repo directory): for ((age=1; age<1024; age++)); do eval $(printf 'git log --pretty=format:"%%H %%cd" --date=format:%%Y-%%m-%%d\ %%H:%%M:%%S --after=%s --before=%s | fgrep -m1 %s' $(date +%Y-%m-%d -d "$(($age + 1)) days ago") $(date +%Y-%m-%d -d "$(($age - 1)) days ago") $(date +%Y-%m-%d -d "$age days ago")) | read hash date time; time git checkout -b $date $hash; done Basically, this command starts at current; `stable`, and finds the first (most recent) commit for a given date and creates a branch, and works backwards for however many days configured; 1024 in the example. *Miscellany* I did `emerge -e @world` 3~5 times throughout the process just to make sure that everything was consistent. I will do this once more tomorrow after a full backup runs tonight. I did end up removing a small list of packages that were blocking emerge in one way or another. -- I decided that removing them to allow emerge to complete on it's own accord was more expedient than fighting them at the time. I will re-add them as necessary. - net-firewall/nftables - net-fs/ncpfs - media-gfx/gimp - dev-python/pycairo - dev-python/fido2 - net-analyzer/scapy - app-crypt/yubikey-manager Some of the packages were subsequently pulled back in. I did run into a bug with app-misc/pax-utils where I needed to add "-seccomp" for the package to be able to move forward. I also did the /usr/portage to /var/db/repos/gentoo et al. migration. "repo" can be ambiguous when there talking about both "Gentoo portage repo" and "git repo". Especially when the latter is managing the former. The following packages take what seems like F O R E V E R to emerge: - gcc - rust - Firefox - Thunderbird Link - xkcd - Wisdom of the Ancients (a.k.a. DenverCoder9) - https://xkcd.com/979/ *Summary* Yes, there are probably faster and / or more efficient processes to get a Gentoo system that's ~1 year behind caught up to current. But I did learn some things along the way. -- I tried to outline the toe stubbers so others can avoid them. Ultimately, I believe I have done in the last 11 days what would have been done over the course of the last ~year. Even 11 days is longer than necessary as I started with the while loop after getting to January of this year. In hindsight, I believe I could have used the while loop all along. I hope that I have
Re: [gentoo-user] Q: What is "python-exec2c"? Why would "python3" dispatched through it not see an installed copy of pyyaml?
On Sun, 7 Mar 2021 18:52:50 -0500, Steven Lembark wrote: > > > Q: Is there no way to have a consistent version of Python on > > >the system? > > > > Yes, make sure PYTHON_TARGETS and your chosen version of python > > match. > > Q: How do I know which verson of python is suitable? > > I never deal with the language... last I saw was some news that > turn off the targets would be preferable. Is there some real > advantage to targets vs target (i.e., at this point is it reasonable > to just have a single target)? In that case, you had no reason to use eselect to switch away from the default. Switch back to 3.8 with eselect and all should be well - at least after an emerge -auDN @world > I'm still not sure, however, why a module installed with python 3.8 > would leave portage disfunctional if that version were selected. That does seem odd, were any dependent modules also installed for 3.8. -- Neil Bothwick Documentation: (n.) a novel sold with software, designed to entertain the operator during episodes of bugs or glitches. pgpTDutIam21I.pgp Description: OpenPGP digital signature
