Re: [gentoo-user] another old box to update
On 08.01.2015 00:02, Alan McKinnon wrote: On 07/01/2015 22:30, Stefan G. Weichinger wrote: Am 07.01.2015 um 20:06 schrieb Tomas Mozes: Strange, I only have successful stories with upgrading old gentoo machines. If you have a machine which you update regularly then you know all the issues during the time and so upgrading per partes leads to no surprises but the same challenges you've handled before. But yes, it takes time. Moreover, if you use configuration management like Ansible, you can even automatically merge changes when applications ship new configuration. Thanks for that posting, it reminds me of some bigger issue I wanted to discuss here for quite a while now. Over the years I am now responsible for dozens of servers and VMs running gentoo linux ... and I wonder how to efficiently keep track of them. I learned my first steps with puppet and use it in a basic setup for my own machines in my LAN. It seems to work better for many identical servers, let's say in a hosting environment. The servers at my customers are somehow similar but not identical: different setups for services ... different update-cycles (which have to be synchronized and shortened as we have seen in this thread!) ... I look for a way of tracking all these systems: a) central database/repo with all the systems and how to access them: * unique system id * what IP, port, ssh-key, etc etc I use git for local tracking of /etc on most of my systems in the last years, but I did never really come up with a clever way how to centralize dozens of separate git-repos ... one repo per server pushed to one central git-home on of my inhouse servers? b) in addition tracking of let's say rules or services: * which server runs e.g. apache? So if there is a new security warning out there for apache ... ask system which servers/customers would need that update? etc etc c) when was my last access to that server? Have I looked into it lately? (or more business-oriented:) Do I even have to / does the customer pay for that?) This should lead to some SLA-kind-of-thing, yes ... a bit off-topic for now. - Puppet is more oriented to push configs out to systems. Maybe a combination would apply ... puppet for building the basement, having stuff generalized (this path, that password/ssh-key ) and some other components to track what has been done over time. I run OTRS ( http://en.wikipedia.org/wiki/OTRS ) for my daily work and looked into their module ITSM ( https://www.otrs.com/homepage/software/otrsitsm-features/ ) lately ... it allows to create configuration items (think: ITIL) etc, so far I think this is a bit of overkill and not really fitting the size of my business. I'd love to keep it simple and CLI-oriented: Gentoo allows to define (nearly?) everything via text-files, combined with the cleverness of git (and maybe puppet) this should give me a way of a) easily deploy new systems with configs according to some standards: I want these packages/users/paths/files ... b) track these systems: what boxes am I responsible for, what is out there and failing? ;-) (not talking monitoring here ... just what are my active systems out there) from there I should slowly get into defining new contracts with my clients including regular checks each 3 or 6 months ... what has to be done, are there any bigger updates to do (think udev, baselayout ...) and tell them if is possible to update the box within a few hours in parallel to normal work or if we need a bigger maintenance window. --- I am sure there are many other gentoo-users out there with similar challenges to face. And I am looking forward to your thoughts, experiences and recommendations! In my opinion, ansible almost always beats puppet. Puppet is a) complex b) built to be able to deal with vast enterprise setups and c) has a definition language I never could wrap my brains around. It always felt to me like puppet was never a good fit for my needs. Then ansible hit the scene. Now ansible is designed to do what sysadmins do anyway, to do it in mostly the same way, and to do it in an automated fashion. It fits nicely into my brain and I can read a playbook at a glance to know what it does. Ansible deploys stuff and makes it like how you want it to be. It is equally good at managing 1000 identical machines as 100 mostly the same ones, or 20 totally different ones. How it manages this miracle is not easy to explain, so I urge you to give it a test drive. Fool around with a bunch of VMs to get a feel of how it works. A tip: keep it simple, and use roles everywhere. Ansible works nicely with other tools like vagrant and docker which build and deploy your base images. Once you have a base machine with sshd running and an administrative login account, ansible takes over an manages the rest. It works really well. played around with ansible today and managed to get this
Re: [gentoo-user] another old box to update
On 2015-01-07 12:52, Stefan G. Weichinger wrote: I am in the process of upgrading an old (~2010) gentoo server. The customer never wanted updates ... and now he wants ... *sigh* I managed to compile basic stuff already ... portage, gcc etc Now I get errors at emerging packages which is bad. Still no openrc installed and the udev-upgrade also is still pending. gcc-4.8.3 When I try to emerge grep for example, the config.log says: configure:4451: $? = 0 configure:4440: i686-pc-linux-gnu-gcc -v 5 Using built-in specs. COLLECT_GCC=/usr/i686-pc-linux-gnu/gcc-bin/4.8.3/i686-pc-linux-gnu-gcc COLLECT_LTO_WRAPPER=/usr/libexec/gcc/i686-pc-linux-gnu/4.8.3/lto-wrapper Target: i686-pc-linux-gnu Configured with: /var/tmp/portage/sys-devel/gcc-4.8.3/work/gcc-4.8.3/configure --host=i686-pc-linux-gnu --build=i686-pc-linux-gnu --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc-bin/4.8.3 --includedir=/usr/lib/gcc/i686-pc-linux-gnu/4.8.3/include --datadir=/usr/share/gcc-data/i686-pc-linux-gnu/4.8.3 --mandir=/usr/share/gcc-data/i686-pc-linux-gnu/4.8.3/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/4.8.3/info --with-gxx-include-dir=/usr/lib/gcc/i686-pc-linux-gnu/4.8.3/include/g++-v4 --with-python-dir=/share/gcc-data/i686-pc-linux-gnu/4.8.3/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.8.3 p1.1, pie-0.5.9' --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --disable-multilib --disable-altivec --disable-fixed-point --with-arch=i686 --enable-targets=all --disable-libgcj --enable-libgomp --disable-libmudflap --disable-libssp --enable-lto --without-cloog --enable-libsanitizer Thread model: posix gcc version 4.8.3 (Gentoo 4.8.3 p1.1, pie-0.5.9) configure:4451: $? = 0 configure:4440: i686-pc-linux-gnu-gcc -V 5 i686-pc-linux-gnu-gcc: error: unrecognized command line option '-V' i686-pc-linux-gnu-gcc: fatal error: no input files compilation terminated. configure:4451: $? = 1 configure:4440: i686-pc-linux-gnu-gcc -qversion 5 i686-pc-linux-gnu-gcc: error: unrecognized command line option '-qversion' i686-pc-linux-gnu-gcc: fatal error: no input files compilation terminated. configure:4451: $? = 1 configure:4440: i686-pc-linux-gnu-gcc -version 5 i686-pc-linux-gnu-gcc: error: unrecognized command line option '-version' i686-pc-linux-gnu-gcc: fatal error: no input files compilation terminated. configure:4451: $? = 1 configure:4471: checking whether the C compiler works configure:4493: i686-pc-linux-gnu-gcc -O2 -march=prescott -pipe -fomit-frame-pointer conftest.c 5 i686-pc-linux-gnu-gcc: error trying to exec 'as': execvp: No such file or directory configure:4497: $? = 2 configure:4535: result: no I understand that configure is called with wrong parameters ... where do they come from? -- For reference: $ emerge --info Portage 2.2.14 (python 2.7.9-final-0, default/linux/x86/13.0, gcc-4.8.3, glibc-2.11.2-r3, 2.6.36-gentoo-r5 i686) = System uname: Linux-2.6.36-gentoo-r5-i686-Intel-R-_Xeon-R-_CPU_X3430_@_2.40GHz-with-gentoo-2.2 KiB Mem: 2065232 total,944892 free KiB Swap:1044216 total, 1044104 free Timestamp of tree: Fri, 02 Jan 2015 17:30:01 + ccache version 2.4 [disabled] app-shells/bash: 4.2_p53 dev-lang/perl:5.12.3-r1 dev-lang/python: 2.6.4-r1, 2.7.9-r1, 3.3.5-r1 dev-util/ccache: 2.4-r9 dev-util/cmake: 2.6.4-r3 dev-util/pkgconfig: 0.28-r1 sys-apps/baselayout: 2.2 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.9.6-r2::unknown repository, 1.11.1, 1.13.4 sys-devel/binutils: 2.24-r3 sys-devel/gcc:4.3.4, 4.4.4-r2, 4.8.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool:2.4.2-r1 sys-devel/make: 4.0-r1 sys-kernel/linux-headers: 3.16 (virtual/os-headers) sys-libs/glibc: 2.11.2-r3 Repositories: gentoo local_overlay ACCEPT_KEYWORDS=x86 ACCEPT_LICENSE=* -@EULA CBUILD=i686-pc-linux-gnu CFLAGS=-O2 -march=prescott -pipe -fomit-frame-pointer CHOST=i686-pc-linux-gnu CONFIG_PROTECT=/etc CONFIG_PROTECT_MASK=/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo CXXFLAGS=-O2 -march=prescott -pipe -fomit-frame-pointer DISTDIR=/usr/portage/distfiles FCFLAGS=-O2 -march=i686 -pipe FEATURES=assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync FFLAGS=-O2 -march=i686 -pipe
Re: [gentoo-user] another old box to update
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Am 07.01.2015 um 13:13 schrieb Neil Bothwick:
On Wed, 07 Jan 2015 13:01:34 +0100, Tomas Mozes wrote:
Try to fetch some older portage snapshots
http://dev.gentoo.org/~swift/snapshots/ and update in steps, not
as a 4 year giant leap. Try to fetch snapshot 2011, then upgrade,
then 2012, upgrade... It takes more time, but it should work.
It may be quicker to backup the data and configs and start a fresh
install. Five years is a long time for any server to go without
updates, ${DEITY} knows what has happened to it in that time.
Sure, I know. I always hesitate to do that ... although I have
up-to-date installations at hand (as images or VMs).
The box is quite remote from me ... and they don't accept much or any
downtime (and they work with their files on the samba-shares while I
hack around ... )
I am fighting my way through all the conflicts now ... should I force
glibc with --nodeps or is it a bad idea?
;-)
(util-linux doesn't install ... and I think it needs a fresh glibc)
S
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
iQIcBAEBAgAGBQJUrSwwAAoJEClcuD1V0Pzm6McQAIJn4MlIxC3xiWxnWzzV5GUR
9t4ELxrVtfKO7meA4deVJKYIJYjBWLV+rKeIZeqd1yoefXilwVga1S8j0BKLSCYF
K4olq8bIzQi6sInj6he4nkh2TXLZklcVdx7fcMfMbFstX/llm25axjNphxFIjL/M
NNnUvN/4DAkhG5/xquWRdXuLEIo+56JGDADm+TRffQsqwzSfkl1Y453RrDm1uab9
naK+xHM9fB7xO9VXzUj0Gif9MColexKcFxRc8rllI3hRdkJvwg/fz/ZOO1nOdrod
iSzI0koOWgGtx0eR3+fPMCXjjsx3abcSCy7t3f/J86qvT5qN/gGBtpKUBdrT06PH
Gmo84je6OU9rjhmklN6rExYAfK+Hh5RHFNwFXCzD2hKzUZsE/98DytLypn55vZ2V
Z82ufjU2Mr15aGR23I5EbEqc18DCYMo3rMyNiJ9i26J8CmWyngbI89COPT5uW+wZ
RfGynU7T04Ear2aBGZLg6yMSDnzbjbvlc9Qz7Hemt3LXHucsViQbxBau3vydPXB+
YN0MkEfo3x6lDslUYn/bElQh2DRpWfU0cMFML9w1l3FZoLlsdWyijsdDqVBsNXsR
GB10WA+ue394k8vkfWM51JronhC7v47hq/JIrfr3+pW8JgcH8GqvBDc9CjsTQoBc
rVZQicBjKfEoxVOUwOgG
=abus
-END PGP SIGNATURE-
Re: [gentoo-user] another old box to update
Am 07.01.2015 um 12:52 schrieb Stefan G. Weichinger: Thanks for any pointers! I *think* I solved it by fixing the binutils-setting ... just testing ... seems solved for now! Stefan
Re: [gentoo-user] another old box to update
Am 07.01.2015 um 14:06 schrieb Alan McKinnon: The tricky one is going to be that persistent interface names from udev 18 months or so back. When you get to that, you'll probably want to re-read the huge threads from that time, as you only get one chance to get it right. One addition: at the reboot time a fellow IT-guy will be there in front of the console so if the NIC doesn't come up correctly I will be able to instruct him to get the box up and reachable for me. I also use to disable persistent names for such updates ... and get good old eth0 UP instead of enpXsY unconfigured ;-) I see in your reply to Neil you have glibc conflicts. I don't know what will happen if you do it with --nodeps, but I wouldn't try that. The box is remote, if something goes wrong... Rather go with Tomas' suggestion of yearly portage snapshots and update in stages. skip that ... I leave glibc for now and focus on udev and openrc, see below. openrc should be seamless. I forget the exact timelines, but IIRC you will also hit baselayout-2 migration. That one was very smooth and well documented so you shouldn't have much trouble. openrc needs an updated udev and I am currently working on getting at least sys-fs/udev-208-r1 installed now (conflicts with lvm2 and stuff ...) Some circular dep between udev and udev-init-scripts blocks things ... I fiddled with this and now decided to gor for a --nodeps ... while I type this udev, lvm2 and udev-init-scripts get emerged OK now. Phew. openrc installs now as well ... so now for dispatch-conf and disabling the persistent nic names Seems as if the biggest problems are solved right now? S
Re: [gentoo-user] another old box to update
On Wed, Jan 7, 2015 at 8:06 AM, Alan McKinnon alan.mckin...@gmail.com wrote: openrc should be seamless. I forget the exact timelines, but IIRC you will also hit baselayout-2 migration. That one was very smooth and well documented so you shouldn't have much trouble. If it is already running openrc then he is past that migration. If it isn't, then it isn't exactly seamless. You might want to give thought in that case to whether long-term you want to be on openrc or systemd. It is probably as much work to migrate to either. However, I suspect that you're already on openrc in which case I wouldn't try to throw a change like that into the mix - you'll be lucky enough to get the thing to boot keeping things the same. I didn't see this suggestion yet. Build binary packages of everything from a new install, and update using those. Updating using a binary package is far more likely to work. Just create a stage3 chroot, set your use flags the way you want them, do an emerge -e @system with buildpkg turned on inside, then migrate those packages to your host and after a backup do an emerge -uk @system. Once you get the system set updated most of the rest should go fairly smoothly, but you could extend this to other packages. You don't necessarily need to do the first pass update with your final use flags either - you could just install binary packages with generic use flags and then just do an emerge -uN world to rebuild everything now that you're past the hump. -- Rich
Re: [gentoo-user] another old box to update
Am 07.01.2015 um 14:28 schrieb Rich Freeman: On Wed, Jan 7, 2015 at 8:06 AM, Alan McKinnon alan.mckin...@gmail.com wrote: openrc should be seamless. I forget the exact timelines, but IIRC you will also hit baselayout-2 migration. That one was very smooth and well documented so you shouldn't have much trouble. If it is already running openrc then he is past that migration. I have it merged now ... but I haven't yet booted using it and won't be until jan. 20th according to the plans. I was informed that the amanda backup threw an error after the first updates (I did basic stuff like portage, glib and gcc a few days ago) so I was forced to repair that today ... and this lead to my current work session. btw amanda works already ;-) If it isn't, then it isn't exactly seamless. You might want to give thought in that case to whether long-term you want to be on openrc or systemd. It is probably as much work to migrate to either. However, I suspect that you're already on openrc in which case I wouldn't try to throw a change like that into the mix - you'll be lucky enough to get the thing to boot keeping things the same. I won't go systemd there with the current installation. Rather reinstall as mentioned ... the current system is still 32bit! S
Re: [gentoo-user] another old box to update
On 07/01/2015 15:19, Stefan G. Weichinger wrote: Am 07.01.2015 um 14:06 schrieb Alan McKinnon: The tricky one is going to be that persistent interface names from udev 18 months or so back. When you get to that, you'll probably want to re-read the huge threads from that time, as you only get one chance to get it right. One addition: at the reboot time a fellow IT-guy will be there in front of the console so if the NIC doesn't come up correctly I will be able to instruct him to get the box up and reachable for me. I also use to disable persistent names for such updates ... and get good old eth0 UP instead of enpXsY unconfigured ;-) I see in your reply to Neil you have glibc conflicts. I don't know what will happen if you do it with --nodeps, but I wouldn't try that. The box is remote, if something goes wrong... Rather go with Tomas' suggestion of yearly portage snapshots and update in stages. skip that ... I leave glibc for now and focus on udev and openrc, see below. openrc should be seamless. I forget the exact timelines, but IIRC you will also hit baselayout-2 migration. That one was very smooth and well documented so you shouldn't have much trouble. openrc needs an updated udev and I am currently working on getting at least sys-fs/udev-208-r1 installed now (conflicts with lvm2 and stuff ...) Some circular dep between udev and udev-init-scripts blocks things ... I fiddled with this and now decided to gor for a --nodeps ... while I type this udev, lvm2 and udev-init-scripts get emerged OK now. Phew. openrc installs now as well ... so now for dispatch-conf and disabling the persistent nic names Seems as if the biggest problems are solved right now? if you ran emerge -avuND world and portage goes ahead and does it without blockers, then I'd agree - the major problems are solved. It's a simple samba server, so not too many packages. I'd recommend you do emerge -e world when done just to ensure that everything is consistently built against the same toolchain and glibc. It's not absolutely required, but it does give peace of mind and you can let it run and do it's thing while you get on with something else -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] another old box to update
On 07/01/2015 21:06, Tomas Mozes wrote: On 2015-01-07 13:47, Alan McKinnon wrote: On 07/01/2015 13:52, Stefan G. Weichinger wrote: I am in the process of upgrading an old (~2010) gentoo server. The customer never wanted updates ... and now he wants ... *sigh* Don't waste your time (you are already experiencing the full reason why). Backup data and configs, reinstall Gentoo, restore data and configs. Downtime? Of course. A few hours. Customer needs to understand he brought this upon himself. Trying to do it in-place will likely takes *days* and fill you with pain and mucho downtime. This list, the forum, and planet are full of horror stories of what it takes to do it and the issues you will run into. Frankly, you do not need to prove you can do it (we know you can), and you have much better things to do with your time (like proper billable hours). It's worth repeating: the customer caused this, he must now feel the pain and not you. Strange, I only have successful stories with upgrading old gentoo machines. If you have a machine which you update regularly then you know all the issues during the time and so upgrading per partes leads to no surprises but the same challenges you've handled before. But yes, it takes time. I also have had good success with this, 100% in fact. As a technical challenge it gives you awesome brownie points with geeky colleagues and folk who understand what magic you had to do to succeed. I like that feeling as much as the next Gentooer. But Stefan is quite clear, this is a business arrangement. The manager in charge probably couldn't care less about Stefan's skill in updating a 4 year old install. He probably wants heap, correct and fast and isn't interested in picking any two. Easiest route, the one of least pain is almost always a reinstall. Moreover, if you use configuration management like Ansible, you can even automatically merge changes when applications ship new configuration. If the box hasn't been updated in 4 years, it hasn't been touched in all that time either. It's virtually a certainty that Ansible came nowhere near it. CM-managed machines tend to be well maintained and updated, or short-lived (i.e. cattle) -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] another old box to update
Am 08.01.2015 um 00:02 schrieb Alan McKinnon: In my opinion, ansible almost always beats puppet. Puppet is a) complex b) built to be able to deal with vast enterprise setups and c) has a definition language I never could wrap my brains around. It always felt to me like puppet was never a good fit for my needs. Then ansible hit the scene. Now ansible is designed to do what sysadmins do anyway, to do it in mostly the same way, and to do it in an automated fashion. It fits nicely into my brain and I can read a playbook at a glance to know what it does. Ansible deploys stuff and makes it like how you want it to be. It is equally good at managing 1000 identical machines as 100 mostly the same ones, or 20 totally different ones. How it manages this miracle is not easy to explain, so I urge you to give it a test drive. Fool around with a bunch of VMs to get a feel of how it works. A tip: keep it simple, and use roles everywhere. Ansible works nicely with other tools like vagrant and docker which build and deploy your base images. Once you have a base machine with sshd running and an administrative login account, ansible takes over an manages the rest. It works really well. Thanks for the pointer, Alan. I will look into ansible asap ... installed it on my basement server already, but it's rather late in my $TIMEZONE ... On the business side of things, yes indeed you need to rationalize things and what you offer to customers. There comes a point where you the business grows and you just can't manage all these different things. Mistakes get made, SLAs slip, and everyone gets annoyed. exactly. $NEXTSTEP. On how to track all that real-world data you mentioned, I have a few rules of my own. Monitor and track everything I can, get and store as much info out of logs as I can. All the info you need is in there somewhere. But how to get it is a problem highly specific to your business. Maybe start some new threads each one with a specific question, and watch for common solutions in the answers? will do as soon as I am there, yes. S
Re: [gentoo-user] another old box to update
On 07/01/2015 22:30, Stefan G. Weichinger wrote: Am 07.01.2015 um 20:06 schrieb Tomas Mozes: Strange, I only have successful stories with upgrading old gentoo machines. If you have a machine which you update regularly then you know all the issues during the time and so upgrading per partes leads to no surprises but the same challenges you've handled before. But yes, it takes time. Moreover, if you use configuration management like Ansible, you can even automatically merge changes when applications ship new configuration. Thanks for that posting, it reminds me of some bigger issue I wanted to discuss here for quite a while now. Over the years I am now responsible for dozens of servers and VMs running gentoo linux ... and I wonder how to efficiently keep track of them. I learned my first steps with puppet and use it in a basic setup for my own machines in my LAN. It seems to work better for many identical servers, let's say in a hosting environment. The servers at my customers are somehow similar but not identical: different setups for services ... different update-cycles (which have to be synchronized and shortened as we have seen in this thread!) ... I look for a way of tracking all these systems: a) central database/repo with all the systems and how to access them: * unique system id * what IP, port, ssh-key, etc etc I use git for local tracking of /etc on most of my systems in the last years, but I did never really come up with a clever way how to centralize dozens of separate git-repos ... one repo per server pushed to one central git-home on of my inhouse servers? b) in addition tracking of let's say rules or services: * which server runs e.g. apache? So if there is a new security warning out there for apache ... ask system which servers/customers would need that update? etc etc c) when was my last access to that server? Have I looked into it lately? (or more business-oriented:) Do I even have to / does the customer pay for that?) This should lead to some SLA-kind-of-thing, yes ... a bit off-topic for now. - Puppet is more oriented to push configs out to systems. Maybe a combination would apply ... puppet for building the basement, having stuff generalized (this path, that password/ssh-key ) and some other components to track what has been done over time. I run OTRS ( http://en.wikipedia.org/wiki/OTRS ) for my daily work and looked into their module ITSM ( https://www.otrs.com/homepage/software/otrsitsm-features/ ) lately ... it allows to create configuration items (think: ITIL) etc, so far I think this is a bit of overkill and not really fitting the size of my business. I'd love to keep it simple and CLI-oriented: Gentoo allows to define (nearly?) everything via text-files, combined with the cleverness of git (and maybe puppet) this should give me a way of a) easily deploy new systems with configs according to some standards: I want these packages/users/paths/files ... b) track these systems: what boxes am I responsible for, what is out there and failing? ;-) (not talking monitoring here ... just what are my active systems out there) from there I should slowly get into defining new contracts with my clients including regular checks each 3 or 6 months ... what has to be done, are there any bigger updates to do (think udev, baselayout ...) and tell them if is possible to update the box within a few hours in parallel to normal work or if we need a bigger maintenance window. --- I am sure there are many other gentoo-users out there with similar challenges to face. And I am looking forward to your thoughts, experiences and recommendations! In my opinion, ansible almost always beats puppet. Puppet is a) complex b) built to be able to deal with vast enterprise setups and c) has a definition language I never could wrap my brains around. It always felt to me like puppet was never a good fit for my needs. Then ansible hit the scene. Now ansible is designed to do what sysadmins do anyway, to do it in mostly the same way, and to do it in an automated fashion. It fits nicely into my brain and I can read a playbook at a glance to know what it does. Ansible deploys stuff and makes it like how you want it to be. It is equally good at managing 1000 identical machines as 100 mostly the same ones, or 20 totally different ones. How it manages this miracle is not easy to explain, so I urge you to give it a test drive. Fool around with a bunch of VMs to get a feel of how it works. A tip: keep it simple, and use roles everywhere. Ansible works nicely with other tools like vagrant and docker which build and deploy your base images. Once you have a base machine with sshd running and an administrative login account, ansible takes over an manages the rest. It works really well. On the business side of things, yes indeed you need to rationalize things and what you offer
Re: [gentoo-user] another old box to update
Am 07.01.2015 um 20:06 schrieb Tomas Mozes: Strange, I only have successful stories with upgrading old gentoo machines. If you have a machine which you update regularly then you know all the issues during the time and so upgrading per partes leads to no surprises but the same challenges you've handled before. But yes, it takes time. Moreover, if you use configuration management like Ansible, you can even automatically merge changes when applications ship new configuration. Thanks for that posting, it reminds me of some bigger issue I wanted to discuss here for quite a while now. Over the years I am now responsible for dozens of servers and VMs running gentoo linux ... and I wonder how to efficiently keep track of them. I learned my first steps with puppet and use it in a basic setup for my own machines in my LAN. It seems to work better for many identical servers, let's say in a hosting environment. The servers at my customers are somehow similar but not identical: different setups for services ... different update-cycles (which have to be synchronized and shortened as we have seen in this thread!) ... I look for a way of tracking all these systems: a) central database/repo with all the systems and how to access them: * unique system id * what IP, port, ssh-key, etc etc I use git for local tracking of /etc on most of my systems in the last years, but I did never really come up with a clever way how to centralize dozens of separate git-repos ... one repo per server pushed to one central git-home on of my inhouse servers? b) in addition tracking of let's say rules or services: * which server runs e.g. apache? So if there is a new security warning out there for apache ... ask system which servers/customers would need that update? etc etc c) when was my last access to that server? Have I looked into it lately? (or more business-oriented:) Do I even have to / does the customer pay for that?) This should lead to some SLA-kind-of-thing, yes ... a bit off-topic for now. - Puppet is more oriented to push configs out to systems. Maybe a combination would apply ... puppet for building the basement, having stuff generalized (this path, that password/ssh-key ) and some other components to track what has been done over time. I run OTRS ( http://en.wikipedia.org/wiki/OTRS ) for my daily work and looked into their module ITSM ( https://www.otrs.com/homepage/software/otrsitsm-features/ ) lately ... it allows to create configuration items (think: ITIL) etc, so far I think this is a bit of overkill and not really fitting the size of my business. I'd love to keep it simple and CLI-oriented: Gentoo allows to define (nearly?) everything via text-files, combined with the cleverness of git (and maybe puppet) this should give me a way of a) easily deploy new systems with configs according to some standards: I want these packages/users/paths/files ... b) track these systems: what boxes am I responsible for, what is out there and failing? ;-) (not talking monitoring here ... just what are my active systems out there) from there I should slowly get into defining new contracts with my clients including regular checks each 3 or 6 months ... what has to be done, are there any bigger updates to do (think udev, baselayout ...) and tell them if is possible to update the box within a few hours in parallel to normal work or if we need a bigger maintenance window. --- I am sure there are many other gentoo-users out there with similar challenges to face. And I am looking forward to your thoughts, experiences and recommendations! Best regards, Stefan
Re: [gentoo-user] another old box to update
Am 07.01.2015 um 17:08 schrieb Rich Freeman: On Wed, Jan 7, 2015 at 7:47 AM, Alan McKinnon alan.mckin...@gmail.com wrote: It's worth repeating: the customer caused this, he must now feel the pain and not you. So, if he made an informed choice and that is what he chose, then that is how it has to be. However, if I were in the position of supporting this installation I'd probably give some thought as to what the right tools for the job are. Gentoo simply isn't designed to be updated twice a decade. If you REALLY want that kind of deployment you should probably be running something like RHEL, which will commercially support your old install for years and help you with any issues with migration (but there aren't likely to be many, since they will have tested moving from one 5-year-old release to the next 5-year-supported one. Debian stable or CentOS are of course free alternatives, or Ubuntu LTS. I love Gentoo, and I think it is the right tool for a lot of jobs, but it isn't always the right tool for EVERY job. If it really is just one box and you don't mind dealing with the downtime/hassle/etc twice a decade then by all means use Gentoo. However, if I were managing 100 of these this is not how I'd want to be doing it. 100% Yes here. See my other posting right now for what my current position is around these issues. Stefan
Re: [gentoo-user] another old box to update
Am 07.01.2015 um 13:47 schrieb Alan McKinnon: On 07/01/2015 13:52, Stefan G. Weichinger wrote: I am in the process of upgrading an old (~2010) gentoo server. The customer never wanted updates ... and now he wants ... *sigh* Don't waste your time (you are already experiencing the full reason why). Backup data and configs, reinstall Gentoo, restore data and configs. Downtime? Of course. A few hours. Customer needs to understand he brought this upon himself. Trying to do it in-place will likely takes *days* and fill you with pain and mucho downtime. This list, the forum, and planet are full of horror stories of what it takes to do it and the issues you will run into. Frankly, you do not need to prove you can do it (we know you can), and you have much better things to do with your time (like proper billable hours). It's worth repeating: the customer caused this, he must now feel the pain and not you. I should print this and learn this at last, right! Thanks for the clear opinion at least I will be able to bill the hours and I am quite sure that I am not that far from success it's a rather simple samba-server ... right now I spent about 3 hrs of time over the last week or so. What's left? migration to openrc, new udev ... kernel is prepared ... then the test of rebooting on their closed day (they don't work on tuesdays). S
Re: [gentoo-user] another old box to update
On Wed, Jan 7, 2015 at 7:47 AM, Alan McKinnon alan.mckin...@gmail.com wrote: It's worth repeating: the customer caused this, he must now feel the pain and not you. So, if he made an informed choice and that is what he chose, then that is how it has to be. However, if I were in the position of supporting this installation I'd probably give some thought as to what the right tools for the job are. Gentoo simply isn't designed to be updated twice a decade. If you REALLY want that kind of deployment you should probably be running something like RHEL, which will commercially support your old install for years and help you with any issues with migration (but there aren't likely to be many, since they will have tested moving from one 5-year-old release to the next 5-year-supported one. Debian stable or CentOS are of course free alternatives, or Ubuntu LTS. I love Gentoo, and I think it is the right tool for a lot of jobs, but it isn't always the right tool for EVERY job. If it really is just one box and you don't mind dealing with the downtime/hassle/etc twice a decade then by all means use Gentoo. However, if I were managing 100 of these this is not how I'd want to be doing it. -- Rich
Re: [gentoo-user] another old box to update
On Wed, 07 Jan 2015 14:19:27 +0100, Stefan G. Weichinger wrote: at the reboot time a fellow IT-guy will be there in front of the console so if the NIC doesn't come up correctly I will be able to instruct him to get the box up and reachable for me. I also use to disable persistent names for such updates ... and get good old eth0 UP instead of enpXsY unconfigured ;-) Just add net.ifnames=0 to the kernel options. -- Neil Bothwick Snacktrek, n.: The peculiar habit, when searching for a snack, of constantly returning to the refrigerator in hopes that something new will have materialized. pgpxzILPetq41.pgp Description: OpenPGP digital signature
Re: [gentoo-user] another old box to update
On 2015-01-07 13:47, Alan McKinnon wrote: On 07/01/2015 13:52, Stefan G. Weichinger wrote: I am in the process of upgrading an old (~2010) gentoo server. The customer never wanted updates ... and now he wants ... *sigh* Don't waste your time (you are already experiencing the full reason why). Backup data and configs, reinstall Gentoo, restore data and configs. Downtime? Of course. A few hours. Customer needs to understand he brought this upon himself. Trying to do it in-place will likely takes *days* and fill you with pain and mucho downtime. This list, the forum, and planet are full of horror stories of what it takes to do it and the issues you will run into. Frankly, you do not need to prove you can do it (we know you can), and you have much better things to do with your time (like proper billable hours). It's worth repeating: the customer caused this, he must now feel the pain and not you. Strange, I only have successful stories with upgrading old gentoo machines. If you have a machine which you update regularly then you know all the issues during the time and so upgrading per partes leads to no surprises but the same challenges you've handled before. But yes, it takes time. Moreover, if you use configuration management like Ansible, you can even automatically merge changes when applications ship new configuration.
[gentoo-user] another old box to update
I am in the process of upgrading an old (~2010) gentoo server. The customer never wanted updates ... and now he wants ... *sigh* I managed to compile basic stuff already ... portage, gcc etc Now I get errors at emerging packages which is bad. Still no openrc installed and the udev-upgrade also is still pending. gcc-4.8.3 When I try to emerge grep for example, the config.log says: configure:4451: $? = 0 configure:4440: i686-pc-linux-gnu-gcc -v 5 Using built-in specs. COLLECT_GCC=/usr/i686-pc-linux-gnu/gcc-bin/4.8.3/i686-pc-linux-gnu-gcc COLLECT_LTO_WRAPPER=/usr/libexec/gcc/i686-pc-linux-gnu/4.8.3/lto-wrapper Target: i686-pc-linux-gnu Configured with: /var/tmp/portage/sys-devel/gcc-4.8.3/work/gcc-4.8.3/configure --host=i686-pc-linux-gnu --build=i686-pc-linux-gnu --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc-bin/4.8.3 --includedir=/usr/lib/gcc/i686-pc-linux-gnu/4.8.3/include --datadir=/usr/share/gcc-data/i686-pc-linux-gnu/4.8.3 --mandir=/usr/share/gcc-data/i686-pc-linux-gnu/4.8.3/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/4.8.3/info --with-gxx-include-dir=/usr/lib/gcc/i686-pc-linux-gnu/4.8.3/include/g++-v4 --with-python-dir=/share/gcc-data/i686-pc-linux-gnu/4.8.3/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.8.3 p1.1, pie-0.5.9' --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --disable-multilib --disable-altivec --disable-fixed-point --with-arch=i686 --enable-targets=all --disable-libgcj --enable-libgomp --disable-libmudflap --disable-libssp --enable-lto --without-cloog --enable-libsanitizer Thread model: posix gcc version 4.8.3 (Gentoo 4.8.3 p1.1, pie-0.5.9) configure:4451: $? = 0 configure:4440: i686-pc-linux-gnu-gcc -V 5 i686-pc-linux-gnu-gcc: error: unrecognized command line option '-V' i686-pc-linux-gnu-gcc: fatal error: no input files compilation terminated. configure:4451: $? = 1 configure:4440: i686-pc-linux-gnu-gcc -qversion 5 i686-pc-linux-gnu-gcc: error: unrecognized command line option '-qversion' i686-pc-linux-gnu-gcc: fatal error: no input files compilation terminated. configure:4451: $? = 1 configure:4440: i686-pc-linux-gnu-gcc -version 5 i686-pc-linux-gnu-gcc: error: unrecognized command line option '-version' i686-pc-linux-gnu-gcc: fatal error: no input files compilation terminated. configure:4451: $? = 1 configure:4471: checking whether the C compiler works configure:4493: i686-pc-linux-gnu-gcc -O2 -march=prescott -pipe -fomit-frame-pointer conftest.c 5 i686-pc-linux-gnu-gcc: error trying to exec 'as': execvp: No such file or directory configure:4497: $? = 2 configure:4535: result: no I understand that configure is called with wrong parameters ... where do they come from? -- For reference: $ emerge --info Portage 2.2.14 (python 2.7.9-final-0, default/linux/x86/13.0, gcc-4.8.3, glibc-2.11.2-r3, 2.6.36-gentoo-r5 i686) = System uname: Linux-2.6.36-gentoo-r5-i686-Intel-R-_Xeon-R-_CPU_X3430_@_2.40GHz-with-gentoo-2.2 KiB Mem: 2065232 total,944892 free KiB Swap:1044216 total, 1044104 free Timestamp of tree: Fri, 02 Jan 2015 17:30:01 + ccache version 2.4 [disabled] app-shells/bash: 4.2_p53 dev-lang/perl:5.12.3-r1 dev-lang/python: 2.6.4-r1, 2.7.9-r1, 3.3.5-r1 dev-util/ccache: 2.4-r9 dev-util/cmake: 2.6.4-r3 dev-util/pkgconfig: 0.28-r1 sys-apps/baselayout: 2.2 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.9.6-r2::unknown repository, 1.11.1, 1.13.4 sys-devel/binutils: 2.24-r3 sys-devel/gcc:4.3.4, 4.4.4-r2, 4.8.3 sys-devel/gcc-config: 1.7.3 sys-devel/libtool:2.4.2-r1 sys-devel/make: 4.0-r1 sys-kernel/linux-headers: 3.16 (virtual/os-headers) sys-libs/glibc: 2.11.2-r3 Repositories: gentoo local_overlay ACCEPT_KEYWORDS=x86 ACCEPT_LICENSE=* -@EULA CBUILD=i686-pc-linux-gnu CFLAGS=-O2 -march=prescott -pipe -fomit-frame-pointer CHOST=i686-pc-linux-gnu CONFIG_PROTECT=/etc CONFIG_PROTECT_MASK=/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo CXXFLAGS=-O2 -march=prescott -pipe -fomit-frame-pointer DISTDIR=/usr/portage/distfiles FCFLAGS=-O2 -march=i686 -pipe FEATURES=assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync FFLAGS=-O2 -march=i686 -pipe GENTOO_MIRRORS= http://gentoo.osuosl.org/
Re: [gentoo-user] another old box to update
On Wed, 07 Jan 2015 13:01:34 +0100, Tomas Mozes wrote:
Try to fetch some older portage snapshots
http://dev.gentoo.org/~swift/snapshots/ and update in steps, not as a 4
year giant leap. Try to fetch snapshot 2011, then upgrade, then 2012,
upgrade... It takes more time, but it should work.
It may be quicker to backup the data and configs and start a fresh
install. Five years is a long time for any server to go without updates,
${DEITY} knows what has happened to it in that time.
--
Neil Bothwick
Psychiatrists say that 1 of 4 people are mentally ill.
Check three friends. If they're OK, you're it.
pgpjaXWZVxVTU.pgp
Description: OpenPGP digital signature
Re: [gentoo-user] another old box to update
On 07/01/2015 14:56, Stefan G. Weichinger wrote: Am 07.01.2015 um 13:47 schrieb Alan McKinnon: On 07/01/2015 13:52, Stefan G. Weichinger wrote: I am in the process of upgrading an old (~2010) gentoo server. The customer never wanted updates ... and now he wants ... *sigh* Don't waste your time (you are already experiencing the full reason why). Backup data and configs, reinstall Gentoo, restore data and configs. Downtime? Of course. A few hours. Customer needs to understand he brought this upon himself. Trying to do it in-place will likely takes *days* and fill you with pain and mucho downtime. This list, the forum, and planet are full of horror stories of what it takes to do it and the issues you will run into. Frankly, you do not need to prove you can do it (we know you can), and you have much better things to do with your time (like proper billable hours). It's worth repeating: the customer caused this, he must now feel the pain and not you. I should print this and learn this at last, right! :-) Thanks for the clear opinion at least I will be able to bill the hours and I am quite sure that I am not that far from success it's a rather simple samba-server ... right now I spent about 3 hrs of time over the last week or so. What's left? migration to openrc, new udev ... kernel is prepared ... then the test of rebooting on their closed day (they don't work on tuesdays). The tricky one is going to be that persistent interface names from udev 18 months or so back. When you get to that, you'll probably want to re-read the huge threads from that time, as you only get one chance to get it right. I see in your reply to Neil you have glibc conflicts. I don't know what will happen if you do it with --nodeps, but I wouldn't try that. The box is remote, if something goes wrong... Rather go with Tomas' suggestion of yearly portage snapshots and update in stages. openrc should be seamless. I forget the exact timelines, but IIRC you will also hit baselayout-2 migration. That one was very smooth and well documented so you shouldn't have much trouble. Any python issues? I don't recall any show-stoppers with it, but you never know. -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] another old box to update
On 07/01/2015 13:52, Stefan G. Weichinger wrote: I am in the process of upgrading an old (~2010) gentoo server. The customer never wanted updates ... and now he wants ... *sigh* Don't waste your time (you are already experiencing the full reason why). Backup data and configs, reinstall Gentoo, restore data and configs. Downtime? Of course. A few hours. Customer needs to understand he brought this upon himself. Trying to do it in-place will likely takes *days* and fill you with pain and mucho downtime. This list, the forum, and planet are full of horror stories of what it takes to do it and the issues you will run into. Frankly, you do not need to prove you can do it (we know you can), and you have much better things to do with your time (like proper billable hours). It's worth repeating: the customer caused this, he must now feel the pain and not you. -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] another old box to update
On Jan 7, 2015, at 14:47, Alan McKinnon alan.mckin...@gmail.com wrote: On 07/01/2015 13:52, Stefan G. Weichinger wrote: I am in the process of upgrading an old (~2010) gentoo server. The customer never wanted updates ... and now he wants ... *sigh* Don't waste your time (you are already experiencing the full reason why). Backup data and configs, reinstall Gentoo, restore data and configs. I had a similar challenge. But it is quite easy to overcome. After the backups just untar the latest stage3 to your root filesystem. Then sync portage and emerge world with a empty tree and keep-going flags. It should get it done mostly. Few packages might fail to merge, but after the world update the list should be fairly short and manageable. You might need to emerge -C few packages, but it's ok. After the system is up-to date restore your backups. -- -Matti
Re: [gentoo-user] another old box to update
Am 07.01.2015 um 14:44 schrieb Alan McKinnon: On 07/01/2015 15:19, Stefan G. Weichinger wrote: Seems as if the biggest problems are solved right now? if you ran emerge -avuND world and portage goes ahead and does it without blockers, then I'd agree - the major problems are solved. It's a simple samba server, so not too many packages. I'd recommend you do emerge -e world when done just to ensure that everything is consistently built against the same toolchain and glibc. It's not absolutely required, but it does give peace of mind and you can let it run and do it's thing while you get on with something else I try emerge -avuDN @system for now ... this includes glibc etc and should result in a quite bootable system. The openssl-upgrade lead to some @preserved-rebuild issues ... but I have to clean up @world first. This box was based on another installation back then and has the whole LAMP stack installed which isn't used at all. If I get rid of this the whole setup and upgrade gets much much simplified. Maybe I can talk my colleague into an earlier reboot-test ... if that works we are running on a current kernel/udev/openrc ... that would be great to know. glibc-2.19-r1 emerged as well now. - But you are still right: If I had backed up /etc and copied some of my images in there I would also be up and running already. Next time! ;-) S
