Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
On 05/01/2023 12:09, m1027 wrote: frederik.pfautsch: > >>> So, ideally, there is c): In a hypothetic case we would prepare a >>> entire OS incl. our app (maybe via catalyst?) which would require >>> a bootloader-like mini-OS on the customer's side, to receive >>> updates over the internet, switch the OS at boot time, and >>> fallback. I was recently playing with systemd-boot and it's >>> interesting try-boot feature. >> >> So essentially it sounds like you want something similar to Yocto / >> Poky / Petalinux for the non-embedded world (and based on Gentoo of >> course; it sounds like catalyst is something like that)? > > I've had a look on that: Wow, another interesting approach to build > customized OSes. Thanks! > >> Just throwing crazy ideas around, what about using net-boot for >> your customer? This way they just need to store an image somewhere >> and can update it whenever necessary. Or (ab-)using an initramfs. >> Or u-boot bootloader, just like in the embedded world. Depending on >> the size of the actual OS/rootfs, taking ideas from e.g. Android >> with their A/B bootslots (i.e. two root-partitions or something, >> where one is active and the other can be updated with clever >> scripts, after a reboot they are swapped). > > ... exactly what is on my wishlist currently! I am missing such an > alternative when in need for updating a remote (customer's) OS, where > ssh + emerge @world is just no option. If we had that, I'd see Gentoo > (e.g. with catalyst or via Yocto) shining bright here as it is > perfect in stipping down things to the required minimum. > > Thanks. Interesting projects in that space could be mender.io, or swupdate. Both are based on the idea of having at least two system partitions (+ maybe a couple of data partitions), and downloading the update on an inactive partion. On next boot, the bootloader tries to boot from the new partition; if that fails, it will fall back to the previous (known working) partition after a few tries. -- Xelnor
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
frederik.pfautsch: > > So, ideally, there is c): In a hypothetic case we would prepare > > a entire OS incl. our app (maybe via catalyst?) which would > > require a bootloader-like mini-OS on the customer's side, to > > receive updates over the internet, switch the OS at boot time, > > and fallback. I was recently playing with systemd-boot and it's > > interesting try-boot feature. > > So essentially it sounds like you want something similar to Yocto / Poky > / Petalinux for the non-embedded world (and based on Gentoo of course; > it sounds like catalyst is something like that)? I've had a look on that: Wow, another interesting approach to build customized OSes. Thanks! > Just throwing crazy ideas around, what about using net-boot for your > customer? This way they just need to store an image somewhere and can > update it whenever necessary. Or (ab-)using an initramfs. Or u-boot > bootloader, just like in the embedded world. Depending on the size of > the actual OS/rootfs, taking ideas from e.g. Android with their A/B > bootslots (i.e. two root-partitions or something, where one is active > and the other can be updated with clever scripts, after a reboot they > are swapped). ... exactly what is on my wishlist currently! I am missing such an alternative when in need for updating a remote (customer's) OS, where ssh + emerge @world is just no option. If we had that, I'd see Gentoo (e.g. with catalyst or via Yocto) shining bright here as it is perfect in stipping down things to the required minimum. Thanks.
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
peter: > > Whenever we need to deliver a updated app to customers whose OS is > > too old (but updating it is too risky), we could either > > a) keep evenly outdated dev build OSes around forever (oh no!), or > > b) post our newly built app in a container (leaving the lovely native > > world); and both ignore the fact that customers wish maintenance of > > the entire OS actually, too. > > So, ideally, there is c): In a hypothetic case we would prepare a > > entire OS incl. our app (maybe via catalyst?) which would require > > a bootloader-like mini-OS on the customer's side, to receive updates > > over the internet, switch the OS at boot time, and fallback. > > I had a) in mind. Why "oh no!" ? I didn't mean forever. Why "Oh no": Well, it works and we are currently going that way. But see the other discussion branch in this thread: It's just a growing maintainment mess over time. In short: Too many OS versions resp. according dev VMs to rebuild new apps against it. > catalyst building specific, "old-like" OSes on which you build new (binpkg) > versions of your app to be installable by emerge on old customer OSes is > admittedly a lot of work, at least initially. Yes, that brings in another approach: *If* we had catalyst controlled versions of the (already...) delivered OSes, we could maybe setup a CI/CD like pipeline to re-create the according developer VM (for building a updated app with the exact old dependencies as on the formerly delivered old customer OS). So to say: VM just in time. Well I'll keep that in mind. That may help against the VM mess (see above). But building exact (older) OSes still requires a solution to distribute the entire OS incl. app remotely and savely. See also the other discussion branch on that if you like. Thanks
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
m1027 wrote: > Wow, wasn't aware of catalyst at all. What a beast in terms of control. It's not so well-known maybe because it was created by and for gentoo-releng but if you know what you want it's a fantastic tool. > (FYI: I enjoyed the links on catalyst you sent me directly. > Unfortunatelly I cannot answer you directly due to the default > TLS guarantee Thanks! Glad you liked them. And yes, TLS is on my list. > While being able to build exact environments with catalyst I wonder > how it could help fixing the issue of my original post. Thank you for connecting back to the original post! Let's see: > Whenever we need to deliver a updated app to customers whose OS is > too old (but updating it is too risky), we could either > a) keep evenly outdated dev build OSes around forever (oh no!), or > b) post our newly built app in a container (leaving the lovely native > world); and both ignore the fact that customers wish maintenance of > the entire OS actually, too. > So, ideally, there is c): In a hypothetic case we would prepare a > entire OS incl. our app (maybe via catalyst?) which would require > a bootloader-like mini-OS on the customer's side, to receive updates > over the internet, switch the OS at boot time, and fallback. I had a) in mind. Why "oh no!" ? I didn't mean forever. I think it's already a very good value proposition that you can deliver updates of your apps for the particular systems that your customers run. catalyst building specific, "old-like" OSes on which you build new (binpkg) versions of your app to be installable by emerge on old customer OSes is admittedly a lot of work, at least initially. Separate from those app updates you can then use catalyst to also tackle OS maintenance/updates. You can create either full milestone OSes or just individual (binpkg) packages through which customers' OSes can become less fragmented over time, at the pace each customer is comfortable with. OS updates can take only small steps at a time, since catalyst allows exact control. This could reduce target OS diversity drastically over time with probably relatively moderate development effort. More effort might go into holding customer hands along the bespoke update paths. So, I see controlled paths forward in both problem dimensions. I don't know if the effort is actually practical for you but it /is/ doable and most importantly it can be customized for the individual systems currently in production at your customers. //Peter
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
Hi, On Mon 02 Jan 2023 23:31:17 GMT, m1027 wrote: > I am not complaining here. Hey, we are on rolling release. Some of > you may even know individual solutions to work around each of it. > However, we just may get into trouble when distributing newly > compiled apps (on new Gentoo systems) to older Gentoo systems. And > we don't know in advance. I am looking for the best way to avoid > that. It’s not really a production environment, but I have this hobby infra I’ve set up when I was a student an my solution is to have a VM compiling the big packages (gcc, glibc, llvm, php, etc.) and publish a binary repo from that. Then, every VM pulls this, and compile what’s specific on top of that. Everything is handled by crons. So far it works. One thing to get in mind is that, if a VM got stuck to an old commit (e.g. because one package needed a use change and emerge fails to update world due to that), I’m always re-compiling everything, because I’m scared of glibc or ncurses ABI breaks. My scale is ~30 VMs on 4 physical servers. -- Alarig
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
Am 04.01.23 um 23:00 schrieb m1027: Wow, wasn't aware of catalyst at all. What a beast in terms of control. (FYI: I enjoyed the links on catalyst you sent me directly. Unfortunatelly I cannot answer you directly due to the default TLS guarantee kicked in by my provider: "TLS is required, but was not offered by host". That's usually to be fixed on the receiving side.) While being able to build exact environments with catalyst I wonder how it could help fixing the issue of my original post. To sum up: Whenever we need to deliver a updated app to customers whose OS is too old (but updating it is too risky), we could either a) keep evenly outdated dev build OSes around forever (oh no!), or b) post our newly built app in a container (leaving the lovely native world); and both ignore the fact that customers wish maintenance of the entire OS actually, too. So, ideally, there is c): In a hypothetic case we would prepare a entire OS incl. our app (maybe via catalyst?) which would require a bootloader-like mini-OS on the customer's side, to receive updates over the internet, switch the OS at boot time, and fallback. I was recently playing with systemd-boot and it's interesting try-boot feature. So essentially it sounds like you want something similar to Yocto / Poky / Petalinux for the non-embedded world (and based on Gentoo of course; it sounds like catalyst is something like that)? Petalinux / Yocto is essentially a cross compiling environment with rootfs, etc. And after everything has been built, the rootfs is just packaged into a flashable ext4-image or something. So your devs wouldn't need to work on a (slightly) outdated system, just use the (in this case non-)cross-compiling environment with its own, separate rootfs. Is, e.g., crossdev able to build a non-cross, native compiler? It already provides a rootfs-folder and a separate emerge-config, etc... Just throwing crazy ideas around, what about using net-boot for your customer? This way they just need to store an image somewhere and can update it whenever necessary. Or (ab-)using an initramfs. Or u-boot bootloader, just like in the embedded world. Depending on the size of the actual OS/rootfs, taking ideas from e.g. Android with their A/B bootslots (i.e. two root-partitions or something, where one is active and the other can be updated with clever scripts, after a reboot they are swapped). OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
peter: > Peter Stuge wrote: > > Essentially you will be maintaining a private fork of gentoo.git, > > If this seems too heavy handed then you can just as well do the reverse: > > Maintain an overlay repo with the packages you care to control in the > state you care to have them, set that in the catalyst stage4.spec > portage_overlay and add unwanted package versions in gentoo.git to > the package.mask directory used by catalyst. > > This may sound complicated but it isn't bad at all. > > For total control also make your own profile, e.g. based on embedded, > but that's not per se neccessary, only if the standard profiles has too > much conflicts with what you want in @system. > > catalyst will rebuild @system according to spec file but with too much > difference that just becomes annoying and feels more trouble than a > controlled profile. > > This approach falls somewhere between your options (1) and (5). Wow, wasn't aware of catalyst at all. What a beast in terms of control. (FYI: I enjoyed the links on catalyst you sent me directly. Unfortunatelly I cannot answer you directly due to the default TLS guarantee kicked in by my provider: "TLS is required, but was not offered by host". That's usually to be fixed on the receiving side.) While being able to build exact environments with catalyst I wonder how it could help fixing the issue of my original post. To sum up: Whenever we need to deliver a updated app to customers whose OS is too old (but updating it is too risky), we could either a) keep evenly outdated dev build OSes around forever (oh no!), or b) post our newly built app in a container (leaving the lovely native world); and both ignore the fact that customers wish maintenance of the entire OS actually, too. So, ideally, there is c): In a hypothetic case we would prepare a entire OS incl. our app (maybe via catalyst?) which would require a bootloader-like mini-OS on the customer's side, to receive updates over the internet, switch the OS at boot time, and fallback. I was recently playing with systemd-boot and it's interesting try-boot feature. Thanks
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
On Mon, Jan 2, 2023 at 4:55 PM m1027 wrote: > > Many thanks for your detailed thoughs for sharing the rich > experiences on this! See below: > > antarus: > > > On Mon, Jan 2, 2023 at 4:48 AM m1027 wrote: > > > > > > Hi and happy new year. > > > > > > When we create apps on Gentoo they become easily incompatible for > > > older Gentoo systems in production where unattended remote world > > > updates are risky. This is due to new glibc, openssl-3 etc. > > > > I wrote a very long reply, but I've removed most of it: I basically > > have a few questions, and then some comments: > > > > I don't quite grasp your problem statement, so I will repeat what I > > think it is and you can confirm / deny. > > > > - Your devs build using gentoo synced against some recent tree, they > > have recent packages, and they build some software that you deploy to > > prod. > > Yes. > > > - Your prod machines are running gentoo synced against some recent > > tree, but not upgraded (maybe only glsa-check runs) and so they are > > running 'old' packages because you are afraid to update them[0] > > Well, we did sync (without updading packages) in the past but today we > even fear to sync against recent trees. Without going into details, > as a rule of thumb, weekly or monthly sync + package updates work > near to perfect. (It's cool to see what a good job emerge does on our > own internal production systems.) Updating systems older than 12 > months or so may, however, be a hugh task. And too risky for remote > production systems of customers. My primary risk I think is that even if you ship your app in a container you still need somewhere to run the containers. Currently that is a fleet of different hardware and gentoo configurations, and while containers certainly simplify your life there, they won't fix all your problems. Now instead of worrying that upgrading your Gentoo OS will break your app, it will instead break your container runtime. It is likely a smaller surface area, but it is not zero. Not saying don't use containers, just that there is no free lunch here necessarily. > > > > - Your software builds OK in dev, but when you deploy it in prod it > > breaks, because prod is really old, and your developments are using > > packages that are too new. > > Exactly. > > > > My main feedback here is: > > - Your "build" environment should be like prod. You said you didn't > > want to build "developer VMs" but I am unsure why. For example I run > > Ubuntu and I do all my gentoo development (admittedly very little > > these days) > >in a systemd-nspawn container, and I have a few shell scripts to > > mount everything and set it up (so it has a tree snapshot, some git > > repos, some writable space etc.) > > Okay, yes. That is way (1) I mentioned in my OP. It works indeed but > has the mentioned drawbacks: VMs and maintenance pile up, and for > each developer. And you don't know when there is the moment to > create a new VM. But yes it seems to me one of the ways to go: > *Before* creating a production system you need to freeze portage, > create dev VMs, and prevent updates on the VMs, too. (Freezing aka > not updating has many disadvantages, of course.) Oh sorry, I failed to understand you were doing that already. I agree it's challenging, I think if you don't have a great method to simplify here, it might not be a great avenue going forward. - Trying to figure out when you can make a new VM. - Trying to figure out when you can take a build and deploy it to a customer safely. I've seen folks try to group customers in some way to reduce the number of prod artifacts required, but if you cannot it might be The benefit of containers here is that you can basically deploy your app at whatever rate you want, and only the OS upgrades remain risky (because they might break the container runtime.) Depending on your business needs, it might be advantageous to go that route. > > > > - Your "prod" environment is too risky to upgrade, and you have > > difficulty crafting builds that run in every prod environment. I think > > this is fixable by making a build environment more like the prod > > environment. > > The challenge here is that if you have not done that (kept the > > copies of ebuilds around, the distfiles, etc) it can be challenging to > > "recreate" the existing older prod environments. > > But if you do the above thing (where devs build in a container) > > and you can make that container like the prod environments, then you > > can enable devs to build for the prod environment (in a container on > > their local machine) and get the outcome you want. > > Not sure I got your point here. But yes, it comes down to what was > said above. > > > > - Understand that not upgrading prod is like, to use a finance term, > > picking up pennies in front of a steamroller. It's a great strategy, > > but eventually you will actually *need* to upgrade something. Maybe > > for a critical security issue, maybe for a feature. Ha
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
Many thanks for your detailed thoughs for sharing the rich experiences on this! See below: antarus: > On Mon, Jan 2, 2023 at 4:48 AM m1027 wrote: > > > > Hi and happy new year. > > > > When we create apps on Gentoo they become easily incompatible for > > older Gentoo systems in production where unattended remote world > > updates are risky. This is due to new glibc, openssl-3 etc. > > I wrote a very long reply, but I've removed most of it: I basically > have a few questions, and then some comments: > > I don't quite grasp your problem statement, so I will repeat what I > think it is and you can confirm / deny. > > - Your devs build using gentoo synced against some recent tree, they > have recent packages, and they build some software that you deploy to > prod. Yes. > - Your prod machines are running gentoo synced against some recent > tree, but not upgraded (maybe only glsa-check runs) and so they are > running 'old' packages because you are afraid to update them[0] Well, we did sync (without updading packages) in the past but today we even fear to sync against recent trees. Without going into details, as a rule of thumb, weekly or monthly sync + package updates work near to perfect. (It's cool to see what a good job emerge does on our own internal production systems.) Updating systems older than 12 months or so may, however, be a hugh task. And too risky for remote production systems of customers. > - Your software builds OK in dev, but when you deploy it in prod it > breaks, because prod is really old, and your developments are using > packages that are too new. Exactly. > My main feedback here is: > - Your "build" environment should be like prod. You said you didn't > want to build "developer VMs" but I am unsure why. For example I run > Ubuntu and I do all my gentoo development (admittedly very little > these days) >in a systemd-nspawn container, and I have a few shell scripts to > mount everything and set it up (so it has a tree snapshot, some git > repos, some writable space etc.) Okay, yes. That is way (1) I mentioned in my OP. It works indeed but has the mentioned drawbacks: VMs and maintenance pile up, and for each developer. And you don't know when there is the moment to create a new VM. But yes it seems to me one of the ways to go: *Before* creating a production system you need to freeze portage, create dev VMs, and prevent updates on the VMs, too. (Freezing aka not updating has many disadvantages, of course.) > - Your "prod" environment is too risky to upgrade, and you have > difficulty crafting builds that run in every prod environment. I think > this is fixable by making a build environment more like the prod > environment. > The challenge here is that if you have not done that (kept the > copies of ebuilds around, the distfiles, etc) it can be challenging to > "recreate" the existing older prod environments. > But if you do the above thing (where devs build in a container) > and you can make that container like the prod environments, then you > can enable devs to build for the prod environment (in a container on > their local machine) and get the outcome you want. Not sure I got your point here. But yes, it comes down to what was said above. > - Understand that not upgrading prod is like, to use a finance term, > picking up pennies in front of a steamroller. It's a great strategy, > but eventually you will actually *need* to upgrade something. Maybe > for a critical security issue, maybe for a feature. Having a build > environment that matches prod is good practice, you should do it, but > you should also really schedule maintenance for these prod nodes to > get them upgraded. (For physical machines, I've often seen businesses > just eat the risk and assume the machine will physically fail before > the steamroller comes, but this is less true with virtualized > environments that have longer real lifetimes.) Yes, haha, I agree. And yes, I totally ignored backporting security here, as well as the need that we might *require* a dependend package upgrade (e.g. to fix a known memory leak). I left that out for simlicity only. > > So, what we've thought of so far is: > > > > (1) Keeping outdated developer boxes around and compile there. We > > would freeze portage against accidental emerge sync by creating a > > git branch in /var/db/repos/gentoo. This feels hacky and requires a > > increating number of develper VMs. And sometimes we are hit by a > > silent incompatibility we were not aware of. > > In general when you build binaries for some target, you should build > on that target when possible. To me, this is the crux of your issue > (that you do not) and one of the main causes of your pain. > You will need to figure out a way to either: > - Upgrade the older environments to new packages. > - Build in copies of the older environments. > > I actually expect the second one to take 1-2 sprints (so like 1 engineer > month?) > - One sprint to make some
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
sam: > > On 2 Jan 2023, at 12:48, m1027 wrote: > > > > Hi and happy new year. > > > > When we create apps on Gentoo they become easily incompatible for > > older Gentoo systems in production where unattended remote world > > updates are risky. This is due to new glibc, openssl-3 etc. [...] > I'd really suggest just using stable in production and a mix > for developers so you can catch any problems beforehand. > > We try to be quite conservative about things like OpenSSL 3, > glibc updates, etc. Thanks, a misunderstanding then: I am talking about stable only. Whilst Gentoo may be conservative and all of you do an excellent job on keeping things smooth, incompatibilities of software being newly created on up-to-date developer systems (and then tried to be distributed to outdated production systems) may arise multiple times per year; *any* of the following alone may trigger a incompatibility. Just some random examples: (1) Most prominent, glibc updates. It has its sophisticated function versioning. You may or may not be hit. It is not depending on glibc's version but the internal function versions which are updated in a quite subtile way. (Function versioning is without doubt an impressive feature.) (2) libjpeg: In the past, libjpeg reached version 9 (like on Ubuntu today) but later was versioned 62 or 6.2 AFAIK. If you have an old Gentoo production system still on libjpeg-9, you have a hard time to update and distribute a new version of your app, as this version of libjpeg is not present in Gentoo anymore. (Don't get me wrong, there have probably been good reasons for this downgrade.) BTW: This little incompatibility is one of the reasons why it is hard to compile a app on Ubuntu for Gentoo. (3) An older one: libressl was removed. Well, we all remember the debate whether to keep it or not. (4) There is openssl-3 showing up on the horizon. I expect incompatibilities when distributing newly built software. (5) Portage EAPIs: If there is a new EAPI and you emerge --sync, then you need to update portage. This however, might require surprisingly many other updates. New python, setuptools and friends. I am not complaining here. Hey, we are on rolling release. Some of you may even know individual solutions to work around each of it. However, we just may get into trouble when distributing newly compiled apps (on new Gentoo systems) to older Gentoo systems. And we don't know in advance. I am looking for the best way to avoid that. Thanks
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
On Mon, Jan 2, 2023 at 4:48 AM m1027 wrote: > > Hi and happy new year. > > When we create apps on Gentoo they become easily incompatible for > older Gentoo systems in production where unattended remote world > updates are risky. This is due to new glibc, openssl-3 etc. I wrote a very long reply, but I've removed most of it: I basically have a few questions, and then some comments: I don't quite grasp your problem statement, so I will repeat what I think it is and you can confirm / deny. - Your devs build using gentoo synced against some recent tree, they have recent packages, and they build some software that you deploy to prod. - Your prod machines are running gentoo synced against some recent tree, but not upgraded (maybe only glsa-check runs) and so they are running 'old' packages because you are afraid to update them[0] - Your software builds OK in dev, but when you deploy it in prod it breaks, because prod is really old, and your developments are using packages that are too new. My main feedback here is: - Your "build" environment should be like prod. You said you didn't want to build "developer VMs" but I am unsure why. For example I run Ubuntu and I do all my gentoo development (admittedly very little these days) in a systemd-nspawn container, and I have a few shell scripts to mount everything and set it up (so it has a tree snapshot, some git repos, some writable space etc.) - Your "prod" environment is too risky to upgrade, and you have difficulty crafting builds that run in every prod environment. I think this is fixable by making a build environment more like the prod environment. The challenge here is that if you have not done that (kept the copies of ebuilds around, the distfiles, etc) it can be challenging to "recreate" the existing older prod environments. But if you do the above thing (where devs build in a container) and you can make that container like the prod environments, then you can enable devs to build for the prod environment (in a container on their local machine) and get the outcome you want. - Understand that not upgrading prod is like, to use a finance term, picking up pennies in front of a steamroller. It's a great strategy, but eventually you will actually *need* to upgrade something. Maybe for a critical security issue, maybe for a feature. Having a build environment that matches prod is good practice, you should do it, but you should also really schedule maintenance for these prod nodes to get them upgraded. (For physical machines, I've often seen businesses just eat the risk and assume the machine will physically fail before the steamroller comes, but this is less true with virtualized environments that have longer real lifetimes.) > > So, what we've thought of so far is: > > (1) Keeping outdated developer boxes around and compile there. We > would freeze portage against accidental emerge sync by creating a > git branch in /var/db/repos/gentoo. This feels hacky and requires a > increating number of develper VMs. And sometimes we are hit by a > silent incompatibility we were not aware of. In general when you build binaries for some target, you should build on that target when possible. To me, this is the crux of your issue (that you do not) and one of the main causes of your pain. You will need to figure out a way to either: - Upgrade the older environments to new packages. - Build in copies of the older environments. I actually expect the second one to take 1-2 sprints (so like 1 engineer month?) - One sprint to make some scripts that makes a new production 'container' - One sprint to sort of integrate that container into your dev workflow, so devs build in the container instead of what they build in now. It might be more or less daunting depending on how many distinct (unique?) prod environments you have (how many containers will you actually need for good build coverage?), how experienced in Gentoo your developers are, and how many artifacts from prod you have. - A few crazy ideas are like: - Snapshot an existing prod machine, strip of it machine-specific bits, and use that as your container. - Use quickpkg to generate a bunch of bin pkgs from a prod machine, use that to bootstrap a container. - Probably some other exciting ideas on the list ;) > > (2) Using Ubuntu LTS for production and Gentoo for development is > hit by subtile libjpeg incompatibilites and such. I would advise, if possible, to make dev and prod as similar as possible[1]. I'd be curious what blockers you think there are to this pattern. Remember that "dev" is not "whatever your devs are using" but is ideally some maintained environment; segmented from their daily driver computer (somehow). > > (3) Distributing apps as VMs or docker: Even those tools advance and > become incompatible, right? And not suitable when for smaller Arm > devices. I think if your apps are small and self-contained and easily rebuilt, your (3) and (4) can be workable. If you need 1000
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
Peter Stuge wrote: > Essentially you will be maintaining a private fork of gentoo.git, If this seems too heavy handed then you can just as well do the reverse: Maintain an overlay repo with the packages you care to control in the state you care to have them, set that in the catalyst stage4.spec portage_overlay and add unwanted package versions in gentoo.git to the package.mask directory used by catalyst. This may sound complicated but it isn't bad at all. For total control also make your own profile, e.g. based on embedded, but that's not per se neccessary, only if the standard profiles has too much conflicts with what you want in @system. catalyst will rebuild @system according to spec file but with too much difference that just becomes annoying and feels more trouble than a controlled profile. This approach falls somewhere between your options (1) and (5). Good luck! //Peter
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
Hi, m1027 wrote: > So, what we've thought of so far is: > > (1) Keeping outdated developer boxes around and compile there. We > would freeze portage against accidental emerge sync by creating a > git branch in /var/db/repos/gentoo. This feels hacky and requires a > increating number of develper VMs. And sometimes we are hit by a > silent incompatibility we were not aware of. .. > (5) Inventing a full fledged OTA Gentoo OS updater and distribute > that together with the apps... Nah. > > Hm... Comments welcome. I recommend taking ownership (and responsibility) of your OS. Gentoo tooling (catalyst) is really fantastic for doing so. Essentially you will be maintaining a private fork of gentoo.git, but one where you only really need to manually process the packages you care to control, only when you care to control them. Use catalyst to build tarballs (and binpkgs) from snapshots of that repo. emerge can install binpkg at least from FTP. //Peter
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
> On 2 Jan 2023, at 12:48, m1027 wrote: > > Hi and happy new year. > > When we create apps on Gentoo they become easily incompatible for > older Gentoo systems in production where unattended remote world > updates are risky. This is due to new glibc, openssl-3 etc. > > So, what we've thought of so far is: > > (1) Keeping outdated developer boxes around and compile there. We > would freeze portage against accidental emerge sync by creating a > git branch in /var/db/repos/gentoo. This feels hacky and requires a > increating number of develper VMs. And sometimes we are hit by a > silent incompatibility we were not aware of. > > (2) Using Ubuntu LTS for production and Gentoo for development is > hit by subtile libjpeg incompatibilites and such. > > (3) Distributing apps as VMs or docker: Even those tools advance and > become incompatible, right? And not suitable when for smaller Arm > devices. > > (4) Flatpak: No experience, does it work well? > > (5) Inventing a full fledged OTA Gentoo OS updater and distribute > that together with the apps... Nah. > > Hm... Comments welcome. I'd really suggest just using stable in production and a mix for developers so you can catch any problems beforehand. We try to be quite conservative about things like OpenSSL 3, glibc updates, etc. signature.asc Description: Message signed with OpenPGP
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
On Mon, 2023-01-02 at 12:48 +, m1027 wrote: > Hi and happy new year. > > When we create apps on Gentoo they become easily incompatible for > older Gentoo systems in production where unattended remote world > updates are risky. This is due to new glibc, openssl-3 etc. > Just update them. YOLO. Seriously though, most of us run Gentoo in some sort of production setting. Updating often is your best bet as it keeps the list of suspects short when things do break. OTOH I'm only about 15km from our servers when I set them on fire, so keep that in mind if yours are on another continent. If updating frequently is truly out of the question, a rolling release probably isn't the best deployment target for you.
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
On Mon, Jan 2, 2023 at 7:48 AM m1027 wrote: > > When we create apps on Gentoo they become easily incompatible for > older Gentoo systems in production where unattended remote world > updates are risky. This is due to new glibc, openssl-3 etc. So, unless you're proposing some improvement this might be better-suited for the -user list. Officially Gentoo doesn't really "support" a stable/LTS/release-based environment. That isn't to say that it couldn't be made to do this, however, a more release-based process is one of the core competencies of most other distros, so I'm not sure how much Gentoo would add here vs just running something else. > (1) Keeping outdated developer boxes around and compile there. We > would freeze portage against accidental emerge sync by creating a > git branch in /var/db/repos/gentoo. This feels hacky and requires a > increating number of develper VMs. And sometimes we are hit by a > silent incompatibility we were not aware of. If you're running a large production environment and want a stable platform to target, I'd think you'd want to minimize the number of those you have. You wouldn't want every server running a different base OS, and then have a branch to target development at it. The idea of basing your own releases on Gentoo is probably something many companies do. All you need is to just host one repository and distfile mirror for it, and point all your production hosts at it. Then you could have staging/development environments, and promote the core OS as appropriate. Then your migration path is the same for all hosts and you can account for major vs minor changes. You do need to mirror distfiles as one of the weaknesses of Gentoo for a release-based strategy is that we do not have a solid solution for archiving things like patches/services/etc that are distributed outside of the repo. If a package is removed from the current repo no QA process ensures that the SRC_URIs for anything they used remain stable. Of course somebody would need to do the work but it shouldn't be THAT difficult to have a Gentoo Reference Release project that basically just snapshots the repository/distfiles at points in time and provides a guarantee of clean updates between milestones. Of course if you want backported security updates on top of that this would be quite a bit more effort. It would mainly be useful for those who aren't updating regularly, but completely ignoring all updates isn't a good practice regardless which is probably part of why this hasn't happened. You could greatly minimize the cost of backporting security updates if you only targeted packages of interest to you, or where the updates are low-risk to you, but that depends quite a bit on your own needs. > (3) Distributing apps as VMs or docker: Even those tools advance and > become incompatible, right? And not suitable when for smaller Arm > devices. I don't see why containers would be unsuitable for ARM. VMs certainly are memory hogs but well-designed containers don't really consume much more memory than the service they are hosting. Of course if you dump a whole OS in a container that will consume a fair bit of RAM. An advantage of containers is that they make the host OS less relevant. You could run a release-based distro with security-only updates on the host, and then application containers are low-risk to update remotely and can be built in whatever environment they are most suited to. There is a reason k8s is popular. -- Rich
[gentoo-dev] Gentoo LTS or: proper backward compatibility?
Hi and happy new year. When we create apps on Gentoo they become easily incompatible for older Gentoo systems in production where unattended remote world updates are risky. This is due to new glibc, openssl-3 etc. So, what we've thought of so far is: (1) Keeping outdated developer boxes around and compile there. We would freeze portage against accidental emerge sync by creating a git branch in /var/db/repos/gentoo. This feels hacky and requires a increating number of develper VMs. And sometimes we are hit by a silent incompatibility we were not aware of. (2) Using Ubuntu LTS for production and Gentoo for development is hit by subtile libjpeg incompatibilites and such. (3) Distributing apps as VMs or docker: Even those tools advance and become incompatible, right? And not suitable when for smaller Arm devices. (4) Flatpak: No experience, does it work well? (5) Inventing a full fledged OTA Gentoo OS updater and distribute that together with the apps... Nah. Hm... Comments welcome. Thanks
