Re: [gentoo-dev] [PATCH] eclass/dune.eclass: introduce edune and dune-compile (v2)
On 03/01/2023 00.19, Maciej Barć wrote: edune is a thin wrapper for dune, which will help to run special, uncommon dune commands; dune-compile is a function to selectively pick which packages will be compiled "for-release" (as dune call it); dune-compile without any arguments replaces the current dune_src_compile Signed-off-by: Maciej Barć --- eclass/dune.eclass | 49 +- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/eclass/dune.eclass b/eclass/dune.eclass index 4bc73eda8..384908a40 100644 --- a/eclass/dune.eclass +++ b/eclass/dune.eclass @@ -1,4 +1,4 @@ -# Copyright 1999-2022 Gentoo Authors +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # @ECLASS: dune.eclass @@ -29,7 +29,7 @@ _DUNE_ECLASS=1 # Set before inheriting the eclass. : ${DUNE_PKG_NAME:=${PN}} -inherit multiprocessing +inherit edo multiprocessing # Do not complain about CFLAGS etc since ml projects do not use them. QA_FLAGS_IGNORED='.*' @@ -44,15 +44,54 @@ BDEPEND=" dev-ml/dune " +# @FUNCTION: edune +# @USAGE: ... +# @DESCRIPTION: +# A thin wrapper for the `dune` command. +# Runs `dune` with given arguments and dies on failure. +# +# Example use: +# @CODE +# edune clean +# @CODE +edune() { + debug-print-function ${FUNCNAME} "${@}" + + edo dune "${@}" +} + +# @FUNCTION: dune-compile +# @USAGE: [package] ... +# @DESCRIPTION: +# Compiles either all of packages sources in current directory or selected +# packages. In case of all packages the package detection is done via dune +# itself. +# +# Example use: +# @CODE +# dune-compile menhir menhirLib menhirSdk +# @CODE +dune-compile() { + local -a myduneopts=( + -j $(makeopts_jobs) + --profile release + ) + if [[ -n "${1}" ]] ; then + myduneopts+=( --for-release-of-packages="$(IFS="," ; echo "${*}")" ) + fi + + edune build @install "${myduneopts[@]}" +} + dune_src_compile() { ebegin "Building" - dune build @install -j $(makeopts_jobs) --profile release + dune-compile eend $? || die } dune_src_test() { ebegin "Testing" - dune runtest -j $(makeopts_jobs) --profile release + edune runtest -j $(makeopts_jobs) --profile release eend $? || die } @@ -80,7 +119,7 @@ dune-install() { local pkg for pkg in "${pkgs[@]}" ; do ebegin "Installing ${pkg}" - dune install ${myduneopts[@]} ${pkg} + edune install ${myduneopts[@]} ${pkg} eend $? || die # Move docs to the appropriate place. It appears there is additional output between the ebegin / eend. You may want to consider dropping ebegin and eend. In general, the pattern ebegin, edo, eend should probably be avoided. - Flow
Re: [gentoo-dev] [PATCH] eclass/dune.eclass: introduce edune and dune-compile (v2)
On Tue, 2023-01-03 at 00:19 +0100, Maciej Barć wrote: > edune is a thin wrapper for dune, which will help to run special, > uncommon dune commands; > dune-compile is a function to selectively pick which packages will be > compiled "for-release" (as dune call it); > dune-compile without any arguments replaces the current dune_src_compile > > Signed-off-by: Maciej Barć > --- > eclass/dune.eclass | 49 +- > 1 file changed, 44 insertions(+), 5 deletions(-) > > diff --git a/eclass/dune.eclass b/eclass/dune.eclass > index 4bc73eda8..384908a40 100644 > --- a/eclass/dune.eclass > +++ b/eclass/dune.eclass > @@ -1,4 +1,4 @@ > -# Copyright 1999-2022 Gentoo Authors > +# Copyright 1999-2023 Gentoo Authors > # Distributed under the terms of the GNU General Public License v2 > > # @ECLASS: dune.eclass > @@ -29,7 +29,7 @@ _DUNE_ECLASS=1 > # Set before inheriting the eclass. > : ${DUNE_PKG_NAME:=${PN}} > > -inherit multiprocessing > +inherit edo multiprocessing > > # Do not complain about CFLAGS etc since ml projects do not use them. > QA_FLAGS_IGNORED='.*' > @@ -44,15 +44,54 @@ BDEPEND=" > dev-ml/dune > " > > +# @FUNCTION: edune > +# @USAGE: ... > +# @DESCRIPTION: > +# A thin wrapper for the `dune` command. > +# Runs `dune` with given arguments and dies on failure. > +# > +# Example use: > +# @CODE > +# edune clean > +# @CODE > +edune() { > + debug-print-function ${FUNCNAME} "${@}" > + > + edo dune "${@}" How do you pronounce it? ;-) > +} > + > +# @FUNCTION: dune-compile > +# @USAGE: [package] ... > +# @DESCRIPTION: > +# Compiles either all of packages sources in current directory or selected > +# packages. In case of all packages the package detection is done via dune > +# itself. > +# > +# Example use: > +# @CODE > +# dune-compile menhir menhirLib menhirSdk > +# @CODE > +dune-compile() { > + local -a myduneopts=( > + -j $(makeopts_jobs) > + --profile release > + ) > + if [[ -n "${1}" ]] ; then > + myduneopts+=( --for-release-of-packages="$(IFS="," ; echo > "${*}")" ) > + fi > + > + edune build @install "${myduneopts[@]}" > +} > + > dune_src_compile() { > ebegin "Building" > - dune build @install -j $(makeopts_jobs) --profile release > + dune-compile > eend $? || die > } > > dune_src_test() { > ebegin "Testing" > - dune runtest -j $(makeopts_jobs) --profile release > + edune runtest -j $(makeopts_jobs) --profile release > eend $? || die > } > > @@ -80,7 +119,7 @@ dune-install() { > local pkg > for pkg in "${pkgs[@]}" ; do > ebegin "Installing ${pkg}" > - dune install ${myduneopts[@]} ${pkg} > + edune install ${myduneopts[@]} ${pkg} > eend $? || die > > # Move docs to the appropriate place. -- Best regards, Michał Górny
[gentoo-dev] [PATCH 2/2] cmake.eclass: add base-system as @MAINTAINER too (align with dev-util/cmake)
Signed-off-by: Sam James --- eclass/cmake.eclass | 1 + 1 file changed, 1 insertion(+) diff --git a/eclass/cmake.eclass b/eclass/cmake.eclass index 6787735d5416d..2c5620adede58 100644 --- a/eclass/cmake.eclass +++ b/eclass/cmake.eclass @@ -4,6 +4,7 @@ # @ECLASS: cmake.eclass # @MAINTAINER: # k...@gentoo.org +# base-sys...@gentoo.org # @AUTHOR: # Tomáš Chvátal # Maciej Mrozowski -- 2.39.0
[gentoo-dev] [PATCH 1/2] cmake.eclass: mark CMAKE_VERBOSE as @USER_VARIABLE
It's a policy requirement that ebuilds produce verbose logs, so ebuilds themselves must not set CMAKE_VERBOSE. But users can. Signed-off-by: Sam James --- eclass/cmake.eclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/eclass/cmake.eclass b/eclass/cmake.eclass index 094b6d706bc2e..6787735d5416d 100644 --- a/eclass/cmake.eclass +++ b/eclass/cmake.eclass @@ -1,4 +1,4 @@ -# Copyright 1999-2022 Gentoo Authors +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # @ECLASS: cmake.eclass @@ -96,6 +96,7 @@ fi # By default it uses current working directory (in EAPI-7: ${S}). # @ECLASS_VARIABLE: CMAKE_VERBOSE +# @USER_VARIABLE # @DESCRIPTION: # Set to OFF to disable verbose messages during compilation : ${CMAKE_VERBOSE:=ON} -- 2.39.0
[gentoo-dev] [PATCH 3/3] unpacker.eclass: pass -S to file to disable seccomp
Files being installed by Portage are generally trusted but also the syscalls allowed by file are quite broad anyway. With e.g. new libc or sandbox version (or any number of things...), the syscalls used by file can change which leads to its seccomp filter killing the process. This is an acceptable tradeoff when users are calling file(1), but it makes less sense with trusted input within Portage, especially where it may lead to confusing errors (swallowed within pipes, subshells, etc). Indeed, it might even be the case that file(1) is broken, but the user needs to complete a world upgrade to get a newer file/portage/???, but can't because of various ebuilds (like ones using this eclass) failing. Disable seccomp for these calls to keep working. Bug: https://bugs.gentoo.org/811462 Bug: https://bugs.gentoo.org/815877 Bug: https://bugs.gentoo.org/889046 Signed-off-by: Sam James --- eclass/unpacker.eclass | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/eclass/unpacker.eclass b/eclass/unpacker.eclass index 5ce681ebaa0d4..326b2fa675249 100644 --- a/eclass/unpacker.eclass +++ b/eclass/unpacker.eclass @@ -1,4 +1,4 @@ -# Copyright 1999-2022 Gentoo Authors +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # @ECLASS: unpacker.eclass @@ -122,7 +122,7 @@ unpack_pdv() { local tmpfile="${T}/${FUNCNAME}" tail -c +$((${tailskip}+1)) ${src} 2>/dev/null | head -c 512 > "${tmpfile}" - local iscompressed=$(file -b "${tmpfile}") + local iscompressed=$(file -S -b "${tmpfile}") if [[ ${iscompressed:0:8} == "compress" ]] ; then iscompressed=1 mv "${tmpfile}"{,.Z} @@ -130,7 +130,7 @@ unpack_pdv() { else iscompressed=0 fi - local istar=$(file -b "${tmpfile}") + local istar=$(file -S -b "${tmpfile}") if [[ ${istar:0:9} == "POSIX tar" ]] ; then istar=1 else @@ -244,7 +244,7 @@ unpack_makeself() { # lets grab the first few bytes of the file to figure out what kind of archive it is local decomp= filetype suffix - filetype=$("${exe[@]}" 2>/dev/null | head -c 512 | file -b -) || die + filetype=$("${exe[@]}" 2>/dev/null | head -c 512 | file -S -b -) || die case ${filetype} in *tar\ archive*) decomp=cat -- 2.39.0
[gentoo-dev] [PATCH 2/3] mono.eclass: pass -S to file to disable seccomp
Files being installed by Portage are generally trusted but also the syscalls allowed by file are quite broad anyway. With e.g. new libc or sandbox version (or any number of things...), the syscalls used by file can change which leads to its seccomp filter killing the process. This is an acceptable tradeoff when users are calling file(1), but it makes less sense with trusted input within Portage, especially where it may lead to confusing errors (swallowed within pipes, subshells, etc). Indeed, it might even be the case that file(1) is broken, but the user needs to complete a world upgrade to get a newer file/portage/???, but can't because of various ebuilds (like ones using this eclass) failing. Disable seccomp for these calls to keep working. Bug: https://bugs.gentoo.org/811462 Bug: https://bugs.gentoo.org/815877 Bug: https://bugs.gentoo.org/889046 Signed-off-by: Sam James --- eclass/mono.eclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/eclass/mono.eclass b/eclass/mono.eclass index ddea2d4b9c641..c096acc8c40ee 100644 --- a/eclass/mono.eclass +++ b/eclass/mono.eclass @@ -1,4 +1,4 @@ -# Copyright 1999-2022 Gentoo Authors +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # @ECLASS: mono.eclass @@ -76,7 +76,7 @@ mono_multilib_comply() { then for exe in "${ED}/usr/bin"/* do - if [[ "$(file "${exe}")" == *"shell script text"* ]] + if [[ "$(file -S "${exe}")" == *"shell script text"* ]] then sed -r -i -e ":/lib(/|$): s:/lib(/|$):/$(get_libdir)\1:" \ "${exe}" || die "Sedding some sense into ${exe} failed" -- 2.39.0
[gentoo-dev] [PATCH 1/3] dotnet.eclass: pass -S to file to disable seccomp
Files being installed by Portage are generally trusted but also the syscalls allowed by file are quite broad anyway. With e.g. new libc or sandbox version (or any number of things...), the syscalls used by file can change which leads to its seccomp filter killing the process. This is an acceptable tradeoff when users are calling file(1), but it makes less sense with trusted input within Portage, especially where it may lead to confusing errors (swallowed within pipes, subshells, etc). Indeed, it might even be the case that file(1) is broken, but the user needs to complete a world upgrade to get a newer file/portage/???, but can't because of various ebuilds (like ones using this eclass) failing. Disable seccomp for these calls to keep working. Bug: https://bugs.gentoo.org/811462 Bug: https://bugs.gentoo.org/815877 Bug: https://bugs.gentoo.org/889046 Signed-off-by: Sam James --- eclass/dotnet.eclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/eclass/dotnet.eclass b/eclass/dotnet.eclass index b92b9c1b40549..1fb288dd10942 100644 --- a/eclass/dotnet.eclass +++ b/eclass/dotnet.eclass @@ -1,4 +1,4 @@ -# Copyright 1999-2022 Gentoo Authors +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # @ECLASS: dotnet.eclass @@ -131,7 +131,7 @@ dotnet_multilib_comply() { then for exe in "${ED}/usr/bin"/* do - if [[ "$(file "${exe}")" == *"shell script text"* ]] + if [[ "$(file -S "${exe}")" == *"shell script text"* ]] then sed -r -i -e ":/lib(/|$): s:/lib(/|$):/$(get_libdir)\1:" \ "${exe}" || die "Sedding some sense into ${exe} failed" -- 2.39.0
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
On Mon, Jan 2, 2023 at 4:55 PM m1027 wrote: > > Many thanks for your detailed thoughs for sharing the rich > experiences on this! See below: > > antarus: > > > On Mon, Jan 2, 2023 at 4:48 AM m1027 wrote: > > > > > > Hi and happy new year. > > > > > > When we create apps on Gentoo they become easily incompatible for > > > older Gentoo systems in production where unattended remote world > > > updates are risky. This is due to new glibc, openssl-3 etc. > > > > I wrote a very long reply, but I've removed most of it: I basically > > have a few questions, and then some comments: > > > > I don't quite grasp your problem statement, so I will repeat what I > > think it is and you can confirm / deny. > > > > - Your devs build using gentoo synced against some recent tree, they > > have recent packages, and they build some software that you deploy to > > prod. > > Yes. > > > - Your prod machines are running gentoo synced against some recent > > tree, but not upgraded (maybe only glsa-check runs) and so they are > > running 'old' packages because you are afraid to update them[0] > > Well, we did sync (without updading packages) in the past but today we > even fear to sync against recent trees. Without going into details, > as a rule of thumb, weekly or monthly sync + package updates work > near to perfect. (It's cool to see what a good job emerge does on our > own internal production systems.) Updating systems older than 12 > months or so may, however, be a hugh task. And too risky for remote > production systems of customers. My primary risk I think is that even if you ship your app in a container you still need somewhere to run the containers. Currently that is a fleet of different hardware and gentoo configurations, and while containers certainly simplify your life there, they won't fix all your problems. Now instead of worrying that upgrading your Gentoo OS will break your app, it will instead break your container runtime. It is likely a smaller surface area, but it is not zero. Not saying don't use containers, just that there is no free lunch here necessarily. > > > > - Your software builds OK in dev, but when you deploy it in prod it > > breaks, because prod is really old, and your developments are using > > packages that are too new. > > Exactly. > > > > My main feedback here is: > > - Your "build" environment should be like prod. You said you didn't > > want to build "developer VMs" but I am unsure why. For example I run > > Ubuntu and I do all my gentoo development (admittedly very little > > these days) > >in a systemd-nspawn container, and I have a few shell scripts to > > mount everything and set it up (so it has a tree snapshot, some git > > repos, some writable space etc.) > > Okay, yes. That is way (1) I mentioned in my OP. It works indeed but > has the mentioned drawbacks: VMs and maintenance pile up, and for > each developer. And you don't know when there is the moment to > create a new VM. But yes it seems to me one of the ways to go: > *Before* creating a production system you need to freeze portage, > create dev VMs, and prevent updates on the VMs, too. (Freezing aka > not updating has many disadvantages, of course.) Oh sorry, I failed to understand you were doing that already. I agree it's challenging, I think if you don't have a great method to simplify here, it might not be a great avenue going forward. - Trying to figure out when you can make a new VM. - Trying to figure out when you can take a build and deploy it to a customer safely. I've seen folks try to group customers in some way to reduce the number of prod artifacts required, but if you cannot it might be The benefit of containers here is that you can basically deploy your app at whatever rate you want, and only the OS upgrades remain risky (because they might break the container runtime.) Depending on your business needs, it might be advantageous to go that route. > > > > - Your "prod" environment is too risky to upgrade, and you have > > difficulty crafting builds that run in every prod environment. I think > > this is fixable by making a build environment more like the prod > > environment. > > The challenge here is that if you have not done that (kept the > > copies of ebuilds around, the distfiles, etc) it can be challenging to > > "recreate" the existing older prod environments. > > But if you do the above thing (where devs build in a container) > > and you can make that container like the prod environments, then you > > can enable devs to build for the prod environment (in a container on > > their local machine) and get the outcome you want. > > Not sure I got your point here. But yes, it comes down to what was > said above. > > > > - Understand that not upgrading prod is like, to use a finance term, > > picking up pennies in front of a steamroller. It's a great strategy, > > but eventually you will actually *need* to upgrade something. Maybe > > for a critical security issue, maybe for a feature.
Re: [gentoo-dev] [PATCH] toolchain-funcs.eclass: Promote tc-env_build to a non-internal function
On Mon, Jan 2, 2023 at 5:34 PM James Le Cuirot wrote: > > It's generally useful and already directly used by three packages. I > need to use it to fix cross-compiling of LLVM. Sounds good to me.
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
Many thanks for your detailed thoughs for sharing the rich experiences on this! See below: antarus: > On Mon, Jan 2, 2023 at 4:48 AM m1027 wrote: > > > > Hi and happy new year. > > > > When we create apps on Gentoo they become easily incompatible for > > older Gentoo systems in production where unattended remote world > > updates are risky. This is due to new glibc, openssl-3 etc. > > I wrote a very long reply, but I've removed most of it: I basically > have a few questions, and then some comments: > > I don't quite grasp your problem statement, so I will repeat what I > think it is and you can confirm / deny. > > - Your devs build using gentoo synced against some recent tree, they > have recent packages, and they build some software that you deploy to > prod. Yes. > - Your prod machines are running gentoo synced against some recent > tree, but not upgraded (maybe only glsa-check runs) and so they are > running 'old' packages because you are afraid to update them[0] Well, we did sync (without updading packages) in the past but today we even fear to sync against recent trees. Without going into details, as a rule of thumb, weekly or monthly sync + package updates work near to perfect. (It's cool to see what a good job emerge does on our own internal production systems.) Updating systems older than 12 months or so may, however, be a hugh task. And too risky for remote production systems of customers. > - Your software builds OK in dev, but when you deploy it in prod it > breaks, because prod is really old, and your developments are using > packages that are too new. Exactly. > My main feedback here is: > - Your "build" environment should be like prod. You said you didn't > want to build "developer VMs" but I am unsure why. For example I run > Ubuntu and I do all my gentoo development (admittedly very little > these days) >in a systemd-nspawn container, and I have a few shell scripts to > mount everything and set it up (so it has a tree snapshot, some git > repos, some writable space etc.) Okay, yes. That is way (1) I mentioned in my OP. It works indeed but has the mentioned drawbacks: VMs and maintenance pile up, and for each developer. And you don't know when there is the moment to create a new VM. But yes it seems to me one of the ways to go: *Before* creating a production system you need to freeze portage, create dev VMs, and prevent updates on the VMs, too. (Freezing aka not updating has many disadvantages, of course.) > - Your "prod" environment is too risky to upgrade, and you have > difficulty crafting builds that run in every prod environment. I think > this is fixable by making a build environment more like the prod > environment. > The challenge here is that if you have not done that (kept the > copies of ebuilds around, the distfiles, etc) it can be challenging to > "recreate" the existing older prod environments. > But if you do the above thing (where devs build in a container) > and you can make that container like the prod environments, then you > can enable devs to build for the prod environment (in a container on > their local machine) and get the outcome you want. Not sure I got your point here. But yes, it comes down to what was said above. > - Understand that not upgrading prod is like, to use a finance term, > picking up pennies in front of a steamroller. It's a great strategy, > but eventually you will actually *need* to upgrade something. Maybe > for a critical security issue, maybe for a feature. Having a build > environment that matches prod is good practice, you should do it, but > you should also really schedule maintenance for these prod nodes to > get them upgraded. (For physical machines, I've often seen businesses > just eat the risk and assume the machine will physically fail before > the steamroller comes, but this is less true with virtualized > environments that have longer real lifetimes.) Yes, haha, I agree. And yes, I totally ignored backporting security here, as well as the need that we might *require* a dependend package upgrade (e.g. to fix a known memory leak). I left that out for simlicity only. > > So, what we've thought of so far is: > > > > (1) Keeping outdated developer boxes around and compile there. We > > would freeze portage against accidental emerge sync by creating a > > git branch in /var/db/repos/gentoo. This feels hacky and requires a > > increating number of develper VMs. And sometimes we are hit by a > > silent incompatibility we were not aware of. > > In general when you build binaries for some target, you should build > on that target when possible. To me, this is the crux of your issue > (that you do not) and one of the main causes of your pain. > You will need to figure out a way to either: > - Upgrade the older environments to new packages. > - Build in copies of the older environments. > > I actually expect the second one to take 1-2 sprints (so like 1 engineer > month?) > - One sprint to make
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
sam: > > On 2 Jan 2023, at 12:48, m1027 wrote: > > > > Hi and happy new year. > > > > When we create apps on Gentoo they become easily incompatible for > > older Gentoo systems in production where unattended remote world > > updates are risky. This is due to new glibc, openssl-3 etc. [...] > I'd really suggest just using stable in production and a mix > for developers so you can catch any problems beforehand. > > We try to be quite conservative about things like OpenSSL 3, > glibc updates, etc. Thanks, a misunderstanding then: I am talking about stable only. Whilst Gentoo may be conservative and all of you do an excellent job on keeping things smooth, incompatibilities of software being newly created on up-to-date developer systems (and then tried to be distributed to outdated production systems) may arise multiple times per year; *any* of the following alone may trigger a incompatibility. Just some random examples: (1) Most prominent, glibc updates. It has its sophisticated function versioning. You may or may not be hit. It is not depending on glibc's version but the internal function versions which are updated in a quite subtile way. (Function versioning is without doubt an impressive feature.) (2) libjpeg: In the past, libjpeg reached version 9 (like on Ubuntu today) but later was versioned 62 or 6.2 AFAIK. If you have an old Gentoo production system still on libjpeg-9, you have a hard time to update and distribute a new version of your app, as this version of libjpeg is not present in Gentoo anymore. (Don't get me wrong, there have probably been good reasons for this downgrade.) BTW: This little incompatibility is one of the reasons why it is hard to compile a app on Ubuntu for Gentoo. (3) An older one: libressl was removed. Well, we all remember the debate whether to keep it or not. (4) There is openssl-3 showing up on the horizon. I expect incompatibilities when distributing newly built software. (5) Portage EAPIs: If there is a new EAPI and you emerge --sync, then you need to update portage. This however, might require surprisingly many other updates. New python, setuptools and friends. I am not complaining here. Hey, we are on rolling release. Some of you may even know individual solutions to work around each of it. However, we just may get into trouble when distributing newly compiled apps (on new Gentoo systems) to older Gentoo systems. And we don't know in advance. I am looking for the best way to avoid that. Thanks
[gentoo-dev] [PATCH] eclass/dune.eclass: introduce edune and dune-compile (v2)
edune is a thin wrapper for dune, which will help to run special, uncommon dune commands; dune-compile is a function to selectively pick which packages will be compiled "for-release" (as dune call it); dune-compile without any arguments replaces the current dune_src_compile Signed-off-by: Maciej Barć --- eclass/dune.eclass | 49 +- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/eclass/dune.eclass b/eclass/dune.eclass index 4bc73eda8..384908a40 100644 --- a/eclass/dune.eclass +++ b/eclass/dune.eclass @@ -1,4 +1,4 @@ -# Copyright 1999-2022 Gentoo Authors +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # @ECLASS: dune.eclass @@ -29,7 +29,7 @@ _DUNE_ECLASS=1 # Set before inheriting the eclass. : ${DUNE_PKG_NAME:=${PN}} -inherit multiprocessing +inherit edo multiprocessing # Do not complain about CFLAGS etc since ml projects do not use them. QA_FLAGS_IGNORED='.*' @@ -44,15 +44,54 @@ BDEPEND=" dev-ml/dune " +# @FUNCTION: edune +# @USAGE: ... +# @DESCRIPTION: +# A thin wrapper for the `dune` command. +# Runs `dune` with given arguments and dies on failure. +# +# Example use: +# @CODE +# edune clean +# @CODE +edune() { + debug-print-function ${FUNCNAME} "${@}" + + edo dune "${@}" +} + +# @FUNCTION: dune-compile +# @USAGE: [package] ... +# @DESCRIPTION: +# Compiles either all of packages sources in current directory or selected +# packages. In case of all packages the package detection is done via dune +# itself. +# +# Example use: +# @CODE +# dune-compile menhir menhirLib menhirSdk +# @CODE +dune-compile() { + local -a myduneopts=( + -j $(makeopts_jobs) + --profile release + ) + if [[ -n "${1}" ]] ; then + myduneopts+=( --for-release-of-packages="$(IFS="," ; echo "${*}")" ) + fi + + edune build @install "${myduneopts[@]}" +} + dune_src_compile() { ebegin "Building" - dune build @install -j $(makeopts_jobs) --profile release + dune-compile eend $? || die } dune_src_test() { ebegin "Testing" - dune runtest -j $(makeopts_jobs) --profile release + edune runtest -j $(makeopts_jobs) --profile release eend $? || die } @@ -80,7 +119,7 @@ dune-install() { local pkg for pkg in "${pkgs[@]}" ; do ebegin "Installing ${pkg}" - dune install ${myduneopts[@]} ${pkg} + edune install ${myduneopts[@]} ${pkg} eend $? || die # Move docs to the appropriate place. -- 2.38.2
Re: [gentoo-dev] [PATCH] eclass/dune.eclass: introduce edune and dune-compile
On 2023-01-02 22:37, Maciej Barć wrote: > edune is a thin wrapper for dune, which will help to run special, > uncommon dune commands; > dune-compile is a function to selectively pick which packages will be > compiled "for-release" (as dune call it); > dune-compile without any arguments replaces the current dune_src_compile > > Signed-off-by: Maciej Barć > --- > eclass/dune.eclass | 47 ++ > 1 file changed, 43 insertions(+), 4 deletions(-) > > diff --git a/eclass/dune.eclass b/eclass/dune.eclass > index 4bc73eda8..6c760accd 100644 > --- a/eclass/dune.eclass > +++ b/eclass/dune.eclass > @@ -1,4 +1,4 @@ > -# Copyright 1999-2022 Gentoo Authors > +# Copyright 1999-2023 Gentoo Authors > # Distributed under the terms of the GNU General Public License v2 > > # @ECLASS: dune.eclass > @@ -44,15 +44,54 @@ BDEPEND=" > dev-ml/dune > " > > +# @FUNCTION: edune > +# @USAGE: ... > +# @DESCRIPTION: > +# A thin wrapper for the `dune` command. > +# Runs `dune` with given arguments and dies on failure. > +# > +# Example use: > +# @CODE > +# edune clean > +# @CODE > +edune() { > + debug-print-function ${FUNCNAME} "${@}" > + > + dune "${@}" || die "dune call failed, given arguments: ${@}" Please use "edo" to also log the command being executed. > +} > + > +# @FUNCTION: dune-compile > +# @USAGE: [package] ... > +# @DESCRIPTION: > +# Compiles either all of packages sources in current directory or selected > +# packages. In case of all packages the package detection is done via dune > +# itself. > +# > +# Example use: > +# @CODE > +# dune-compile menhir menhirLib menhirSdk > +# @CODE > +dune-compile() { > + local -a myduneopts=( > + -j $(makeopts_jobs) > + --profile release > + ) > + if [[ -n "${1}" ]] ; then > + myduneopts+=( --for-release-of-packages="$(IFS="," ; echo > "${*}")" ) > + fi > + > + edune build @install "${myduneopts[@]}" > +} > + > dune_src_compile() { > ebegin "Building" > - dune build @install -j $(makeopts_jobs) --profile release > + dune-compile > eend $? || die > } > > dune_src_test() { > ebegin "Testing" > - dune runtest -j $(makeopts_jobs) --profile release > + edune runtest -j $(makeopts_jobs) --profile release > eend $? || die > } > > @@ -80,7 +119,7 @@ dune-install() { > local pkg > for pkg in "${pkgs[@]}" ; do > ebegin "Installing ${pkg}" > - dune install ${myduneopts[@]} ${pkg} > + edune install ${myduneopts[@]} ${pkg} > eend $? || die > > # Move docs to the appropriate place. > -- > 2.38.2 > >
[gentoo-dev] [PATCH] toolchain-funcs.eclass: Promote tc-env_build to a non-internal function
It's generally useful and already directly used by three packages. I need to use it to fix cross-compiling of LLVM. Signed-off-by: James Le Cuirot --- eclass/toolchain-funcs.eclass | 1 - 1 file changed, 1 deletion(-) diff --git a/eclass/toolchain-funcs.eclass b/eclass/toolchain-funcs.eclass index 61a29d1b6ea6..bfcd6819ed0b 100644 --- a/eclass/toolchain-funcs.eclass +++ b/eclass/toolchain-funcs.eclass @@ -377,7 +377,6 @@ tc-export_build_env() { # @FUNCTION: tc-env_build # @USAGE: [command args] -# @INTERNAL # @DESCRIPTION: # Setup the compile environment to the build tools and then execute the # specified command. We use tc-getBUILD_XX here so that we work with -- 2.38.1
[gentoo-dev] Last rites: dev-cpp/pngpp
# David Seifert (2023-01-02) # EAPI 6, pretty much no upstream activity, outdated, last upstream # release over 3 years ago, no revdeps. Removal on 2023-02-01. dev-cpp/pngpp signature.asc Description: This is a digitally signed message part
[gentoo-dev] [PATCH] eclass/dune.eclass: introduce edune and dune-compile
edune is a thin wrapper for dune, which will help to run special, uncommon dune commands; dune-compile is a function to selectively pick which packages will be compiled "for-release" (as dune call it); dune-compile without any arguments replaces the current dune_src_compile Signed-off-by: Maciej Barć --- eclass/dune.eclass | 47 ++ 1 file changed, 43 insertions(+), 4 deletions(-) diff --git a/eclass/dune.eclass b/eclass/dune.eclass index 4bc73eda8..6c760accd 100644 --- a/eclass/dune.eclass +++ b/eclass/dune.eclass @@ -1,4 +1,4 @@ -# Copyright 1999-2022 Gentoo Authors +# Copyright 1999-2023 Gentoo Authors # Distributed under the terms of the GNU General Public License v2 # @ECLASS: dune.eclass @@ -44,15 +44,54 @@ BDEPEND=" dev-ml/dune " +# @FUNCTION: edune +# @USAGE: ... +# @DESCRIPTION: +# A thin wrapper for the `dune` command. +# Runs `dune` with given arguments and dies on failure. +# +# Example use: +# @CODE +# edune clean +# @CODE +edune() { + debug-print-function ${FUNCNAME} "${@}" + + dune "${@}" || die "dune call failed, given arguments: ${@}" +} + +# @FUNCTION: dune-compile +# @USAGE: [package] ... +# @DESCRIPTION: +# Compiles either all of packages sources in current directory or selected +# packages. In case of all packages the package detection is done via dune +# itself. +# +# Example use: +# @CODE +# dune-compile menhir menhirLib menhirSdk +# @CODE +dune-compile() { + local -a myduneopts=( + -j $(makeopts_jobs) + --profile release + ) + if [[ -n "${1}" ]] ; then + myduneopts+=( --for-release-of-packages="$(IFS="," ; echo "${*}")" ) + fi + + edune build @install "${myduneopts[@]}" +} + dune_src_compile() { ebegin "Building" - dune build @install -j $(makeopts_jobs) --profile release + dune-compile eend $? || die } dune_src_test() { ebegin "Testing" - dune runtest -j $(makeopts_jobs) --profile release + edune runtest -j $(makeopts_jobs) --profile release eend $? || die } @@ -80,7 +119,7 @@ dune-install() { local pkg for pkg in "${pkgs[@]}" ; do ebegin "Installing ${pkg}" - dune install ${myduneopts[@]} ${pkg} + edune install ${myduneopts[@]} ${pkg} eend $? || die # Move docs to the appropriate place. -- 2.38.2
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
On Mon, Jan 2, 2023 at 4:48 AM m1027 wrote: > > Hi and happy new year. > > When we create apps on Gentoo they become easily incompatible for > older Gentoo systems in production where unattended remote world > updates are risky. This is due to new glibc, openssl-3 etc. I wrote a very long reply, but I've removed most of it: I basically have a few questions, and then some comments: I don't quite grasp your problem statement, so I will repeat what I think it is and you can confirm / deny. - Your devs build using gentoo synced against some recent tree, they have recent packages, and they build some software that you deploy to prod. - Your prod machines are running gentoo synced against some recent tree, but not upgraded (maybe only glsa-check runs) and so they are running 'old' packages because you are afraid to update them[0] - Your software builds OK in dev, but when you deploy it in prod it breaks, because prod is really old, and your developments are using packages that are too new. My main feedback here is: - Your "build" environment should be like prod. You said you didn't want to build "developer VMs" but I am unsure why. For example I run Ubuntu and I do all my gentoo development (admittedly very little these days) in a systemd-nspawn container, and I have a few shell scripts to mount everything and set it up (so it has a tree snapshot, some git repos, some writable space etc.) - Your "prod" environment is too risky to upgrade, and you have difficulty crafting builds that run in every prod environment. I think this is fixable by making a build environment more like the prod environment. The challenge here is that if you have not done that (kept the copies of ebuilds around, the distfiles, etc) it can be challenging to "recreate" the existing older prod environments. But if you do the above thing (where devs build in a container) and you can make that container like the prod environments, then you can enable devs to build for the prod environment (in a container on their local machine) and get the outcome you want. - Understand that not upgrading prod is like, to use a finance term, picking up pennies in front of a steamroller. It's a great strategy, but eventually you will actually *need* to upgrade something. Maybe for a critical security issue, maybe for a feature. Having a build environment that matches prod is good practice, you should do it, but you should also really schedule maintenance for these prod nodes to get them upgraded. (For physical machines, I've often seen businesses just eat the risk and assume the machine will physically fail before the steamroller comes, but this is less true with virtualized environments that have longer real lifetimes.) > > So, what we've thought of so far is: > > (1) Keeping outdated developer boxes around and compile there. We > would freeze portage against accidental emerge sync by creating a > git branch in /var/db/repos/gentoo. This feels hacky and requires a > increating number of develper VMs. And sometimes we are hit by a > silent incompatibility we were not aware of. In general when you build binaries for some target, you should build on that target when possible. To me, this is the crux of your issue (that you do not) and one of the main causes of your pain. You will need to figure out a way to either: - Upgrade the older environments to new packages. - Build in copies of the older environments. I actually expect the second one to take 1-2 sprints (so like 1 engineer month?) - One sprint to make some scripts that makes a new production 'container' - One sprint to sort of integrate that container into your dev workflow, so devs build in the container instead of what they build in now. It might be more or less daunting depending on how many distinct (unique?) prod environments you have (how many containers will you actually need for good build coverage?), how experienced in Gentoo your developers are, and how many artifacts from prod you have. - A few crazy ideas are like: - Snapshot an existing prod machine, strip of it machine-specific bits, and use that as your container. - Use quickpkg to generate a bunch of bin pkgs from a prod machine, use that to bootstrap a container. - Probably some other exciting ideas on the list ;) > > (2) Using Ubuntu LTS for production and Gentoo for development is > hit by subtile libjpeg incompatibilites and such. I would advise, if possible, to make dev and prod as similar as possible[1]. I'd be curious what blockers you think there are to this pattern. Remember that "dev" is not "whatever your devs are using" but is ideally some maintained environment; segmented from their daily driver computer (somehow). > > (3) Distributing apps as VMs or docker: Even those tools advance and > become incompatible, right? And not suitable when for smaller Arm > devices. I think if your apps are small and self-contained and easily rebuilt, your (3) and (4) can be workable. If you need
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
Peter Stuge wrote: > Essentially you will be maintaining a private fork of gentoo.git, If this seems too heavy handed then you can just as well do the reverse: Maintain an overlay repo with the packages you care to control in the state you care to have them, set that in the catalyst stage4.spec portage_overlay and add unwanted package versions in gentoo.git to the package.mask directory used by catalyst. This may sound complicated but it isn't bad at all. For total control also make your own profile, e.g. based on embedded, but that's not per se neccessary, only if the standard profiles has too much conflicts with what you want in @system. catalyst will rebuild @system according to spec file but with too much difference that just becomes annoying and feels more trouble than a controlled profile. This approach falls somewhere between your options (1) and (5). Good luck! //Peter
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
Hi, m1027 wrote: > So, what we've thought of so far is: > > (1) Keeping outdated developer boxes around and compile there. We > would freeze portage against accidental emerge sync by creating a > git branch in /var/db/repos/gentoo. This feels hacky and requires a > increating number of develper VMs. And sometimes we are hit by a > silent incompatibility we were not aware of. .. > (5) Inventing a full fledged OTA Gentoo OS updater and distribute > that together with the apps... Nah. > > Hm... Comments welcome. I recommend taking ownership (and responsibility) of your OS. Gentoo tooling (catalyst) is really fantastic for doing so. Essentially you will be maintaining a private fork of gentoo.git, but one where you only really need to manually process the packages you care to control, only when you care to control them. Use catalyst to build tarballs (and binpkgs) from snapshots of that repo. emerge can install binpkg at least from FTP. //Peter
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
> On 2 Jan 2023, at 12:48, m1027 wrote: > > Hi and happy new year. > > When we create apps on Gentoo they become easily incompatible for > older Gentoo systems in production where unattended remote world > updates are risky. This is due to new glibc, openssl-3 etc. > > So, what we've thought of so far is: > > (1) Keeping outdated developer boxes around and compile there. We > would freeze portage against accidental emerge sync by creating a > git branch in /var/db/repos/gentoo. This feels hacky and requires a > increating number of develper VMs. And sometimes we are hit by a > silent incompatibility we were not aware of. > > (2) Using Ubuntu LTS for production and Gentoo for development is > hit by subtile libjpeg incompatibilites and such. > > (3) Distributing apps as VMs or docker: Even those tools advance and > become incompatible, right? And not suitable when for smaller Arm > devices. > > (4) Flatpak: No experience, does it work well? > > (5) Inventing a full fledged OTA Gentoo OS updater and distribute > that together with the apps... Nah. > > Hm... Comments welcome. I'd really suggest just using stable in production and a mix for developers so you can catch any problems beforehand. We try to be quite conservative about things like OpenSSL 3, glibc updates, etc. signature.asc Description: Message signed with OpenPGP
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
On Mon, 2023-01-02 at 12:48 +, m1027 wrote: > Hi and happy new year. > > When we create apps on Gentoo they become easily incompatible for > older Gentoo systems in production where unattended remote world > updates are risky. This is due to new glibc, openssl-3 etc. > Just update them. YOLO. Seriously though, most of us run Gentoo in some sort of production setting. Updating often is your best bet as it keeps the list of suspects short when things do break. OTOH I'm only about 15km from our servers when I set them on fire, so keep that in mind if yours are on another continent. If updating frequently is truly out of the question, a rolling release probably isn't the best deployment target for you.
Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?
On Mon, Jan 2, 2023 at 7:48 AM m1027 wrote: > > When we create apps on Gentoo they become easily incompatible for > older Gentoo systems in production where unattended remote world > updates are risky. This is due to new glibc, openssl-3 etc. So, unless you're proposing some improvement this might be better-suited for the -user list. Officially Gentoo doesn't really "support" a stable/LTS/release-based environment. That isn't to say that it couldn't be made to do this, however, a more release-based process is one of the core competencies of most other distros, so I'm not sure how much Gentoo would add here vs just running something else. > (1) Keeping outdated developer boxes around and compile there. We > would freeze portage against accidental emerge sync by creating a > git branch in /var/db/repos/gentoo. This feels hacky and requires a > increating number of develper VMs. And sometimes we are hit by a > silent incompatibility we were not aware of. If you're running a large production environment and want a stable platform to target, I'd think you'd want to minimize the number of those you have. You wouldn't want every server running a different base OS, and then have a branch to target development at it. The idea of basing your own releases on Gentoo is probably something many companies do. All you need is to just host one repository and distfile mirror for it, and point all your production hosts at it. Then you could have staging/development environments, and promote the core OS as appropriate. Then your migration path is the same for all hosts and you can account for major vs minor changes. You do need to mirror distfiles as one of the weaknesses of Gentoo for a release-based strategy is that we do not have a solid solution for archiving things like patches/services/etc that are distributed outside of the repo. If a package is removed from the current repo no QA process ensures that the SRC_URIs for anything they used remain stable. Of course somebody would need to do the work but it shouldn't be THAT difficult to have a Gentoo Reference Release project that basically just snapshots the repository/distfiles at points in time and provides a guarantee of clean updates between milestones. Of course if you want backported security updates on top of that this would be quite a bit more effort. It would mainly be useful for those who aren't updating regularly, but completely ignoring all updates isn't a good practice regardless which is probably part of why this hasn't happened. You could greatly minimize the cost of backporting security updates if you only targeted packages of interest to you, or where the updates are low-risk to you, but that depends quite a bit on your own needs. > (3) Distributing apps as VMs or docker: Even those tools advance and > become incompatible, right? And not suitable when for smaller Arm > devices. I don't see why containers would be unsuitable for ARM. VMs certainly are memory hogs but well-designed containers don't really consume much more memory than the service they are hosting. Of course if you dump a whole OS in a container that will consume a fair bit of RAM. An advantage of containers is that they make the host OS less relevant. You could run a release-based distro with security-only updates on the host, and then application containers are low-risk to update remotely and can be built in whatever environment they are most suited to. There is a reason k8s is popular. -- Rich
[gentoo-dev] Gentoo LTS or: proper backward compatibility?
Hi and happy new year. When we create apps on Gentoo they become easily incompatible for older Gentoo systems in production where unattended remote world updates are risky. This is due to new glibc, openssl-3 etc. So, what we've thought of so far is: (1) Keeping outdated developer boxes around and compile there. We would freeze portage against accidental emerge sync by creating a git branch in /var/db/repos/gentoo. This feels hacky and requires a increating number of develper VMs. And sometimes we are hit by a silent incompatibility we were not aware of. (2) Using Ubuntu LTS for production and Gentoo for development is hit by subtile libjpeg incompatibilites and such. (3) Distributing apps as VMs or docker: Even those tools advance and become incompatible, right? And not suitable when for smaller Arm devices. (4) Flatpak: No experience, does it work well? (5) Inventing a full fledged OTA Gentoo OS updater and distribute that together with the apps... Nah. Hm... Comments welcome. Thanks
Re: [gentoo-dev] [PATCH 1/5] out-of-source-utils.eclass: New utility eclass
On Mon, 2023-01-02 at 10:25 +0100, Ulrich Mueller wrote: > > > > > > On Sun, 01 Jan 2023, Michał Górny wrote: > > > +case ${EAPI} in > > + 6|7|8) ;; > > + *) die "${ECLASS}: EAPI ${EAPI} unsupported." > > Are you sure that this will work without the final ;; terminator? > (Bash documentation says that a terminator is mandatory.) Interesting enough, it does work. > Apart from that, I'd suggest to use the standard clause as in other > eclasses: > > *) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;; > Actually, this is what app-vim/gentoo-syntax spews by default. I guess I need to change it. Anyway, replaced by your snippet. I suppose it's fine if I don't resubmit the batch for this change. -- Best regards, Michał Górny
Re: [gentoo-dev] [PATCH 1/5] out-of-source-utils.eclass: New utility eclass
> On Sun, 01 Jan 2023, Michał Górny wrote: > +case ${EAPI} in > + 6|7|8) ;; > + *) die "${ECLASS}: EAPI ${EAPI} unsupported." Are you sure that this will work without the final ;; terminator? (Bash documentation says that a terminator is mandatory.) Apart from that, I'd suggest to use the standard clause as in other eclasses: *) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;; > +esac signature.asc Description: PGP signature