[gentoo-dev] [PATCH 3/3] texlive-module.eclass: include PIPESTATUS in die message

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-module.eclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index 7d40bfc58994..98c28f578216 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -539,11 +539,12 @@ texlive-module_src_install() {
find texmf-dist/doc/man -type f -name '*.[0-9n]' -print 
|
grep -v "${grep_expressions[@]}" |
xargs -d '\n' --no-run-if-empty nonfatal doman
+   local pipestatus="${PIPESTATUS[*]}"
# The grep in the middle of the pipe may return 1 in 
case
# everything from the input is dropped.
# See https://bugs.gentoo.org/931994
-   [[ "${PIPESTATUS[*]}" == "0 "[01]" 0" ]]
-   eend $? || die "error installing man pages"
+   [[ "${pipestatus}" == "0 "[01]" 0" ]]
+   eend $? || die "error installing man pages (pipestatus: 
${pipestatus})"
 
# Delete all man pages under texmf-dist/doc/man
find texmf-dist/doc/man -type f -name '*.[0-9n]' 
-delete ||
-- 
2.44.1

[gentoo-dev] [PATCH 2/3] texlive-module.eclass: invoke doman with nonfatal

[Florian Schmaus]

In case doman fails, we do not want to die immediatly as it would
"break" the ebegin/eend combination. Instead, the exit status is passed
through xargs. It will then subsequently appear in PIPESTATUS, where it
is processed by eend || die.

Signed-off-by: Florian Schmaus 
Suggested-by: Ulrich Müller 
---
 eclass/texlive-module.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index b202a0188b66..7d40bfc58994 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -538,7 +538,7 @@ texlive-module_src_install() {
ebegin "Installing man pages"
find texmf-dist/doc/man -type f -name '*.[0-9n]' -print 
|
grep -v "${grep_expressions[@]}" |
-   xargs -d '\n' --no-run-if-empty doman
+   xargs -d '\n' --no-run-if-empty nonfatal doman
# The grep in the middle of the pipe may return 1 in 
case
# everything from the input is dropped.
# See https://bugs.gentoo.org/931994
-- 
2.44.1

[gentoo-dev] [PATCH 1/3] texlive-module.eclass: do not treat grep returning 1 as error

[Florian Schmaus]

In case every man page of the dev-texlive/* package is filtered, because
the man pages are already installed by texlive-core, grep returns an
exit status of 1, which we must not treat as an error condition.

Adjust the PIPESTATUS comparison accordingly.

Closes: https://bugs.gentoo.org/931994
Signed-off-by: Florian Schmaus 
---
 eclass/texlive-module.eclass | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index 0daca41961ff..b202a0188b66 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -539,7 +539,10 @@ texlive-module_src_install() {
find texmf-dist/doc/man -type f -name '*.[0-9n]' -print 
|
grep -v "${grep_expressions[@]}" |
xargs -d '\n' --no-run-if-empty doman
-   [[ "${PIPESTATUS[*]}" =~ ^0(" 0")*$ ]]
+   # The grep in the middle of the pipe may return 1 in 
case
+   # everything from the input is dropped.
+   # See https://bugs.gentoo.org/931994
+   [[ "${PIPESTATUS[*]}" == "0 "[01]" 0" ]]
eend $? || die "error installing man pages"
 
# Delete all man pages under texmf-dist/doc/man
-- 
2.44.1

[gentoo-dev] Obtaining values for --jobs and --load-average

[Florian Schmaus]

On 12/05/2024 20.21, Michał Górny wrote:

On Sun, 2024-05-12 at 19:22 +0200, Florian Schmaus wrote:

On 12/05/2024 04.26, Michał Górny wrote:

+   if [[ ${PKGBUMPING} != ${PVR} ]]; then
+   pushd "${DISTDIR}" >/dev/null || die
+
+   ebegin "Unpacking crates"
+   printf '%s\0' "${crates[@]}" |
+   xargs -0 -P "$(makeopts_jobs)" -n 1 -- \


Consider using get_makeopts_jobs instead of makeopts_jobs, as it
searches more variables for --jobs.

I'm sorry, I was wrong.

I thought I remembered that we only extended the new get_makeopts_* 
functions so that they also search other relevant variables besides 
MAKEOPTS for --jobs/--load-avgerage. This is wrong, the already existing 
makeopts_* functions were also changed at that time so that they take 
GNUMAKEFLAGS & Co. into account.


As a reminder, this change was due to newer Portage versions setting 
reasonable default values for --jobs and --load-average. And 
GNUMAKEFLAGS is used for --load-average.


The problem is that the order of the parameters of the makeopts_* 
functions is unfavorable. First comes the (optional) opts/flags string, 
then the (optional) default value. This means that as soon as you want 
to specify the default value explicitly, you also have to specify the 
opts/flags string. As a result, it can easily happen that only $MAKEOPTS 
is specified, which can lead to the default value set by portage not 
being taken into account, especially --load-average. This is, for 
example, the case with app-i18n/mozc and dev-qt/qtwebengine.


The get_makeopts_* functions do not suffer from this issue.


> My suggestion would be for the person who introduced new methods
> and implicitly claimed the old methods to be "legacy" to put an actual
> effort to migrate consumers.

That is probably sensible and I believe also possible.

I could only find a single ebuild where makeopts_ was used with a custom 
make opts/flags string: sci-libs/vtk which has some logic for NINJAOPTS.


However, I wonder if it is ever sensible to specify other make 
opts/flags besides what _get_all_makeopts() from multiprocessing.eclass 
considers.


If your make.conf contains

NINJAOPS="-j 4 -l 4 "

then you most likely always also want to set

MAKEOPTS="-j 4"

otherwise, only ninja builds are done with "-j 4".

Of course, we could also simply add $NINJAOPTS to _get_all_makeopts(). 
Consequentially this raises the question if other similar variables 
should also be added (SAMUFLAGS, maybe?).


I'd love to hear some opinions on this.

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] cargo.eclass: Optimize crate unpacking

[Florian Schmaus]

On 12/05/2024 20.21, Michał Górny wrote:

On Sun, 2024-05-12 at 19:22 +0200, Florian Schmaus wrote:

On 12/05/2024 04.26, Michał Górny wrote:

+   if [[ ${PKGBUMPING} != ${PVR} ]]; then
+   pushd "${DISTDIR}" >/dev/null || die
+
+   ebegin "Unpacking crates"
+   printf '%s\0' "${crates[@]}" |
+   xargs -0 -P "$(makeopts_jobs)" -n 1 -- \


Consider using get_makeopts_jobs instead of makeopts_jobs, as it
searches more variables for --jobs.


Whose bright idea was to add a second similarly named function that does
roughly the same thing but apparently differently?  It can hardly get
more confusing.


You are absolutely right, it sucks that we have two very similar methods.

You are invited to suggest how the situation can be improved. However, 
rambling without presenting alternatives is not helpful in any way.


Potentially, you will either discover that there is a reason why things 
are the way they are, or find a better solution.


- Flow

Re: [gentoo-dev] [PATCH] cargo.eclass: Optimize crate unpacking

[Florian Schmaus]

On 12/05/2024 04.26, Michał Górny wrote:

Unpack crates in parallel using xargs to utilize multicore systems
better.  Perform checksumming via a single sha256sum invocation.

For dev-python/watchfiles, this speeds up unpacking on my machine
from 2.6 s to 0.75 s (warm cache).

Signed-off-by: Michał Górny 
---
  eclass/cargo.eclass | 56 ++---
  1 file changed, 33 insertions(+), 23 deletions(-)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 0f2da982f60c..5a16d3a30528 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -329,40 +329,50 @@ _cargo_gen_git_config() {
  cargo_src_unpack() {
debug-print-function ${FUNCNAME} "$@"
  
-	mkdir -p "${ECARGO_VENDOR}" || die

-   mkdir -p "${S}" || die
+   mkdir -p "${ECARGO_VENDOR}" "${S}" || die
  
  	local archive shasum pkg

+   local crates=()
for archive in ${A}; do
case "${archive}" in
*.crate)
-   # when called by pkgdiff-mg, do not unpack 
crates
-   [[ ${PKGBUMPING} == ${PVR} ]] && continue
-
-   ebegin "Loading ${archive} into Cargo registry"
-   tar -xf "${DISTDIR}"/${archive} -C 
"${ECARGO_VENDOR}/" || die
-   # generate sha256sum of the crate itself as 
cargo needs this
-   shasum=$(sha256sum "${DISTDIR}"/${archive} | 
cut -d ' ' -f 1)
-   pkg=$(basename ${archive} .crate)
-   cat <<- EOF > 
${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json
-   {
-   "package": "${shasum}",
-   "files": {}
-   }
-   EOF
-   # if this is our target package we need it in 
${WORKDIR} too
-   # to make ${S} (and handle any revisions too)
-   if [[ ${P} == ${pkg}* ]]; then
-   tar -xf "${DISTDIR}"/${archive} -C 
"${WORKDIR}" || die
-   fi
-   eend $?
+   crates+=( "${archive}" )
;;
*)
-   unpack ${archive}
+   unpack "${archive}"
;;
esac
done
  
+	if [[ ${PKGBUMPING} != ${PVR} ]]; then

+   pushd "${DISTDIR}" >/dev/null || die
+
+   ebegin "Unpacking crates"
+   printf '%s\0' "${crates[@]}" |
+   xargs -0 -P "$(makeopts_jobs)" -n 1 -- \


Consider using get_makeopts_jobs instead of makeopts_jobs, as it 
searches more variables for --jobs.


N.B.: If this where asking for a load-average limit, then using 
get_makeopts_loadavg would be the ideal way to pick up portage's default 
wrt --load-average. Therefore we should IMHO encourage using the 
get_makeopts_* functions over the (legacy?) non-get_ variants.



+   tar -x -C "${ECARGO_VENDOR}" -f
+   assert
+   eend $?
+
+   while read -d '' -r shasum archive; do
+   pkg=${archive%.crate}
+   cat <<- EOF > 
${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json || die
+   {
+   "package": "${shasum}",
+   "files": {}
+   }
+   EOF
+
+   # if this is our target package we need it in 
${WORKDIR} too
+   # to make ${S} (and handle any revisions too)
+   if [[ ${P} == ${pkg}* ]]; then
+   tar -xf "${archive}" -C "${WORKDIR}" || die
+   fi
+   done < <(sha256sum -z "${crates[@]}" || die)
+
+   popd >/dev/null || die
+   fi
+
cargo_gen_config
  }
  




OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH 2/2] eftmutil-sys: use edob

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-common.eclass | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/eclass/texlive-common.eclass b/eclass/texlive-common.eclass
index 15d475799a88..072581dde78e 100644
--- a/eclass/texlive-common.eclass
+++ b/eclass/texlive-common.eclass
@@ -22,6 +22,8 @@ case ${EAPI} in
*) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
 esac
 
+inherit edo
+
 if [[ -z ${_TEXLIVE_COMMON_ECLASS} ]]; then
 _TEXLIVE_COMMON_ECLASS=1
 
@@ -199,9 +201,9 @@ etexmf-update() {
 efmtutil-sys() {
if has_version 'app-text/texlive-core' ; then
if [[ -z ${ROOT} && -x "${EPREFIX}"/usr/bin/fmtutil-sys ]] ; 
then
-   einfo "Rebuilding formats"
-   "${EPREFIX}"/usr/bin/fmtutil-sys --all &> /dev/null ||
-   die -n "fmtutil-sys returned non-zero exit 
status ${?}"
+   edob -m "Rebuilding TexLive formats" \
+-l fmtutils-sys-all \
+"${EPREFIX}"/usr/bin/fmtutil-sys --all
else
ewarn "Cannot run fmtutil-sys for some reason."
ewarn "Your formats might be inconsistent with your 
installed ${PN} version"
-- 
2.43.2

[gentoo-dev] [PATCH 1/2] edo.eclass: enhace edob for usage with nosiy commands

[Florian Schmaus]

Normally, edob can, or rather should, not be used with noisy commands,
i.e., commands that produce an output. This is because the output
destroys the concept of ebegin and eend, where the eend marker is shown
on the same line that is produced by ebegin.

However, it sometimes would be nice to use edob with noisy commands, but
this means to redirect stdout and stderr of those commands. Instead of
redirecting the output to /dev/null, we save the output in a log file
under T. This allows us to present the output to the user in case the
command fails, making it futhermore part of the build.log, which we
expect users to attach to bug reports.

Signed-off-by: Florian Schmaus 
---
 eclass/edo.eclass | 66 ---
 1 file changed, 56 insertions(+), 10 deletions(-)

diff --git a/eclass/edo.eclass b/eclass/edo.eclass
index c2e7ed60083f..0d410719675c 100644
--- a/eclass/edo.eclass
+++ b/eclass/edo.eclass
@@ -1,4 +1,4 @@
-# Copyright 2022 Gentoo Authors
+# Copyright 2022-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: edo.eclass
@@ -12,10 +12,14 @@
 # This eclass provides the 'edo' command, and an 'edob' variant for 
ebegin/eend,
 # which logs the command used verbosely and dies (exits) on failure.
 #
-# This eclass should be used only where needed to give a more verbose log, e.g.
-# for invoking non-standard ./configure scripts, or building objects/binaries
-# directly within ebuilds via compiler invocations. It is NOT to be used
-# in place of generic 'command || die' where verbosity is unnecessary.
+# The 'edo' command should be used only where needed to give a more verbose 
log,
+# e.g. for invoking non-standard ./configure scripts, or building
+# objects/binaries directly within ebuilds via compiler invocations.  It is NOT
+# to be used in place of generic 'command || die' where verbosity is
+# unnecessary.
+#
+# The 'edob' command should be used if its output is not of general interest,
+# as it will be only shown if the command returns a non-zero exit status.
 case ${EAPI} in
7|8) ;;
*) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
@@ -35,14 +39,56 @@ edo() {
 }
 
 # @FUNCTION: edob
-# @USAGE:  [...]
+# @USAGE: [-m ] [-l ]  [...]
 # @DESCRIPTION:
 # Executes 'command' with ebegin & eend with any given arguments and exits
-# on failure unless called under 'nonfatal'.
+# on failure unless called under 'nonfatal'.  This function redirects
+# stdout and stderr to a log file.  The content of the log file is shown
+# if the command returns with a non-zero exit status.
+#
+# If -m  is provided, then invokes ebegin with , otherwise
+# a default message is used.  If -l  is provided, then  is
+# used to construct the name of the log file where stdout and stderr of the
+# command is redirected to.
 edob() {
-   ebegin "Running $@"
-   "$@"
-   eend $? || die -n "Failed to run command: $@"
+   local message
+   local log_name
+
+   while true; do
+   case "${1}" in
+   -m|-n)
+   [[ $# -lt 2 ]] && die "Must provide an argument 
to ${1}"
+   case "${1}" in
+   -m)
+   message="${2}"
+   ;;
+   -n)
+   log="${2}"
+   ;;
+   esac
+   shift 2
+   ;;
+   *)
+   break
+   ;;
+   esac
+   done
+
+   [[ -z ${message} ]] && message="Running $@"
+   [[ -z ${log_name} ]] && log_name="$(basename ${1})"
+
+   local log_file="${T}/${log_name}.log"
+   [[ -f ${log_file} ]] && die "Log file ${log_file} exists. Consider 
using \"edob -l\""
+
+   ebegin "${message}"
+
+   "$@" &> "${log_file}"
+   local ret=$?
+
+   if ! eend $ret; then
+   cat "${log_file}"
+   die -n "Command \"$@\" failed with exit status $ret"
+   fi
 }
 
 fi
-- 
2.43.2

[gentoo-dev] [PATCH 0/2] edo.eclass: enhace edob for usage with nosiy commands

[Florian Schmaus]

The motivation for this change is to allow edob to be used with noisy
commands, i.e., commands that produce a lot of output, in cases where
the output is in general not of interest. However, if the command
fails, the output should be shown and appear in build.log.

We do this by simply redirecting the output to a file in $T, and show
this file if the command returned a non-zero exit status.

We already have a few cases in ::gentoo where such output is simply
redirected to /dev/null, hindering post-mortem analysis. Those could
be converted to edob with its new behavior.

PR at https://github.com/gentoo/gentoo/pull/36117

Florian Schmaus (2):
  edo.eclass: enhace edob for usage with nosiy commands
  eftmutil-sys: use edob

 eclass/edo.eclass| 54 
 eclass/texlive-common.eclass |  8 --
 2 files changed, 53 insertions(+), 9 deletions(-)

-- 
2.43.2

[gentoo-dev] [PATCH v2] texlive-module.eclass: install man pages in right location

[Florian Schmaus]

The TeX Live packages under dev-texlive/* currently do not install the
man pages in the right location. They are under
/usr/share/texmf-dist/doc/man.

Install them at the right location using doman.

Having dev-texlive/* packages installing man pages requires that we
filter man pages already installed by app-text/texlive-core to avoid
file collisions. As a consequence, unfortunately, we have to hard code
the list of man pages installed by texlive-core. Or, at least, folks
from #-tex and myself could not come up with a solution that does not
require hard coding that list.

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-module.eclass | 186 +++
 1 file changed, 186 insertions(+)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index 401b75bc4d11..293b6591a6b1 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -359,6 +359,192 @@ texlive-module_src_install() {
if [[ -d texmf-doc ]]; then
cp -pR texmf-doc "${ED}/usr/share/" || die
fi
+
+   if ver_test -ge 2023 && [[ ${CATEGORY} == dev-texlive ]]; then
+   local texlive_core_man_pages=(
+   afm2pl.1
+   aleph.1
+   allcm.1
+   allec.1
+   allneeded.1
+   amstex.1
+   autosp.1
+   axohelp.1
+   bibtex.1
+   chkdvifont.1
+   chktex.1
+   chkweb.1
+   ctangle.1
+   ctie.1
+   ctwill.1
+   ctwill-refsort.1
+   ctwill-twinx.1
+   cweave.1
+   cweb.1
+   detex.1
+   devnag.1
+   deweb.1
+   disdvi.1
+   dt2dv.1
+   dv2dt.1
+   dvi2fax.1
+   dvi2tty.1
+   dvibook.1
+   dviconcat.1
+   dvicopy.1
+   dvidvi.1
+   dvihp.1
+   dvilj.1
+   dvilj2p.1
+   dvilj4.1
+   dvilj4l.1
+   dvilj6.1
+   dvilualatex-dev.1
+   dviluatex.1
+   dvipdfm.1
+   dvipdfmx.1
+   dvipdft.1
+   dvipos.1
+   dvired.1
+   dviselect.1
+   dvispc.1
+   dvitodvi.1
+   dvitomp.1
+   dvitype.1
+   e2pall.1
+   ebb.1
+   eptex.1
+   euptex.1
+   extractbb.1
+   fmtutil.1
+   fmtutil.cnf.5
+   fmtutil-sys.1
+   fontinst.1
+   gftodvi.1
+   gftopk.1
+   gftype.1
+   gsftopk.1
+   hishrink.1
+   histretch.1
+   hitex.1
+   inimf.1
+   initex.1
+   kpsepath.1
+   kpsetool.1
+   kpsewhere.1
+   kpsexpand.1
+   lacheck.1
+   latex.1
+   latex-dev.1
+   luahbtex.1
+   luajittex.1
+   lualatex-dev.1
+   luatex.1
+   makeindex.1
+   makejvf.1
+   mendex.1
+   mf.1
+   mf-nowin.1
+   mft.1
+   mkindex.1
+   mkocp.1
+   mkofm.1
+   mktexfmt.1
+   mktexlsr.1
+

[gentoo-dev] [PATCH] texlive-module.eclass: install man pages in right location

[Florian Schmaus]

The TeX Live packages under dev-texlive/* currently do not install the
man pages in the right location. They are under
/usr/share/texmf-dist/doc/man.

Install them at the right location using doman.

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-module.eclass | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index 401b75bc4d11..11d869a7fa7b 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -79,7 +79,7 @@ esac
 if [[ -z ${_TEXLIVE_MODULE_ECLASS} ]]; then
 _TEXLIVE_MODULE_ECLASS=1
 
-inherit texlive-common
+inherit estack texlive-common
 
 HOMEPAGE="https://www.tug.org/texlive/;
 
@@ -359,6 +359,16 @@ texlive-module_src_install() {
if [[ -d texmf-doc ]]; then
cp -pR texmf-doc "${ED}/usr/share/" || die
fi
+
+   if ver_test -ge 2023 && [[ ${CATEGORY} == dev-texlive ]]; then
+   eshopts_push -s nullglob
+   local man_page
+   for man_page in texmf-dist/doc/man/man[1-8]/*.[1-8]; do
+   doman "${man_page}"
+   rm "${man_page}" || die
+   done
+   eshopts_pop
+   fi
else
if [[ -d texmf-dist/doc ]]; then
rm -rf texmf-dist/doc || die
-- 
2.43.2

Re: [gentoo-dev] [PATCH] texlive-common.eclass: add newline between tlbobj entries in tlpdb

[Florian Schmaus]

On 28/04/2024 14.13, Ulrich Mueller wrote:

On Sun, 28 Apr 2024, Florian Schmaus wrote:



--- a/eclass/texlive-common.eclass
+++ b/eclass/texlive-common.eclass
@@ -270,10 +270,18 @@ texlive-common_update_tlpdb() {
touch "${new_tlpdb}" || die

if [[ -d "${tlpobj}" ]]; then
-   find "${tlpobj}" -maxdepth 1 -type f -name "*.tlpobj" -print0 |
-   sort -z |
-   xargs -0 --no-run-if-empty cat >> "${new_tlpdb}"
-   assert "generating tlpdb failed"
+   all_tlpobjs() {
+   find "${tlpobj}" -maxdepth 1 -type f -name "*.tlpobj" 
-print0 |
+   sort -z
+   assert "Error enumerating all tlpobj files"
+   }
+   (
+   while IFS="" read -d $'\0' -r f; do
+   cat "${f}" || die "Failed to read ${f}"
+   # Ensure there is an empty line between every 
tlpobj entry.
+   echo
+   done < <(all_tlpobjs)
+   ) >> "${new_tlpdb}"
fi

if [[ -f ${tlpdb} ]]; then


Looks complicated. AFAICS a one-line change like the following would
also do the job:

-   xargs -0 --no-run-if-empty cat >> "${new_tlpdb}"
+   xargs -0 --no-run-if-empty sed -s '$G' >> "${new_tlpdb}"


That is exactly the feedback I was hoping for.

Thanks ulm.



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH] texlive-common.eclass: add newline between tlbobj entries in tlpdb

[Florian Schmaus]

We previously created the texlive.tlpdb by concatenating all tlpobj
files. This means that the entries will be right after another.

As it turns out, some TeX Live tools require a newline (e.g., tlmgr)
between the entries, while others do not (e.g., texdoc). And the
"official" tlpdb also has the entries separated by newlines. Therefore
this changes texlive-common_update_tlpdb() to also add them.

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-common.eclass | 16 
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/eclass/texlive-common.eclass b/eclass/texlive-common.eclass
index b32ea2af1121..929062c0444a 100644
--- a/eclass/texlive-common.eclass
+++ b/eclass/texlive-common.eclass
@@ -270,10 +270,18 @@ texlive-common_update_tlpdb() {
touch "${new_tlpdb}" || die
 
if [[ -d "${tlpobj}" ]]; then
-   find "${tlpobj}" -maxdepth 1 -type f -name "*.tlpobj" -print0 |
-   sort -z |
-   xargs -0 --no-run-if-empty cat >> "${new_tlpdb}"
-   assert "generating tlpdb failed"
+   all_tlpobjs() {
+   find "${tlpobj}" -maxdepth 1 -type f -name "*.tlpobj" 
-print0 |
+   sort -z
+   assert "Error enumerating all tlpobj files"
+   }
+   (
+   while IFS="" read -d $'\0' -r f; do
+   cat "${f}" || die "Failed to read ${f}"
+   # Ensure there is an empty line between every 
tlpobj entry.
+   echo
+   done < <(all_tlpobjs)
+   ) >> "${new_tlpdb}"
fi
 
if [[ -f ${tlpdb} ]]; then
-- 
2.43.2

Re: [gentoo-dev] Update on the 23.0 profiles

[Florian Schmaus]

On 06/04/2024 17.06, Andreas K. Huettel wrote:

Hi all,

so here's a small update on the state of the 23.0 profiles:


Thanks for the update and the work on the 23.0 profiles. :)



Most 17.x profiles have been downgraded to "exp".


I could imagine there is a reason to downgrade those back to 'exp', 
could you elaborate a bit on that?


Isn't it bit strange that a 'stable' profiles gets downgraded back to 
'exp'? Then again, I am not sure about the implications of this nor 
about the rationale behind it.


However, I also notice that there is a outstanding PR that reverts that 
[1]. Maybe we should introduce a new state 'oldstable' or so?


- Flow


1: https://github.com/gentoo/gentoo/pull/35871


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: [PATCH] efmtutil-sys: use ebegin/eend and log output

[Florian Schmaus]

I just noticed two things seconds after sending the patch:

Commit message is missing "texlive-common.eclass" prefix.

On 04/04/2024 15.01, Florian Schmaus wrote:

Use ebegin/eend and instead of redirecting the output to /dev/null
capture stdout and stderr under a file under $T.

Signed-off-by: Florian Schmaus 
---
  eclass/texlive-common.eclass | 8 +---
  1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/eclass/texlive-common.eclass b/eclass/texlive-common.eclass
index 66d3999bd103..0cc61f735cb2 100644
--- a/eclass/texlive-common.eclass
+++ b/eclass/texlive-common.eclass
@@ -199,9 +199,11 @@ etexmf-update() {
  efmtutil-sys() {
if has_version 'app-text/texlive-core' ; then
if [[ -z ${ROOT} && -x "${EPREFIX}"/usr/bin/fmtutil-sys ]] ; 
then
-   einfo "Rebuilding formats"
-   "${EPREFIX}"/usr/bin/fmtutil-sys --all &> /dev/null ||
-   die -n "fmtutil-sys returned non-zero exit status 
${?}"
+   ebegin "Rebuilding TexLive formats"
+   "${EPREFIX}"/usr/bin/fmtutil-sys --all \
+   > "${T}"/fmutil-sys-all.log \
+   &> "${T}"/fmutil-sys-all.err.log

This should be 2> instead of &>

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH] efmtutil-sys: use ebegin/eend and log output

[Florian Schmaus]

Use ebegin/eend and instead of redirecting the output to /dev/null
capture stdout and stderr under a file under $T.

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-common.eclass | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/eclass/texlive-common.eclass b/eclass/texlive-common.eclass
index 66d3999bd103..0cc61f735cb2 100644
--- a/eclass/texlive-common.eclass
+++ b/eclass/texlive-common.eclass
@@ -199,9 +199,11 @@ etexmf-update() {
 efmtutil-sys() {
if has_version 'app-text/texlive-core' ; then
if [[ -z ${ROOT} && -x "${EPREFIX}"/usr/bin/fmtutil-sys ]] ; 
then
-   einfo "Rebuilding formats"
-   "${EPREFIX}"/usr/bin/fmtutil-sys --all &> /dev/null ||
-   die -n "fmtutil-sys returned non-zero exit 
status ${?}"
+   ebegin "Rebuilding TexLive formats"
+   "${EPREFIX}"/usr/bin/fmtutil-sys --all \
+   > "${T}"/fmutil-sys-all.log \
+   &> "${T}"/fmutil-sys-all.err.log
+   eend $? || die -n "fmtutil-sys returned non-zero exit 
status ${?}"
else
ewarn "Cannot run fmtutil-sys for some reason."
ewarn "Your formats might be inconsistent with your 
installed ${PN} version"
-- 
2.43.2

[gentoo-dev] [PATCH 2/2] texlive-module.eclass: add texlive-module_update_tlpdb

[Florian Schmaus]

Update (or create) the tlpdb based on the contents of
/usr/share/tlpkg/tlpobj.

Closes: https://bugs.gentoo.org/928162
Signed-off-by: Florian Schmaus 
---
 eclass/texlive-module.eclass | 57 
 1 file changed, 57 insertions(+)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index d19e02f02647..15346a3535eb 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -420,6 +420,61 @@ texlive-module_src_install() {
texlive-common_handle_config_files
 }
 
+# @FUNCTION: texlive-module_update_tlpdb
+# @DESCRIPTION:
+# Update the TexLive package database at /usr/share/tlpkg/texlive.tlpdb.
+
+texlive-module_update_tlpdb() {
+   [[ ${TL_PV} -lt 2023 ]] && return
+
+   # If we are updating this package, then there is no need to update
+   # the tlpdb in postrm, as it will be again updated in postinst.
+   [[ -n ${REPLACING_VERSIONS} && ${EBUILD_PHASE} == postrm ]] && return
+
+   local tlpkg="${EROOT}"/usr/share/tlpkg
+   local tlpobj="${tlpkg}"/tlpobj
+   local tlpdb="${tlpkg}"/texlive.tlpdb
+
+   ebegin "Regenerating TexLive package database (${tlpdb}, 
${EBUILD_PHASE})"
+
+   local new_tlpdb="${T}"/texlive.tlpdb
+
+   touch "${new_tlpdb}" || die
+
+   find "${tlpobj}" -maxdepth 1 -type f -name "*.tlpobj" -print0 |
+   sort -z |
+   xargs -0 --no-run-if-empty cat >> "${new_tlpdb}"
+   assert "generating tlpdb failed"
+
+   if [[ -f ${tlpdb} ]]; then
+   cmp -s "${new_tlpdb}" "${tlpdb}"
+   local ret=$?
+   case ${ret} in
+   # content equal
+   0)
+   # Nothing to do, return.
+   eend 0
+   return
+   ;;
+   # content differs
+   1)
+   ;;
+   # cmp failed with an error
+   *)
+   eend ${ret} "comparing new and existing tlpdb 
failed (exit status: ${ret})"
+   die
+   ;;
+   esac
+   fi
+
+   mv "${new_tlpdb}" "${tlpdb}"
+   eend $? "moving tlpdb into position failed (exit status: ${?})" || die
+
+   if [[ ! -s ${tlpdb} ]]; then
+   rm "${tlpdb}" || die
+   fi
+}
+
 # @FUNCTION: texlive-module_pkg_postinst
 # @DESCRIPTION:
 # exported function:
@@ -428,6 +483,7 @@ texlive-module_src_install() {
 
 texlive-module_pkg_postinst() {
etexmf-update
+   texlive-module_update_tlpdb
[[ -n ${TL_MODULE_INFORMATION} ]] && elog "${TL_MODULE_INFORMATION}"
 }
 
@@ -439,6 +495,7 @@ texlive-module_pkg_postinst() {
 
 texlive-module_pkg_postrm() {
[[ -z ${REPLACING_VERSIONS} ]] && etexmf-update
+   texlive-module_update_tlpdb
 }
 
 fi
-- 
2.43.2

[gentoo-dev] [PATCH 1/2] texlive-module.eclass: only invoke etexmf-update in postinst if not replacing versions

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-module.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index 9fc4e619ff9b..d19e02f02647 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -438,7 +438,7 @@ texlive-module_pkg_postinst() {
 # installed texmf trees.
 
 texlive-module_pkg_postrm() {
-   etexmf-update
+   [[ -z ${REPLACING_VERSIONS} ]] && etexmf-update
 }
 
 fi
-- 
2.43.2

[gentoo-dev] [PATCH] texlive-common.eclass: etexmf-update to call 'die' for non-"dev-telive"

[Florian Schmaus]

This also makes etexmf-update fail for packages that are outside
dev-texlive/* and don't use the TexLive version scheme like
dev-libs/kpathsea and soon dev-tex/hevea.

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-common.eclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/eclass/texlive-common.eclass b/eclass/texlive-common.eclass
index 1e5c8a53d5cc..e8a740df1f27 100644
--- a/eclass/texlive-common.eclass
+++ b/eclass/texlive-common.eclass
@@ -179,7 +179,8 @@ etexmf-update() {
if [[ -z ${ROOT} && -x "${EPREFIX}"/usr/sbin/texmf-update ]] ; 
then
"${EPREFIX}"/usr/sbin/texmf-update
local res="${?}"
-   if [[ "${res}" -ne 0 ]] && ver_test -ge 2023; then
+   if [[ "${res}" -ne 0 ]] &&
+  { [[ ${CATEGORY} != dev-texlive ]] || 
ver_test -ge 2023; } then
die -n "texmf-update returned non-zero exit 
status ${res}"
fi
else
-- 
2.43.2

Re: [gentoo-dev] [PATCH 2/3] texlive-common.eclass: check exit status of texmf-update

[Florian Schmaus]

On 29/02/2024 21.40, Ulrich Mueller wrote:

On Thu, 29 Feb 2024, Florian Schmaus wrote:



@@ -178,6 +178,10 @@ etexmf-update() {
if has_version 'app-text/texlive-core' ; then
if [[ -z ${ROOT} && -x "${EPREFIX}"/usr/sbin/texmf-update ]] ; 
then
"${EPREFIX}"/usr/sbin/texmf-update
+   local res="${?}"
+   if [[ "${?}" -ne 0 ]] && ver_test -ge 2023; then


This condition will always be false.


Is it because assigning 'res' will set '$?'?

- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: [PATCH 1/3] texlive-module.eclass: implicitly set TL_PV if not explicitly set

[Florian Schmaus]

On 29/02/2024 15.34, Michael Orlitzky wrote:

On Thu, 2024-02-29 at 15:21 +0100, Florian Schmaus wrote:


The eclass only supports EAPIs {7,8,...} so it should suffice to
blacklist EAPI=7.


Fair point, but that would mean to remember to adjust this line once the
eclass gets support for EAPI 9.



It should do the right thing automatically when EAPI=9 is added, no? If
the goal of "-gt 8" is to match EAPI=8 and newer, then rejecting (only)
the one EAPI that's older than 8 should be equivalent without involving
a numerical comparison.


Of course, you are right. Not sure how I could miss that.

I'll go with [[ ${EAPI} != 7 ]] then.

Thanks

- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: [PATCH 1/3] texlive-module.eclass: implicitly set TL_PV if not explicitly set

[Florian Schmaus]

On 29/02/2024 15.08, Michael Orlitzky wrote:

On Thu, 2024-02-29 at 14:47 +0100, Florian Schmaus wrote:
   
+if [[ -z ${TL_PV} ]] \

+  && [[ ${EAPI} -ge 8 ]] \


I am skeptical of this construct, as in the past we had non-numeric
EAPIs. So I may have to go with EAPI == 8 for now. Input appreciated.




The eclass only supports EAPIs {7,8,...} so it should suffice to
blacklist EAPI=7.


Fair point, but that would mean to remember to adjust this line once the 
eclass gets support for EAPI 9.


It appears that bash does the right thing:

$ if [[ "eapi-future" -gt 8 ]]; then echo "is greater than 8"; else echo 
"is NOT greater than 8"; fi

is NOT greater than 8

even considering

$ if [[ "9-eapi-future" -gt 8 ]]; then echo "is greater than 8"; else 
echo "is NOT greater than 8"; fi

is greater than 8

which would be fine.

Although I prefer the current approach, it is not a hill to die on for me.

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: [PATCH 1/3] texlive-module.eclass: implicitly set TL_PV if not explicitly set

[Florian Schmaus]

On 29/02/2024 14.38, Florian Schmaus wrote:

Signed-off-by: Florian Schmaus 
---
  eclass/texlive-module.eclass | 6 ++
  1 file changed, 6 insertions(+)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index afcd4532975a..d1bf0f86185b 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -85,6 +85,12 @@ HOMEPAGE="https://www.tug.org/texlive/;
  
  IUSE="doc source"
  
+if [[ -z ${TL_PV} ]] \

+  && [[ ${EAPI} -ge 8 ]] \


I am skeptical of this construct, as in the past we had non-numeric 
EAPIs. So I may have to go with EAPI == 8 for now. Input appreciated.


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH 3/3] texlive-common.eclass: Use nonfatal-respecting die for fmtutil-sys

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-common.eclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/eclass/texlive-common.eclass b/eclass/texlive-common.eclass
index 96e962cb8027..85cdb8ff204e 100644
--- a/eclass/texlive-common.eclass
+++ b/eclass/texlive-common.eclass
@@ -199,7 +199,8 @@ efmtutil-sys() {
if has_version 'app-text/texlive-core' ; then
if [[ -z ${ROOT} && -x "${EPREFIX}"/usr/bin/fmtutil-sys ]] ; 
then
einfo "Rebuilding formats"
-   "${EPREFIX}"/usr/bin/fmtutil-sys --all &> /dev/null || 
die
+   "${EPREFIX}"/usr/bin/fmtutil-sys --all &> /dev/null \
+   || die -n "fmtutil-sys returned non-zero exit 
status ${res}"
else
ewarn "Cannot run fmtutil-sys for some reason."
ewarn "Your formats might be inconsistent with your 
installed ${PN} version"
-- 
2.43.0

[gentoo-dev] [PATCH 2/3] texlive-common.eclass: check exit status of texmf-update

[Florian Schmaus]

The texlive eclasses where traditionally lenient when it comes to the
exit status of texmf-update and fmtutil-sys, as they would return a
non-zero exit status in certain situations, especially when bootstraping
the texlive installation, i.e., when texlive-core is installed.

With the upcoming Texlive 2023 bbump we can make this more strict,
having texlive-core use nonfatal.

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-common.eclass | 4 
 1 file changed, 4 insertions(+)

diff --git a/eclass/texlive-common.eclass b/eclass/texlive-common.eclass
index fab6ff66ecd5..96e962cb8027 100644
--- a/eclass/texlive-common.eclass
+++ b/eclass/texlive-common.eclass
@@ -178,6 +178,10 @@ etexmf-update() {
if has_version 'app-text/texlive-core' ; then
if [[ -z ${ROOT} && -x "${EPREFIX}"/usr/sbin/texmf-update ]] ; 
then
"${EPREFIX}"/usr/sbin/texmf-update
+   local res="${?}"
+   if [[ "${?}" -ne 0 ]] && ver_test -ge 2023; then
+   die -n "texmf-update returned non-zero exit 
status ${res}"
+   fi
else
ewarn "Cannot run texmf-update for some reason."
ewarn "Your texmf tree might be inconsistent with your 
configuration"
-- 
2.43.0

[gentoo-dev] [PATCH 1/3] texlive-module.eclass: implicitly set TL_PV if not explicitly set

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-module.eclass | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index afcd4532975a..d1bf0f86185b 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -85,6 +85,12 @@ HOMEPAGE="https://www.tug.org/texlive/;
 
 IUSE="doc source"
 
+if [[ -z ${TL_PV} ]] \
+  && [[ ${EAPI} -ge 8 ]] \
+  && [[ ${CATEGORY} == dev-texlive ]]; then
+   TL_PV=$(ver_cut 1)
+fi
+
 RDEPEND=">=app-text/texlive-core-${TL_PV:-${PV}}"
 # We do not need anything from SYSROOT:
 #   Everything is built from the texlive install in /
-- 
2.43.0

[gentoo-dev] [PATCH 0/3] *** Three minor changes to texlive-(common|mmodule).elcass ***

[Florian Schmaus]

Following are three minor changes to texlive-(common|module).eclass,
which I expect to be the last changes to the eclasses before start
moving texlive 2023 from ::tex-overlay into ::gentoo (initally
pmasked).

Florian Schmaus (3):
  texlive-module.eclass: implicitly set TL_PV if not explicitly set
  texlive-common.eclass: check exit status of texmf-update
  texlive-common.eclass: Use nonfatal-respecting die for fmtutil-sys

 eclass/texlive-common.eclass | 7 ++-
 eclass/texlive-module.eclass | 6 ++
 2 files changed, 12 insertions(+), 1 deletion(-)

-- 
2.43.0

[gentoo-dev] [PATCH 2/2] texlive-{common,module}.eclass: update for TeX Live 2023

[Florian Schmaus]

Update the texlive eclasses for the upcoming TeX Live 2023 bump. This
includes general cleanup of the eclasses, e.g., reducing the scope of
variables.

Notably changes include that installation of the TeX Live distribution
changes from /usr/share/texmf to /usr/share/texmf-dist, which is the
correct place, according to upstream.

Co-authored-by: Paul Zander 
Signed-off-by: Paul Zander 
Signed-off-by: Florian Schmaus 
---
 eclass/texlive-common.eclass | 113 +--
 eclass/texlive-module.eclass |  77 
 2 files changed, 108 insertions(+), 82 deletions(-)

diff --git a/eclass/texlive-common.eclass b/eclass/texlive-common.eclass
index f43d10926857..5de58300f2f3 100644
--- a/eclass/texlive-common.eclass
+++ b/eclass/texlive-common.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2023 Gentoo Authors
+# Copyright 1999-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: texlive-common.eclass
@@ -6,7 +6,7 @@
 # t...@gentoo.org
 # @AUTHOR:
 # Original Author: Alexis Ballier 
-# @SUPPORTED_EAPIS: 7
+# @SUPPORTED_EAPIS: 7 8
 # @BLURB: Provide various functions used by both texlive-core and texlive 
modules
 # @DESCRIPTION:
 # Purpose: Provide various functions used by both texlive-core and texlive
@@ -15,53 +15,66 @@
 # Note that this eclass *must* not assume the presence of any standard tex too
 
 case ${EAPI} in
-   7) inherit eapi8-dosym ;;
+   7)
+   inherit eapi8-dosym
+   dosym(){ dosym8 "$@"; }
+   ;;
+   8) ;;
*) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
 esac
 
 if [[ -z ${_TEXLIVE_COMMON_ECLASS} ]]; then
 _TEXLIVE_COMMON_ECLASS=1
 
-TEXMF_PATH=/usr/share/texmf
-TEXMF_DIST_PATH=/usr/share/texmf-dist
-TEXMF_VAR_PATH=/var/lib/texmf
-
 # @FUNCTION: texlive-common_handle_config_files
 # @DESCRIPTION:
 # Has to be called in src_install after having installed the files in ${D}
-# This function will move the relevant files to /etc/texmf and symling them
+# This function will move the relevant files to /etc/texmf and symlink them
 # from their original location. This is to allow easy update of texlive's
-# configuration
-
+# configuration.
+# Called by app-text/texlive-core and texlive-module.eclass.
 texlive-common_handle_config_files() {
+   local texmf_path
+   # Starting with TeX Live 2023, we install in texmf-dist, where a
+   # distribution-provided TeX Live installation is supposed to be,
+   # instead of texmf.
+   if ver_test -ge 2023; then
+   texmf_path=/usr/share/texmf-dist
+   else
+   texmf_path=/usr/share/texmf
+   fi
+
# Handle config files properly
-   [[ -d ${ED}${TEXMF_PATH} ]] || return
-   cd "${ED}${TEXMF_PATH}" || die
+   [[ -d ${ED}${texmf_path} ]] || return
+   cd "${ED}${texmf_path}" || die
 
while read -r f; do
-   if [[ ${f#*config} != ${f} || ${f#doc} != ${f} || ${f#source} 
!= ${f} || ${f#tex} != ${f} ]] ; then
+   if [[ ${f#*config} != "${f}" || ${f#doc} != "${f}" || 
${f#source} != "${f}" || ${f#tex} != "${f}" ]] ; then
continue
fi
-   dodir /etc/texmf/$(dirname ${f}).d
-   einfo "Moving (and symlinking) ${EPREFIX}${TEXMF_PATH}/${f} to 
${EPREFIX}/etc/texmf/$(dirname ${f}).d"
-   mv "${ED}/${TEXMF_PATH}/${f}" "${ED}/etc/texmf/$(dirname 
${f}).d" || die "mv ${f} failed."
-   dosym8 -r /etc/texmf/$(dirname ${f}).d/$(basename ${f}) 
${TEXMF_PATH}/${f}
-   done < <(find  -name '*.cnf' -type f -o -name '*.cfg' -type f | sed -e 
"s:\./::g")
+   local rel_dir
+   rel_dir="$(dirname "${f}")"
+
+   dodir "/etc/texmf/${rel_dir}.d"
+   einfo "Moving (and symlinking) ${EPREFIX}${texmf_path}/${f} to 
${EPREFIX}/etc/texmf/${rel_dir}.d"
+   mv "${ED}/${texmf_path}/${f}" "${ED}/etc/texmf/${rel_dir}.d" || 
die "mv ${f} failed."
+   dosym -r "/etc/texmf/${rel_dir}.d/$(basename "${f}")" 
"${texmf_path}/${f}"
+   done < <(find . -name '*.cnf' -type f -o -name '*.cfg' -type f | sed -e 
"s:\./::g")
 }
 
 # @FUNCTION: texlive-common_is_file_present_in_texmf
 # @DESCRIPTION:
 # Return if a file is present in the texmf tree
 # Call it from the directory containing texmf and texmf-dist
-
+# Called by app-text/texlive-core.
 texlive-common_is_file_present_in_texmf() {
local mark="${T}/${1}.found"
if [[ -d texmf ]]; then
-   find texmf -name ${1} -exec touch ${mark} {} + || die
+   find texmf -name "${1}" -exec touch "${mark}" {} + || die
fi
 
 

[gentoo-dev] [PATCH 1/2] profiles/thirdpartymirrors: add 'ctan' mirror

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 profiles/thirdpartymirrors | 1 +
 1 file changed, 1 insertion(+)

diff --git a/profiles/thirdpartymirrors b/profiles/thirdpartymirrors
index b890610255f5..1db91760893b 100644
--- a/profiles/thirdpartymirrors
+++ b/profiles/thirdpartymirrors
@@ -1,5 +1,6 @@
 apache https://dlcdn.apache.org/ https://apache.mirror.iphh.net/ 
https://artfiles.org/apache.org/ 
https://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/ 
https://ftp.fau.de/apache/ https://apache.osuosl.org/
 cpan   https://cpan.metacpan.org https://www.cpan.org
+ctan   https://mirrors.ctan.org/ https://ftp.fau.de/ctan/ 
https://mirror.physik.tu-berlin.de/pub/CTAN/ https://ftp.sun.ac.za/ftp/CTAN/ 
https://mirror.math.princeton.edu/pub/CTAN/ 
https://mirrors.sjtug.sjtu.edu.cn/ctan/ https://mirrors.mit.edu/CTAN/ 
https://tug.ctan.org/
 debian https://deb.debian.org/debian/ http://ftp.au.debian.org/debian/ 
http://ftp.at.debian.org/debian/ http://ftp.by.debian.org/debian/ 
http://ftp.be.debian.org/debian/ http://ftp.br.debian.org/debian/ 
http://ftp.bg.debian.org/debian/ http://ftp.ca.debian.org/debian/ 
http://ftp2.cn.debian.org/debian/ http://ftp.cn.debian.org/debian/ 
http://ftp.hr.debian.org/debian/ http://ftp.cz.debian.org/debian/ 
http://ftp.dk.debian.org/debian/ http://ftp.sv.debian.org/debian/ 
http://ftp.ee.debian.org/debian/ http://ftp.fi.debian.org/debian/ 
http://ftp.fr.debian.org/debian/ http://ftp2.de.debian.org/debian/ 
http://ftp.de.debian.org/debian/ http://ftp.gr.debian.org/debian/ 
http://ftp.hu.debian.org/debian/ http://ftp.is.debian.org/debian/ 
http://ftp.ie.debian.org/debian/ http://ftp.it.debian.org/debian/ 
http://ftp.jp.debian.org/debian/ http://ftp.lt.debian.org/debian/ 
http://ftp.mx.debian.org/debian/ http://ftp.md.debian.org/debian/ 
http://ftp.nl.debian.org/debian/ http://ftp.nc.debian.org/debian/ 
http://ftp.nz.debian.org/debian/ http://ftp.no.debian.org/debian/ 
http://ftp.pl.debian.org/debian/ http://ftp.pt.debian.org/debian/ 
http://ftp.ro.debian.org/debian/ http://ftp.ru.debian.org/debian/ 
http://ftp.sg.debian.org/debian/ http://ftp.sk.debian.org/debian/ 
http://ftp.si.debian.org/debian/ http://ftp.es.debian.org/debian/ 
http://ftp.se.debian.org/debian/ http://ftp.ch.debian.org/debian/ 
http://ftp.tw.debian.org/debian/ http://ftp.tr.debian.org/debian/ 
http://ftp.ua.debian.org/debian/ http://ftp.uk.debian.org/debian/ 
http://ftp.us.debian.org/debian/
 gentoo https://distfiles.gentoo.org/distfiles 
https://gentoo.osuosl.org/distfiles 
https://ftp.halifax.rwth-aachen.de/gentoo/distfiles 
https://ftp.fau.de/gentoo/distfiles
 gcchttps://gcc.gnu.org/pub/gcc/ 
http://mirrors.concertpass.com/gcc/ 
https://mirrorservice.org/sites/sourceware.org/pub/gcc/ 
https://ftp.mpi-inf.mpg.de/mirrors/gnu/mirror/gcc.gnu.org/pub/gcc/ 
https://bigsearcher.com/mirrors/gcc/
-- 
2.43.0

[gentoo-dev] [PATCH 0/2] texlive eclass updates, in preperation of TeX Live 2023 bump

[Florian Schmaus]

After negril and I worked for the past months on the TeX Live 2023
bump, it is now time to submit the eclass changes to the mailing list
for review.

Note that we tried to keep the changes to the eclasses to a minimum.
There are many more places where the code of the eclasses could be
improved. But bumping TeX Live is already a delicate matter and we do
not want to add more distraction from the actual changes to keep the
changes manageable and reviewable.

Once TeX Live 2023 hits ::gentoo, we will iteratively work on revising
the code of the eclasses.

Florian Schmaus (2):
  profiles/thirdpartymirrors: add 'ctan' mirror
  texlive-{common,module}.eclass: update for TeX Live 2023

 eclass/texlive-common.eclass | 113 +--
 eclass/texlive-module.eclass |  77 
 profiles/thirdpartymirrors   |   1 +
 3 files changed, 109 insertions(+), 82 deletions(-)

-- 
2.43.0

Re: [gentoo-dev] [PATCH 0/1] [RFC] greadme.eclass

[Florian Schmaus]

On 10/01/2024 16.10, Ulrich Mueller wrote:

On Wed, 10 Jan 2024, Florian Schmaus wrote:



On 10/01/2024 14.58, Ulrich Mueller wrote:

Looks like readme.gentoo-r1 already gives you control over this:
# If you want to show them always, please set FORCE_PRINT_ELOG to a non empty
# value in your ebuild before this function is called.
# This can be useful when, for example, DOC_CONTENTS is modified, then, you can
# rely on specific REPLACING_VERSIONS handling in your ebuild to print messages
# when people update from versions still providing old message.



It is easy to forget setting FORCE_PRINT_ELOG, just as it is easy to
forget to unset it again.



An automatism is always preferable over a manual solution.


Maybe I want manual control? For example, when I fix a typo in the
README file then I don't want to show it to users again.


An automatism does not exclude an manual override. That can easily be 
added to greadme.eclass.




There seems to be a big win-win if we override the compression
settingin this case.


I tend to disagree. We shouldn't override users' choices unless
absolutely necessary.


Just that there is no misunderstanding: you are aware that it is a file 
of typically just a few lines of text that we are discussing here 
whether or not it should be compressed?


Seems like overshooting the mark to argue with users' choices in this case.

- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 0/1] [RFC] greadme.eclass

[Florian Schmaus]

On 10/01/2024 14.58, Ulrich Mueller wrote:

On Wed, 10 Jan 2024, Florian Schmaus wrote:



On 10/01/2024 12.04, Sam James wrote:

1) The name seems odd (why not readme.gentoo-r2)?
2) Why can't the existing eclass be improved?



Both points, the name of the eclass and the question if this should be
added to the existing eclass or as a new eclass, are absolutely *no*
hill I want to die on.



What I *really* care about is having the functionality that there is a
readme eclass that *also* shows the elog message if the README's
content changed (and not just on the first installation of the
package).


Looks like readme.gentoo-r1 already gives you control over this:

# If you want to show them always, please set FORCE_PRINT_ELOG to a non empty
# value in your ebuild before this function is called.
# This can be useful when, for example, DOC_CONTENTS is modified, then, you can
# rely on specific REPLACING_VERSIONS handling in your ebuild to print messages
# when people update from versions still providing old message.


It is easy to forget setting FORCE_PRINT_ELOG, just as it is easy to 
forget to unset it again.


An automatism is always preferable over a manual solution.



4) The compression deal seems not worth bothering with.



Just to clarify: you are agreeing that excluding the readme doc from
being compressed is fine?


Please respect the user's compression settings there. IMHO overriding
them with docompress -x is a big no-no.


Then why does "docompress -x" exist at all?

There seems to be a big win-win if we override the compression settingin 
this case.




It exports phase functions, which readme.gentoo-r1 does not.


Looking at the history, readme.gentoo[-r0] used to export phase
functions:
https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/readme.gentoo.eclass?id=1e7b2242de29ec60105df1ef31939aed85a8b0eb#n32
It turned out to be a bad design choice, so -r1 no longer does that.


Interesting find.

It is not obvious to me why the eclass exporting phase function should 
is a bad design choice.


@pacho: could you shed some light into this?



The readme.gentoo-r1 eclass always shoves the full content of the
readme into an environment variable.


Why is this a problem?


Nobody described that as a problem. Not adding stuff into the 
environment is simply nice to have.


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 0/1] [RFC] greadme.eclass

[Florian Schmaus]

On 10/01/2024 12.04, Sam James wrote:


Florian Schmaus  writes:


I really like the functionality of readme.gentoo-r1.eclass, as it
allows to communicate Gentoo-specific information about a package to
the user. Especially as it improves the signal-to-noise ratio of
messages arriving to our users.

However, readme.gentoo-r1.eclass will only show this information on
new installs, but not if the information changed. This is a major
drawback. Furthermore, readme.gentoo-r1.eclass does not provide an API
to assemble the information via heredoc.

The following patch includes a new eclass named "greadme", that
addresses those shortcomings of readme.gentoo-r1.eclass and hopefully
can be seen as a general overhaul.


I have a few concerns at the moment, but of varying severity:



1) The name seems odd (why not readme.gentoo-r2)? > 2) Why can't the existing 
eclass be improved?


Both points, the name of the eclass and the question if this should be 
added to the existing eclass or as a new eclass, are absolutely *no* 
hill I want to die on.


What I *really* care about is having the functionality that there is a 
readme eclass that *also* shows the elog message if the README's content 
changed (and not just on the first installation of the package).



> 4) The compression deal seems not worth bothering with.

Just to clarify: you are agreeing that excluding the readme doc from 
being compressed is fine?




3) Is the reason for 2) strong enough to introduce a new eclass?
I don't really want to see a bifurcation in our README eclasses
if we can help it. Porting to something new takes ages and it's
a lot of churn.


I think the arguments for introducing a new eclass are strong enough. I 
will elaborate on this in the appendix of this mail, since I don't think 
its what we should focus on at the moment.


However, right now I want clarify that using "docompress -x" in this 
case is agreeable by everyone.


- Flow


# Appendix: Arguments for introducing a new class

readme.gentoo-r1 can be viewed at

https://github.com/gentoo/gentoo/blob/master/eclass/readme.gentoo-r1.eclass

while the simple and short version (203 lines including comments!) of 
the proposed greadme.eclass can be seen at


https://github.com/Flowdalic/gentoo/blob/greadme.eclass-simple/eclass/greadme.eclass

The proposed API of the greadme eclass is already different from 
readme.gentoo-r1. It exports phase functions, which readme.gentoo-r1 
does not. The readme.gentoo-r1 eclass always shoves the full content of 
the readme into an environment variable. Excluding the readme file from 
compression means we do not have to do that in greadme.


Please feel invited take a look at the code of both eclasses and make 
yourself an opinion if it is sensible to incorporate greadme.eclass into 
the existing eclass.


Then maybe also also look a the API of both eclasses. Switching from 
readme.gentoo-r1 to greadme is simple and straightforward in most cases.


Furthermore, the name readme.gentoo-r1 seems odd, it is one of the few 
eclasses that use a dot as separator (although, I can see why this may 
been done). Looking at eclasses/ it appears that readme-gentoo-r1.eclass 
would be more in-line with the existing eclasses. Or, as a matter of 
opinion, maybe better, because shorter: greadme


Adding a new readme eclass hopefully is not a bifurcation, cause we 
ideally would be able to phase out readme.gentoo-r1.






OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [RFC] New category: dev-debug

[Florian Schmaus]

On 10/01/2024 12.01, Sam James wrote:


Florian Schmaus  writes:


[[PGP Signed Part:Undecided]]
On 07/01/2024 13.48, Michał Górny wrote:

Hi,
I'd like to propose adding a new dev-debug category.  The
description
could be:
Code debuggers and debugging-related tools.


Seems sensible. I'd probably drop the "Code" from the description,
since packages like d-spy do not primarily debug code.

I assume profilers are explicitly not part of this category?


leio had the same question and we came up with a list of about 5-6
packages at least which would fit in a dev-profile category...


Right and a sensible next step would probably be an extra category for 
those.


The problem is that that the lines between profilers, tracers, and 
performance counter tools sometimes gets blurry. But I think it could be 
fine to shove those all into one category, but maybe dev-perf then 
(e.g., "Tools for performance analysis")?


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [RFC] New category: dev-debug

[Florian Schmaus]

On 07/01/2024 13.48, Michał Górny wrote:

Hi,

I'd like to propose adding a new dev-debug category.  The description
could be:

   Code debuggers and debugging-related tools.


Seems sensible. I'd probably drop the "Code" from the description, since 
packages like d-spy do not primarily debug code.


I assume profilers are explicitly not part of this category?

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH v2 1/3] greadme.eclass: new eclass

[Florian Schmaus]

On 09/01/2024 12.23, David Seifert wrote:

On Tue, 2024-01-09 at 09:39 +0100, Florian Schmaus wrote:

This new eclass is similar to readme.gentoo-r1.eclass. The main
differences are as follows. Firstly, it also displays the doc file
contents if they have changed. Secondly, it provides a convenient API
to
install the doc file via stdin.

Furthermore, this eclass dos not store the doc file's contents in an
environment variable, which helps to keep the environment size of
ebuilds using the eclass small.

Signed-off-by: Florian Schmaus 
---
  eclass/greadme.eclass | 307
++
  1 file changed, 307 insertions(+)
  create mode 100644 eclass/greadme.eclass

diff --git a/eclass/greadme.eclass b/eclass/greadme.eclass
new file mode 100644
index ..25e0210406c1
--- /dev/null
+++ b/eclass/greadme.eclass
@@ -0,0 +1,307 @@
+# Copyright 1999-2024 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: greadme.eclass
+# @MAINTAINER:
+# Florian Schmaus 
+# @AUTHOR:
+# Author: Florian Schmaus 
+# @SUPPORTED_EAPIS: 6 7 8
+# @BLURB: install a doc file, that will be conditionally shown via
elog messages
+# @DESCRIPTION:
+# An eclass for installing a README.gentoo doc file recording tips
+# shown via elog messages.  With this eclass, those elog messages
will only be
+# shown at first package installation or if the contents of the file
have changed.
+# Furthermore, a file for later reviewing will be installed under
+# /usr/share/doc/${PF}
+#
+# This eclass is similar to readme.gentoo-r1.eclass.  The main
+# differences are as follows.  Firstly, it also displays the doc file
+# contents if they have changed.  Secondly, it provides a convenient
API to
+# install the doc file via stdin.
+#
+# @CODE
+# inherit greadme
+#
+# src_install() {
+#   …
+#   greadme_stdin <<- EOF
+#   This is the content of the created readme doc file.
+#   EOF
+#   …
+#   if use foo; then
+# greadme_stdin --apend <<-EOF
+# This is conditional readme content, based on USE=foo.
+# EOF
+#   fi
+# }
+# @CODE
+#
+# You must call greadme_pkg_preinst and greadme_pkg_postinst
explicitly, if
+# you override the default pkg_preinst or respectively pkg_postinst.
+#
+# TODO:
+# - Should this be named README.Distribution instead of
README.Gentoo?
+#   Would that make things easier for Gentoo derivates?
+#   Similary, (g → d)readme, (G → D)README?
+# - Incooperate changes into readme.gentoo-r1.elcass?
+# - Compressing the readme doc file is probably fragile, as it is not
+#   guaranteed that the required binaries for decompression are
installed
+#   in pkg_preinst/pkg_postinst. Note that it is even possible that
two
+#   different compression algorithms are used, in case of binpkgs.
+
+if [[ -z ${_README_GENTOO_ECLASS} ]]; then
+_README_GENTOO_ECLASS=1
+
+case ${EAPI} in
+   6|7|8) ;;
+   *) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
+esac
+
+if [[ ${_GREADME_COMPRESS} ]]; then
+   inherit unpacker
+fi
+
+_GREADME_FILENAME="README.gentoo"
+_GREADME_TMP_FILE="${T}/${_GREADME_FILENAME}"
+_GREADME_REL_PATH="usr/share/doc/${PF}/${_GREADME_FILENAME}"
+
+# @FUNCTION: greadme_stdin
+# @USAGE: [--append]
+# @DESCRIPTION:
+# Create the readme doc via stdin.  You can use --append to append to
an
+# existing readme doc.
+greadme_stdin() {
+   debug-print-function ${FUNCNAME} "${@}"
+
+   local append=false
+   while [[ -n ${1} ]] && [[ ${1} =~ --* ]]; do
+   case ${1} in
+   --append)
+   append=true
+   shift
+   ;;
+   esac
+   done
+
+   if $append; then
+   if [[ ! -f "${_GREADME_TMP_FILE}" ]]; then
+   die "Gentoo README does not exist when trying
to append to it"
+   fi
+
+   cat >> "${_GREADME_TMP_FILE}" || die
+   else
+   if [[ -f "${_GREADME_TMP_FILE}" ]]; then
+   die "Gentoo README already exists while
trying to create it"
+   fi
+
+   cat > "${_GREADME_TMP_FILE}" || die
+   fi
+
+   _greadme_install_doc
+}
+
+# @FUNCTION: greadme_file
+# @USAGE: 
+# @DESCRIPTION:
+# Installs the provided file as readme doc.
+greadme_file() {
+   debug-print-function ${FUNCNAME} "${@}"
+
+   local input_doc_file="${1}"
+   if [[ -z "${input_doc_file}" ]]; then
+   die "No file specified"
+   fi
+
+   if [[ -f "${_GREADME_TMP_FILE}" ]]; then
+   die "Gentoo README already exists"
+   fi
+
+   cp "${input_doc_file}" "${_GREADME_TMP_FILE}" || die
+
+   _greadme_install_doc
+}
+
+# @FUNCTION: _greadme_instal

Re: [gentoo-dev] [PATCH 0/1] [RFC] greadme.eclass

[Florian Schmaus]

On 09/01/2024 11.43, Michał Górny wrote:

On Tue, 2024-01-09 at 11:39 +0100, Florian Schmaus wrote:

Even if we say it is the user's fault, then the problem of handling a
decompressor failure would still exist. The eclass does not gracefully
continue when decompressing the doc file, but instead it runs into a
die(). To address this we would need to make unpacker.eclass and
unpack() aware of nonfatal. The latter would require a PMS change.

Or, I could duplicate unpack logic --- which is probably a lot to
account for the various compression options --- in the eclass. But I
suspect be both do not want that.


Perhaps this is the moment when you realize that there is no guarantee
that unpacker.eclass and/or unpack will support the compression format
set for docompress.  The two were never intended to interact.


Hence I wrote about making then nonfatal aware. Then greadme.eclass 
could try to opportunistically decompress the doc file, and simply 
continue with a sensible behavior it it fails.


A look at the greadme code reveals that it tries to do so already, but 
unfortunately unsuccessfully because nonfatal has no effect on 
unpacker/unpack.


- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 0/1] [RFC] greadme.eclass

[Florian Schmaus]

On 09/01/2024 10.59, Michał Górny wrote:

On Tue, 2024-01-09 at 09:30 +0100, Florian Schmaus wrote:

On 06/01/2024 18.21, Michał Górny wrote:

On Sat, 2024-01-06 at 18:01 +0100, Florian Schmaus wrote:

I really like the functionality of readme.gentoo-r1.eclass, as it
allows to communicate Gentoo-specific information about a package to
the user. Especially as it improves the signal-to-noise ratio of
messages arriving to our users.

However, readme.gentoo-r1.eclass will only show this information on
new installs, but not if the information changed. This is a major
drawback. Furthermore, readme.gentoo-r1.eclass does not provide an API
to assemble the information via heredoc.


Are you implying that readme.gentoo-r1 is unfixable and you need to
start over, and have a third generation of eclasses to install a readme
file?


Not at all. See the second item of the TODO list in the eclass' description.

That said, both eclasses are rather small,



Since when, 300 lines to install a README file is "small"?


The eclass becomes very small if you remove the _GREADME_COMPRESS code. 
As I wrote, adding compression support to the eclass makes the code very 
complex.




The main item is doc compression. Right now, greadme.eclass defaults
to add the readme doc to the compression exclusion list via
"docompress -x". A mode where the readme doc is compressed, just as
readme.gentoo-r1.eclass does, can be activated by setting
_GREADME_COMPRESS. However, I believe this mode is fragile as it can
not be guaranteed that a binary for the used compression algorithms is
installed on the host [1].


Dangling reference here.  In any case, documentation compression is
a standard feature of the package manager.  If it doesn't work for
whatever reason, I'd rather see you focus on find a good solution rather
than working it around via abusing `docompress -x`.


The problem is the lack of a guarantee that the utilities required to
decompress the file are available at installation time.


It's user's responsibility to ensure that the tools set in their
make.conf are available.


What if the user obtained a binpkg that was compressed with a different 
algorithm than the one set in their make.conf? Of course, the binhost 
could have set FEATURES=-binpkg-docompress, but what if not?


Even if we say it is the user's fault, then the problem of handling a 
decompressor failure would still exist. The eclass does not gracefully 
continue when decompressing the doc file, but instead it runs into a 
die(). To address this we would need to make unpacker.eclass and 
unpack() aware of nonfatal. The latter would require a PMS change.


Or, I could duplicate unpack logic --- which is probably a lot to 
account for the various compression options --- in the eclass. But I 
suspect be both do not want that.




It gets even worse if you consider binpkgs, as you have surely read the
comment while looking at the code of the greadme.eclass.


See FEATURES=-binpkg-docompress.  That's the correct way to distribute
binary packages.


Is that documented somewhere that this is the right way to distribute 
binary packages?



With the information I have right now, I do not see a better alternative 
than excluding the doc file from compression.


Are you willing a sacrifice a very sensible feature just to be able to 
compress what is usually a file of a few hundred of bytes?




- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: [PATCH v2 2/3] add UNPACKER_NO_BANNER variable

[Florian Schmaus]

On 09/01/2024 09.39, Florian Schmaus wrote:

Signed-off-by: Florian Schmaus 
---
  eclass/-cover-letter.patch  |  49 
  eclass/0001-greadme.eclass-new-eclass.patch | 305 


Ignore those to patch files. They are accidentally added to the commit.



diff --git a/eclass/unpacker.eclass b/eclass/unpacker.eclass
index 2957ca02d3f4..73b763ca2018 100644
--- a/eclass/unpacker.eclass
+++ b/eclass/unpacker.eclass
@@ -42,6 +42,11 @@ inherit multiprocessing toolchain-funcs
  # `xz`, `plzip`, `pdlzip`, and `lzip`.  Make sure your choice accepts the 
"-dc" options.
  # Note: this is meant for users to set, not ebuilds.
  
+# @ECLASS_VARIABLE: UNPACKER_NO_BANNER

+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# If set, do not display the unpack banner with information what got unpacked 
where.
+
  # for internal use only (unpack_pdv and unpack_makeself)
  find_unpackable_file() {
local src=$1
@@ -63,6 +68,8 @@ find_unpackable_file() {
  }
  
  unpack_banner() {

+   [[ ${UNPACKER_NO_BANNER} ]] && return
+
echo ">>> Unpacking ${1##*/} to ${PWD}"
  }

Also note that this only prevents the banner emitted by the 
unpacker.eclass. The eclass also invokes unpack() from PMS/portage, 
emitting a banner which can not be disabled via the eclass.


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH v2 3/3] greadme.eclass: set UNPACKER_NO_BANNER

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/greadme.eclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/eclass/greadme.eclass b/eclass/greadme.eclass
index 25e0210406c1..ec01d090cc10 100644
--- a/eclass/greadme.eclass
+++ b/eclass/greadme.eclass
@@ -174,7 +174,7 @@ greadme_pkg_preinst() {
if [[ "${image_doc_file_basename}" == "${_GREADME_FILENAME}" 
]]; then
cp "${image_doc_file}" . || die
else
-   nonfatal unpacker "${image_doc_file}"
+   UNPACKER_NO_BANNER=1 nonfatal unpacker 
"${image_doc_file}"
if [[ $? -gt 0 ]]; then
# We failed to unpack the readme doc from the
# image, therefore, we can't show it (unless we
@@ -222,7 +222,7 @@ greadme_pkg_preinst() {
if [[ "${live_doc_file_basename}" == 
"${_GREADME_FILENAME}" ]]; then
cp "${live_doc_file}" .
else
-   nonfatal unpacker "${live_doc_file}"
+   UNPACKER_NO_BANNER=1 nonfatal unpacker 
"${live_doc_file}"
if [[ $? -gt 0 ]]; then
# We failed to unpack the live readme 
doc, fallback
# to show the new readme contents.
-- 
2.41.0

[gentoo-dev] [PATCH v2 2/3] add UNPACKER_NO_BANNER variable

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/-cover-letter.patch  |  49 
 eclass/0001-greadme.eclass-new-eclass.patch | 305 
 eclass/unpacker.eclass  |   7 +
 3 files changed, 361 insertions(+)
 create mode 100644 eclass/-cover-letter.patch
 create mode 100644 eclass/0001-greadme.eclass-new-eclass.patch

diff --git a/eclass/-cover-letter.patch b/eclass/-cover-letter.patch
new file mode 100644
index ..2e162d419a44
--- /dev/null
+++ b/eclass/-cover-letter.patch
@@ -0,0 +1,49 @@
+From edff06160e33b361c9d6a7e273fc7808337c4518 Mon Sep 17 00:00:00 2001
+From: Florian Schmaus 
+Date: Sat, 6 Jan 2024 17:44:44 +0100
+Subject: [PATCH 0/1] [RFC] greadme.eclass
+
+I really like the functionality of readme.gentoo-r1.eclass, as it
+allows to communicate Gentoo-specific information about a package to
+the user. Especially as it improves the signal-to-noise ratio of
+messages arriving to our users.
+
+However, readme.gentoo-r1.eclass will only show this information on
+new installs, but not if the information changed. This is a major
+drawback. Furthermore, readme.gentoo-r1.eclass does not provide an API
+to assemble the information via heredoc.
+
+The following patch includes a new eclass named "greadme", that
+addresses those shortcomings of readme.gentoo-r1.eclass and hopefully
+can be seen as a general overhaul.
+
+This is a first draft of the eclass and therefore I'd like to ask for
+feedback.
+
+The greadme.eclass contains some TODO items at the end of its
+@DESCRIPTION.
+
+The main item is doc compression. Right now, greadme.eclass defaults
+to add the readme doc to the compression exclusion list via
+"docompress -x". A mode where the readme doc is compressed, just as
+readme.gentoo-r1.eclass does, can be activated by setting
+_GREADME_COMPRESS. However, I believe this mode is fragile as it can
+not be guaranteed that a binary for the used compression algorithms is
+installed on the host [1].
+
+I believe it is reasonable to simply install the readme doc
+uncompressed, since they are typically only a few lines long. However,
+if anyone can point out a way to achieve the desired functionality with
+a compressed readme doc, then please let me know.
+
+Thanks for reviewing the eclass.
+
+Florian Schmaus (1):
+  greadme.eclass: new eclass
+
+ eclass/greadme.eclass | 281 ++
+ 1 file changed, 281 insertions(+)
+ create mode 100644 eclass/greadme.eclass
+
+-- 
+2.41.0
diff --git a/eclass/0001-greadme.eclass-new-eclass.patch 
b/eclass/0001-greadme.eclass-new-eclass.patch
new file mode 100644
index ..52f4d9b6ff4d
--- /dev/null
+++ b/eclass/0001-greadme.eclass-new-eclass.patch
@@ -0,0 +1,305 @@
+From edff06160e33b361c9d6a7e273fc7808337c4518 Mon Sep 17 00:00:00 2001
+From: Florian Schmaus 
+Date: Sat, 6 Jan 2024 17:39:45 +0100
+Subject: [PATCH 1/1] greadme.eclass: new eclass
+
+This new eclass is similar to readme.gentoo-r1.eclass. The main
+differences are as follows. Firstly, it also displays the doc file
+contents if they have changed. Secondly, it provides a convenient API to
+install the doc file via stdin.
+
+Signed-off-by: Florian Schmaus 
+---
+ eclass/greadme.eclass | 281 ++
+ 1 file changed, 281 insertions(+)
+ create mode 100644 eclass/greadme.eclass
+
+diff --git a/eclass/greadme.eclass b/eclass/greadme.eclass
+new file mode 100644
+index ..ac83c36983ef
+--- /dev/null
 b/eclass/greadme.eclass
+@@ -0,0 +1,281 @@
++# Copyright 1999-2024 Gentoo Authors
++# Distributed under the terms of the GNU General Public License v2
++
++# @ECLASS: greadme.eclass
++# @MAINTAINER:
++# Florian Schmaus 
++# @AUTHOR:
++# Author: Florian Schmaus 
++# @SUPPORTED_EAPIS: 6 7 8
++# @BLURB: install a doc file, that will be conditionally shown via elog 
messages
++# @DESCRIPTION:
++# An eclass for installing a README.gentoo doc file recording tips
++# shown via elog messages.  With this eclass, those elog messages will only be
++# shown at first package installation or if the contents of the file have 
changed.
++# Furthermore, a file for later reviewing will be installed under
++# /usr/share/doc/${PF}
++#
++# This eclass is similar to readme.gentoo-r1.eclass.  The main
++# differences are as follows.  Firstly, it also displays the doc file
++# contents if they have changed.  Secondly, it provides a convenient API to
++# install the doc file via stdin.
++#
++# @CODE
++# inherit greadme
++#
++# src_install() {
++#   …
++#   greadme_stdin <<- EOF
++#   This is the content of the created readme doc file.
++#   EOF
++#   …
++#   if use foo; then
++# greadme_stdin --apend <<-EOF
++# This is conditional readme content, based on USE=foo.
++# EOF
++#   fi
++# }
++# @CODE
++#
++# You must call greadme_pkg_preinst and greadme_pkg_postinst explicitly, if
++# you override the default pkg_preinst or respect

[gentoo-dev] [PATCH v2 1/3] greadme.eclass: new eclass

[Florian Schmaus]

This new eclass is similar to readme.gentoo-r1.eclass. The main
differences are as follows. Firstly, it also displays the doc file
contents if they have changed. Secondly, it provides a convenient API to
install the doc file via stdin.

Furthermore, this eclass dos not store the doc file's contents in an
environment variable, which helps to keep the environment size of
ebuilds using the eclass small.

Signed-off-by: Florian Schmaus 
---
 eclass/greadme.eclass | 307 ++
 1 file changed, 307 insertions(+)
 create mode 100644 eclass/greadme.eclass

diff --git a/eclass/greadme.eclass b/eclass/greadme.eclass
new file mode 100644
index ..25e0210406c1
--- /dev/null
+++ b/eclass/greadme.eclass
@@ -0,0 +1,307 @@
+# Copyright 1999-2024 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: greadme.eclass
+# @MAINTAINER:
+# Florian Schmaus 
+# @AUTHOR:
+# Author: Florian Schmaus 
+# @SUPPORTED_EAPIS: 6 7 8
+# @BLURB: install a doc file, that will be conditionally shown via elog 
messages
+# @DESCRIPTION:
+# An eclass for installing a README.gentoo doc file recording tips
+# shown via elog messages.  With this eclass, those elog messages will only be
+# shown at first package installation or if the contents of the file have 
changed.
+# Furthermore, a file for later reviewing will be installed under
+# /usr/share/doc/${PF}
+#
+# This eclass is similar to readme.gentoo-r1.eclass.  The main
+# differences are as follows.  Firstly, it also displays the doc file
+# contents if they have changed.  Secondly, it provides a convenient API to
+# install the doc file via stdin.
+#
+# @CODE
+# inherit greadme
+#
+# src_install() {
+#   …
+#   greadme_stdin <<- EOF
+#   This is the content of the created readme doc file.
+#   EOF
+#   …
+#   if use foo; then
+# greadme_stdin --apend <<-EOF
+# This is conditional readme content, based on USE=foo.
+# EOF
+#   fi
+# }
+# @CODE
+#
+# You must call greadme_pkg_preinst and greadme_pkg_postinst explicitly, if
+# you override the default pkg_preinst or respectively pkg_postinst.
+#
+# TODO:
+# - Should this be named README.Distribution instead of README.Gentoo?
+#   Would that make things easier for Gentoo derivates?
+#   Similary, (g → d)readme, (G → D)README?
+# - Incooperate changes into readme.gentoo-r1.elcass?
+# - Compressing the readme doc file is probably fragile, as it is not
+#   guaranteed that the required binaries for decompression are installed
+#   in pkg_preinst/pkg_postinst. Note that it is even possible that two
+#   different compression algorithms are used, in case of binpkgs.
+
+if [[ -z ${_README_GENTOO_ECLASS} ]]; then
+_README_GENTOO_ECLASS=1
+
+case ${EAPI} in
+   6|7|8) ;;
+   *) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
+esac
+
+if [[ ${_GREADME_COMPRESS} ]]; then
+   inherit unpacker
+fi
+
+_GREADME_FILENAME="README.gentoo"
+_GREADME_TMP_FILE="${T}/${_GREADME_FILENAME}"
+_GREADME_REL_PATH="usr/share/doc/${PF}/${_GREADME_FILENAME}"
+
+# @FUNCTION: greadme_stdin
+# @USAGE: [--append]
+# @DESCRIPTION:
+# Create the readme doc via stdin.  You can use --append to append to an
+# existing readme doc.
+greadme_stdin() {
+   debug-print-function ${FUNCNAME} "${@}"
+
+   local append=false
+   while [[ -n ${1} ]] && [[ ${1} =~ --* ]]; do
+   case ${1} in
+   --append)
+   append=true
+   shift
+   ;;
+   esac
+   done
+
+   if $append; then
+   if [[ ! -f "${_GREADME_TMP_FILE}" ]]; then
+   die "Gentoo README does not exist when trying to append 
to it"
+   fi
+
+   cat >> "${_GREADME_TMP_FILE}" || die
+   else
+   if [[ -f "${_GREADME_TMP_FILE}" ]]; then
+   die "Gentoo README already exists while trying to 
create it"
+   fi
+
+   cat > "${_GREADME_TMP_FILE}" || die
+   fi
+
+   _greadme_install_doc
+}
+
+# @FUNCTION: greadme_file
+# @USAGE: 
+# @DESCRIPTION:
+# Installs the provided file as readme doc.
+greadme_file() {
+   debug-print-function ${FUNCNAME} "${@}"
+
+   local input_doc_file="${1}"
+   if [[ -z "${input_doc_file}" ]]; then
+   die "No file specified"
+   fi
+
+   if [[ -f "${_GREADME_TMP_FILE}" ]]; then
+   die "Gentoo README already exists"
+   fi
+
+   cp "${input_doc_file}" "${_GREADME_TMP_FILE}" || die
+
+   _greadme_install_doc
+}
+
+# @FUNCTION: _greadme_install_doc
+# @INTERNAL
+# @DESCRIPTION:
+# Installs the readme file from the temp directory into the image.
+_gread

Re: [gentoo-dev] [PATCH 0/1] [RFC] greadme.eclass

[Florian Schmaus]

On 06/01/2024 18.21, Michał Górny wrote:

On Sat, 2024-01-06 at 18:01 +0100, Florian Schmaus wrote:

I really like the functionality of readme.gentoo-r1.eclass, as it
allows to communicate Gentoo-specific information about a package to
the user. Especially as it improves the signal-to-noise ratio of
messages arriving to our users.

However, readme.gentoo-r1.eclass will only show this information on
new installs, but not if the information changed. This is a major
drawback. Furthermore, readme.gentoo-r1.eclass does not provide an API
to assemble the information via heredoc.


Are you implying that readme.gentoo-r1 is unfixable and you need to
start over, and have a third generation of eclasses to install a readme
file?


Not at all. See the second item of the TODO list in the eclass' description.

That said, both eclasses are rather small, and "fixing" readme.gentoo-r1 
would consist of copying most of greadme into readme.gentoo-r1, while 
still having the "legacy" readme.gentoo-r1 functionality in it.


Starting over strikes me as sensible in this case.



The main item is doc compression. Right now, greadme.eclass defaults
to add the readme doc to the compression exclusion list via
"docompress -x". A mode where the readme doc is compressed, just as
readme.gentoo-r1.eclass does, can be activated by setting
_GREADME_COMPRESS. However, I believe this mode is fragile as it can
not be guaranteed that a binary for the used compression algorithms is
installed on the host [1].


Dangling reference here.  In any case, documentation compression is
a standard feature of the package manager.  If it doesn't work for
whatever reason, I'd rather see you focus on find a good solution rather
than working it around via abusing `docompress -x`.  


The problem is the lack of a guarantee that the utilities required to 
decompress the file are available at installation time.


It gets even worse if you consider binpkgs, as you have surely read the 
comment while looking at the code of the greadme.eclass.



> It's basically
> a case of "standard feature X doesn't work for me sometimes, so I now
> randomly disable X via my eclass, and hope nobody notices".

The primary motivation for posting this RFC was to find a solution and 
avoid the "docompress -x" approach. However, I don't think there is one 
that does not require adjusting PMS to provide the guarantee that the 
eclass can decompress the compressed doc file.


Adjusting PMS seems overkill just for avoiding to exclude a single small 
file from compression.


I also tried to make the greadme eclass handle unpacking errors 
gracefully. This turned out to be harder than I hoped because most 
(all?) functions of unpacker.eclass are not nonfatal compatible. The 
same is true for unpack() from PMS, which is used by unpacker.eclass.


I'll send out v2 of greadme.eclass soon.

Looking at the conditional _GREADME_COMPRESS code, you will find that 
adding support for compressing the doc file introduces a lot of 
complexity to the eclass. That would still make me consider it, but I 
can not see any reliable approach to unpacking that does not involve 
adjusting PMS.


Surely you would be able to correct me if I am wrong.



I believe it is reasonable to simply install the readme doc
uncompressed, since they are typically only a few lines long. However,
if anyone can point out a way to achieve the desired functionality with
a compressed readme doc, then please let me know.


The compression mechanism automatically detects when the file is too
small to be worth compressing.  See PORTAGE_DOCOMPRESS_SIZE_LIMIT.


How is this mechanism helpful here?

- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH 1/1] greadme.eclass: new eclass

[Florian Schmaus]

This new eclass is similar to readme.gentoo-r1.eclass. The main
differences are as follows. Firstly, it also displays the doc file
contents if they have changed. Secondly, it provides a convenient API to
install the doc file via stdin.

Signed-off-by: Florian Schmaus 
---
 eclass/greadme.eclass | 281 ++
 1 file changed, 281 insertions(+)
 create mode 100644 eclass/greadme.eclass

diff --git a/eclass/greadme.eclass b/eclass/greadme.eclass
new file mode 100644
index ..ac83c36983ef
--- /dev/null
+++ b/eclass/greadme.eclass
@@ -0,0 +1,281 @@
+# Copyright 1999-2024 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: greadme.eclass
+# @MAINTAINER:
+# Florian Schmaus 
+# @AUTHOR:
+# Author: Florian Schmaus 
+# @SUPPORTED_EAPIS: 6 7 8
+# @BLURB: install a doc file, that will be conditionally shown via elog 
messages
+# @DESCRIPTION:
+# An eclass for installing a README.gentoo doc file recording tips
+# shown via elog messages.  With this eclass, those elog messages will only be
+# shown at first package installation or if the contents of the file have 
changed.
+# Furthermore, a file for later reviewing will be installed under
+# /usr/share/doc/${PF}
+#
+# This eclass is similar to readme.gentoo-r1.eclass.  The main
+# differences are as follows.  Firstly, it also displays the doc file
+# contents if they have changed.  Secondly, it provides a convenient API to
+# install the doc file via stdin.
+#
+# @CODE
+# inherit greadme
+#
+# src_install() {
+#   …
+#   greadme_stdin <<- EOF
+#   This is the content of the created readme doc file.
+#   EOF
+#   …
+#   if use foo; then
+# greadme_stdin --apend <<-EOF
+# This is conditional readme content, based on USE=foo.
+# EOF
+#   fi
+# }
+# @CODE
+#
+# You must call greadme_pkg_preinst and greadme_pkg_postinst explicitly, if
+# you override the default pkg_preinst or respectively pkg_postinst.
+#
+# TODO:
+# - Should this be named README.Distribution instead of README.Gentoo?
+#   Would that make things easier for Gentoo derivates?
+#   Similary, (g → d)readme, (G → D)README?
+# - Incooperate changes into readme.gentoo-r1.elcass?
+# - Compressing the readme doc file is probably fragile, as it is not
+#   guaranteed that the required binaries for decompression are installed
+#   in pkg_preinst/pkg_postinst. Note that it is even possible that two
+#   different compression algorithms are used, in case of binpkgs.
+
+if [[ -z ${_README_GENTOO_ECLASS} ]]; then
+_README_GENTOO_ECLASS=1
+
+case ${EAPI} in
+   6|7|8) ;;
+   *) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
+esac
+
+if [[ ${_GREADME_COMPRESS} ]]; then
+   inherit unpacker
+fi
+
+_GREADME_FILENAME="README.gentoo"
+_GREADME_TMP_FILE="${T}/${_GREADME_FILENAME}"
+_GREADME_REL_PATH="usr/share/doc/${PF}/${_GREADME_FILENAME}"
+
+# @FUNCTION: greadme_stdin
+# @USAGE: [--append]
+# @DESCRIPTION:
+# Create the readme doc via stdin.  You can use --append to append to an
+# existing readme doc.
+greadme_stdin() {
+   debug-print-function ${FUNCNAME} "${@}"
+
+   local append=false
+   while [[ -n ${1} ]] && [[ ${1} =~ --* ]]; do
+   case ${1} in
+   --append)
+   append=true
+   shift
+   ;;
+   esac
+   done
+
+   if $append; then
+   if [[ ! -f "${_GREADME_TMP_FILE}" ]]; then
+   die "Gentoo README does not exist when trying to append 
to it"
+   fi
+
+   cat >> "${_GREADME_TMP_FILE}" || die
+   else
+   if [[ -f "${_GREADME_TMP_FILE}" ]]; then
+   die "Gentoo README already exists"
+   fi
+
+   cat > "${_GREADME_TMP_FILE}" || die
+   fi
+
+   _greadme_install_doc
+}
+
+# @FUNCTION: greadme_file
+# @USAGE: 
+# @DESCRIPTION:
+# Installs the provided file as readme doc.
+greadme_file() {
+   debug-print-function ${FUNCNAME} "${@}"
+
+   local input_doc_file="${1}"
+   if [[ -z "${input_doc_file}" ]]; then
+   die "No file specified"
+   fi
+
+   if [[ -f "${_GREADME_TMP_FILE}" ]]; then
+   die "Gentoo README already exists"
+   fi
+
+   cp "${input_doc_file}" "${_GREADME_TMP_FILE}" || die
+
+   _greadme_install_doc
+}
+
+# @FUNCTION: _greadme_install_doc
+# @INTERNAL
+# @DESCRIPTION:
+# Installs the readme file from the temp directory into the image.
+_greadme_install_doc() {
+   debug-print-function ${FUNCNAME} "${@}"
+
+   if [[ ! -f "${_GREADME_TMP_FILE}" ]]; then
+   die "Gentoo README does not exist"
+  

[gentoo-dev] [PATCH 0/1] [RFC] greadme.eclass

[Florian Schmaus]

I really like the functionality of readme.gentoo-r1.eclass, as it
allows to communicate Gentoo-specific information about a package to
the user. Especially as it improves the signal-to-noise ratio of
messages arriving to our users.

However, readme.gentoo-r1.eclass will only show this information on
new installs, but not if the information changed. This is a major
drawback. Furthermore, readme.gentoo-r1.eclass does not provide an API
to assemble the information via heredoc.

The following patch includes a new eclass named "greadme", that
addresses those shortcomings of readme.gentoo-r1.eclass and hopefully
can be seen as a general overhaul.

This is a first draft of the eclass and therefore I'd like to ask for
feedback.

The greadme.eclass contains some TODO items at the end of its
@DESCRIPTION.

The main item is doc compression. Right now, greadme.eclass defaults
to add the readme doc to the compression exclusion list via
"docompress -x". A mode where the readme doc is compressed, just as
readme.gentoo-r1.eclass does, can be activated by setting
_GREADME_COMPRESS. However, I believe this mode is fragile as it can
not be guaranteed that a binary for the used compression algorithms is
installed on the host [1].

I believe it is reasonable to simply install the readme doc
uncompressed, since they are typically only a few lines long. However,
if anyone can point out a way to achieve the desired functionality with
a compressed readme doc, then please let me know.

Thanks for reviewing the eclass.

Florian Schmaus (1):
  greadme.eclass: new eclass

 eclass/greadme.eclass | 281 ++
 1 file changed, 281 insertions(+)
 create mode 100644 eclass/greadme.eclass

-- 
2.41.0

Re: [gentoo-dev] [RFC] global USE=gpg

[Florian Schmaus]

On 30/12/2023 16.54, Andreas K. Huettel wrote:

we have many local gpg useflags which basically just enable gpg.
Should we merge these to one global useflag?

Additionally we have a few gpgme useflags.
See also https://bugs.gentoo.org/679634

What are your ideas?



We have also have a bunch of USE=pgp and USE=openpgp, both of which are
more correct than USE=gpg.


I am always confused when people use "gpg" to talk about OpenPGP.



Yeah, typical case of "formally correct thing being way more difficult to
understand than colloquially practical thing" ...


It is only a matter of time until the more users of gnupg-alternative 
libraries, like sequoia or librnp, appear. USE=gpg is probably already 
sometimes a misnomer, and will definitely be one if we make it a global 
USE flag and there are packages that declare it without pulling in gpg.


- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH] webapp.eclass: optimize webapp_serverowned() and inlining webapp_strip_*()

[Florian Schmaus]

Some ebuilds invoke webapp_serverowned with -r, causing the eclass to
iterate over a large set of files. Within the iteration body, the eclass
forks multiple times to invoke the webapp_strip.

This optimizes webapp_serverowned() by replacing the call to the "strip"
functions with an equivalent invocation of find that iterates over all
the files.

Furthermore, all remaining invocations of webapp_strip_*() are inlined.

Closes: https://bugs.gentoo.org/781860
Signed-off-by: Florian Schmaus 
---
 eclass/webapp.eclass | 40 
 1 file changed, 16 insertions(+), 24 deletions(-)

diff --git a/eclass/webapp.eclass b/eclass/webapp.eclass
index 8bd8e2aef03e..5b091c84851f 100644
--- a/eclass/webapp.eclass
+++ b/eclass/webapp.eclass
@@ -96,21 +96,6 @@ webapp_check_installedat() {
${WEBAPP_CONFIG} --show-installed -h localhost -d "${INSTALL_DIR}" 2> 
/dev/null
 }
 
-webapp_strip_appdir() {
-   debug-print-function $FUNCNAME $*
-   echo "${1#${MY_APPDIR}/}"
-}
-
-webapp_strip_d() {
-   debug-print-function $FUNCNAME $*
-   echo "${1#${D}}"
-}
-
-webapp_strip_cwd() {
-   debug-print-function $FUNCNAME $*
-   echo "${1/#.\///}"
-}
-
 webapp_getinstalltype() {
debug-print-function $FUNCNAME $*
 
@@ -195,8 +180,11 @@ webapp_configfile() {
for m in "$@"; do
webapp_checkfileexists "${m}" "${D}"
 
-   local my_file="$(webapp_strip_appdir "${m}")"
-   my_file="$(webapp_strip_cwd "${my_file}")"
+   local my_file
+   # Strip appdir
+   my_file="${m#${MY_APPDIR}/}"
+   # Strip cwd
+   my_file="${my_file/#.\///}"
 
elog "(config) ${my_file}"
echo "${my_file}" >> "${D}/${WA_CONFIGLIST}"
@@ -249,8 +237,11 @@ _webapp_serverowned() {
debug-print-function $FUNCNAME $*
 
webapp_checkfileexists "${1}" "${D}"
-   local my_file="$(webapp_strip_appdir "${1}")"
-   my_file="$(webapp_strip_cwd "${my_file}")"
+   local my_file
+   # Strip appdir
+   my_file="${1#${MY_APPDIR}/}"
+   # Strip cwd
+   my_file="${my_file/#.\///}"
 
echo "${my_file}" >> "${D}/${WA_SOLIST}"
 }
@@ -264,14 +255,15 @@ _webapp_serverowned() {
 webapp_serverowned() {
debug-print-function $FUNCNAME $*
 
-   local a m
+   local m
if [[ "${1}" == "-R" ]]; then
shift
for m in "$@"; do
-   find "${D}${m}" | while read a; do
-   a=$(webapp_strip_d "${a}")
-   _webapp_serverowned "${a}"
-   done
+   pushd "${D}${MY_APPDIR}" > /dev/null || die
+   # Strip appdir
+   m="${m#${MY_APPDIR}/}"
+   find "${m}" >> "${D}/${WA_SOLIST}" || die
+   popd > /dev/null || die
done
else
for m in "$@"; do
-- 
2.41.0

[gentoo-dev] Last rites: acct-{user,group}/{tpm,ultimaker} acct-group/systemd-hostname

[Florian Schmaus]

# Florian Schmaus  (2023-10-14)
# Obsolete acct-* packages that became leaf packages.
# Removal on 2023-11-14.
acct-user/tpm
acct-group/tpm
acct-user/ultimaker
acct-group/ultimaker
acct-group/systemd-hostname

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH 3/3] meson.eclass: use get_makeopts_{jobs,loadavg}

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/meson.eclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/eclass/meson.eclass b/eclass/meson.eclass
index 5aff3eb58930..4757f3fa5eef 100644
--- a/eclass/meson.eclass
+++ b/eclass/meson.eclass
@@ -388,8 +388,8 @@ meson_src_compile() {
 
local mesoncompileargs=(
-C "${BUILD_DIR}"
-   --jobs "$(makeopts_jobs "${MAKEOPTS}" 0)"
-   --load-average "$(makeopts_loadavg "${MAKEOPTS}" 0)"
+   --jobs "$(get_makeopts_jobs 0)"
+   --load-average "$(get_makeopts_loadavg 0)"
)
 
case ${MESON_VERBOSE} in
-- 
2.41.0

[gentoo-dev] [PATCH 2/3] ninja-utils.eclass: use get_makeopts_{jobs,loadavg}

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/ninja-utils.eclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/eclass/ninja-utils.eclass b/eclass/ninja-utils.eclass
index 5a211e81131d..1af3df9f71e5 100644
--- a/eclass/ninja-utils.eclass
+++ b/eclass/ninja-utils.eclass
@@ -1,3 +1,4 @@
+
 # Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
@@ -72,7 +73,7 @@ esac
 # Get the value of NINJAOPTS, inferring them from MAKEOPTS if unset.
 get_NINJAOPTS() {
if [[ -z ${NINJAOPTS+set} ]]; then
-   NINJAOPTS="-j$(makeopts_jobs "${MAKEOPTS}" 999) 
-l$(makeopts_loadavg "${MAKEOPTS}" 0)"
+   NINJAOPTS="-j$(get_makeopts_jobs 999) -l$(get_makeopts_loadavg 
0)"
fi
echo "${NINJAOPTS}"
 }
-- 
2.41.0

[gentoo-dev] [PATCH 1/3] multiprocessing.eclass: consider (GNU)MAKEFLAGS, add get_makeopts_{jobs,loadavg}

[Florian Schmaus]

Since --load-average may not be found in other Make implementations
besides GNU Make, it is potentially found in GNUMAKEFLAGS and not in
MAKEOPTS.

Signed-off-by: Florian Schmaus 
---
 eclass/multiprocessing.eclass | 40 ---
 eclass/tests/multiprocessing_makeopts_jobs.sh | 24 ++-
 2 files changed, 57 insertions(+), 7 deletions(-)

diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
index e55be636a02c..0768e7cb1e1f 100644
--- a/eclass/multiprocessing.eclass
+++ b/eclass/multiprocessing.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2022 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: multiprocessing.eclass
@@ -64,17 +64,36 @@ get_nproc() {
fi
 }
 
+# @FUNCTION: get_all_makeopts
+# @INTERNAL
+# @DESCRIPTION:
+# Returns ${MAKEOPTS} ${GNUMAKEFLAGS} ${MAKEFLAGS}.
+_get_all_makeopts() {
+   echo "${MAKEOPTS} ${GNUMAKEFLAGS} ${MAKEFLAGS}"
+}
+
+# @FUNCTION: get_makeopts_jobs
+# @USAGE: [default-jobs]
+# @DESCRIPTION:
+# Return the number of jobs extracted from the make options (MAKEOPTS,
+# GNUMAKEFLAGS, MAKEFLAGS). If the make options do not specify a number,
+# then either the provided default is returned, or 1.
+get_makeopts_jobs() {
+   local makeopts="$(_get_all_makeopts)"
+   echo $(makeopts_jobs ${makeopts} ${1:-1})
+}
+
 # @FUNCTION: makeopts_jobs
 # @USAGE: [${MAKEOPTS}] [${inf:-$(( $(get_nproc) + 1 ))}]
 # @DESCRIPTION:
-# Searches the arguments (defaults to ${MAKEOPTS}) and extracts the jobs number
+# Searches the arguments (defaults are obtained via get_all_makeopts) and 
extracts the jobs number
 # specified therein.  Useful for running non-make tools in parallel too.
 # i.e. if the user has MAKEOPTS=-j9, this will echo "9" -- we can't return the
 # number as bash normalizes it to [0, 255].  If the flags haven't specified a
 # -j flag, then "1" is shown as that is the default `make` uses.  If the flags
 # specify -j without a number, ${inf} is returned (defaults to nproc).
 makeopts_jobs() {
-   [[ $# -eq 0 ]] && set -- "${MAKEOPTS}"
+   [[ $# -eq 0 ]] && set -- "$(_get_all_makeopts)"
# This assumes the first .* will be more greedy than the second .*
# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
local jobs=$(echo " $* " | sed -r -n \
@@ -83,10 +102,21 @@ makeopts_jobs() {
echo ${jobs:-1}
 }
 
+# @FUNCTION: get_makeopts_loadavg
+# @USAGE: [default-loadavg]
+# @DESCRIPTION:
+# Return the value for the load-average extracted from the make options 
(MAKEOPTS,
+# GNUMAKEFLAGS, MAKEFLAGS).  If the make options do not specify a value, then
+# either the optional provided default is returned, or 999.
+get_makeopts_loadavg() {
+   local makeopts="$(_get_all_makeopts)"
+   echo $(makeopts_loadavg ${makeopts} ${1:-999})
+}
+
 # @FUNCTION: makeopts_loadavg
 # @USAGE: [${MAKEOPTS}] [${inf:-999}]
 # @DESCRIPTION:
-# Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set
+# Searches the arguments (defaults are obtained via get_all_makeopts()) and 
extracts the value set
 # for load-average. For make and ninja based builds this will mean new jobs are
 # not only limited by the jobs-value, but also by the current load - which 
might
 # get excessive due to I/O and not just due to CPU load.
@@ -95,7 +125,7 @@ makeopts_jobs() {
 # If no limit is specified or --load-average is used without a number, ${inf}
 # (defaults to 999) is returned.
 makeopts_loadavg() {
-   [[ $# -eq 0 ]] && set -- "${MAKEOPTS}"
+   [[ $# -eq 0 ]] && set -- "$(get_all_makeopts)"
# This assumes the first .* will be more greedy than the second .*
# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
local lavg=$(echo " $* " | sed -r -n \
diff --git a/eclass/tests/multiprocessing_makeopts_jobs.sh 
b/eclass/tests/multiprocessing_makeopts_jobs.sh
index 37d5a7257775..56d73ef48b3c 100755
--- a/eclass/tests/multiprocessing_makeopts_jobs.sh
+++ b/eclass/tests/multiprocessing_makeopts_jobs.sh
@@ -1,5 +1,5 @@
 #!/bin/bash
-# Copyright 1999-2021 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 EAPI=7
@@ -9,7 +9,13 @@ inherit multiprocessing
 
 test-makeopts_jobs() {
local exp=$1; shift
-   tbegin "makeopts_jobs($1${2+; inf=${2}}) == ${exp}"
+   local targs
+   if [[ -v 1 ]]; then
+   targs="$1${2+; inf=${2}}"
+   else
+   targs="MAKEOPTS=\"${MAKEOPTS}\" 
GNUMAKEFLAGS=\"${GNUMAKEFLAGS}\" MAKEFLAGS=\"${MAKEFLAGS}\""
+   fi
+   tbegin "makeopts_jobs(${targs}) == ${exp}"
local indirect=$(MAKEOPTS="$*&quo

[gentoo-dev] [PATCH 0/3] multiprocessing.eclass: consider (GNU)MAKEFLAGS

[Florian Schmaus]

We'd like portage to provide a sane default with of the "makeopts",
i.e., the maximum number of parallel jobs (--jobs) and the maximum
load average (--load-average) that build systems should try to
establish at once.

However, while --jobs is a pretty standard feature for Make-ish build
systems, --load-average is not. For example, it is not provided by BSD
Make. As such, it was decided that portage should set --load-average
in GNUMAKEFLAGS, instead of MAKEOPTS (see
https://marc.info/?l=gentoo-dev=169027605717531=2 and
https://github.com/gentoo/portage/pull/1072).

As a result, the multiprocessing.eclass should now also try to extract
the value for --jobs and --load-average not only from MAKEOPTS, but
also from (GNU)MAKEFLAGS.

While we add it, we also overhaul multiprocessing.eclass's API by
providing two new simple methods to extract the values:
get_multiprocessing_jobs() and get_multiprocessing_loadavg().

PR at https://github.com/gentoo/gentoo/pull/32385

Florian Schmaus (3):
  multiprocessing.eclass: consider (GNU)MAKEFLAGS, add
get_makeopts_{jobs,loadavg}
  ninja-utils.eclass: use get_makeopts_{jobs,loadavg}
  meson.eclass: use get_makeopts_{jobs,loadavg}

 eclass/meson.eclass   |  4 +-
 eclass/multiprocessing.eclass | 40 ---
 eclass/ninja-utils.eclass |  3 +-
 eclass/tests/multiprocessing_makeopts_jobs.sh | 24 ++-
 4 files changed, 61 insertions(+), 10 deletions(-)

-- 
2.41.0

[gentoo-dev] [PATCH 1/1] systemd.eclass: add systemd_install_dropin

[Florian Schmaus]

Closes: https://bugs.gentoo.org/876658
Signed-off-by: Florian Schmaus 
---
 eclass/systemd.eclass | 44 ++-
 1 file changed, 43 insertions(+), 1 deletion(-)

diff --git a/eclass/systemd.eclass b/eclass/systemd.eclass
index fbed387e0ca0..03d6a82fd310 100644
--- a/eclass/systemd.eclass
+++ b/eclass/systemd.eclass
@@ -1,4 +1,4 @@
-# Copyright 2011-2022 Gentoo Authors
+# Copyright 2011-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: systemd.eclass
@@ -227,6 +227,48 @@ systemd_install_serviced() {
)
 }
 
+# @FUNCTION: systemd_install_dropin
+# @USAGE: [--user]  
+# @DESCRIPTION:
+# Install  as the dropin file .d/00gentoo.conf,
+# overriding the settings of .
+# Defaults to system unit dropins, unless --user is provided,
+# which causes the dropin to be installed for user units.
+# The required argument  may be '-', in which case the
+# file is read from stdin and  must also be specified.
+# @EXAMPLE:
+# systemd_install_dropin foo.service "${FILESDIR}/foo.service.conf"
+# systemd_install_dropin foo.service - <<-EOF
+#  [Service]
+#  RestartSec=120
+# EOF
+systemd_install_dropin() {
+   debug-print-function ${FUNCNAME} "${@}"
+
+   local basedir
+   if [[ $# -ge 1 ]] && [[ $1 == "--user" ]]; then
+   basedir=$(_systemd_unprefix systemd_get_userunitdir)
+   shift 1
+   else
+   basedir=$(_systemd_unprefix systemd_get_systemunitdir)
+   fi
+
+   local unit=${1}
+   local src=${2}
+
+   [[ ${unit} ]] || die "No unit specified"
+   [[ ${src} ]] || die "No conf file specified"
+
+   # avoid potentially common mistake
+   [[ ${unit} == *.d ]] && die "Unit ${unit} must not have .d suffix"
+
+   (
+   insopts -m 0644
+   insinto "${basedir}/${unit}".d
+   newins "${src}" 00gentoo.conf
+   )
+}
+
 # @FUNCTION: systemd_enable_service
 # @USAGE:  
 # @DESCRIPTION:
-- 
2.41.0

[gentoo-dev] [PATCH 0/1] systemd.eclass: add systemd_install_dropin

[Florian Schmaus]

PR at https://github.com/gentoo/gentoo/pull/33089

Florian Schmaus (1):
  systemd.eclass: add systemd_install_dropin

 eclass/systemd.eclass | 44 ++-
 1 file changed, 43 insertions(+), 1 deletion(-)

-- 
2.41.0

[gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Florian Schmaus]

On 21/09/2023 21.40, Arthur Zamarin wrote:

If this is a last-rite message, the last line must list the last-rite
last date (removal date) and the last-rite bug number. You can also list


FWIW, I would assume the last-rite date to be the date where the 
package's last rites where initiated, i.e., where it was p.masked. The 
removal date would be the date on which the packages was cleaned from 
the tree.



other bugs relevant to the last-rite. So I think a format of: "Removal
on ${REMOVAL_DATE}.


I now we, myself included, often wrote "Removal on…", however, I wonder 
if we should reflect reality and write "Removal after…" instead.


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Florian Schmaus]

On 22/09/2023 08.39, Florian Schmaus wrote:
Some, including me, consider timestamps without timezone specifiers to 
be in local time (either of the consumer or producer of the timestamp). 
Hence, if you really must have UTC here, then at least consider making 
it explicit my requiring the 'Z' timezone specifier.

I forgot the to include the motivation for doing so:

Timestamps with a timezone specifier make it clear to the producer of 
the timestamp that they should produce an offset-ed timestamp and make 
it evident to the consumer of the timestamp which timezone it is in.


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Florian Schmaus]

On 21/09/2023 23.48, Sam James wrote:

Ulrich Mueller  writes:

On Thu, 21 Sep 2023, Florian Schmaus wrote:

The first line of the "#"-prefixed explanation block must be of the
format "${AUTHOR_NAME} <${EMAIL}> (${SINGLE_DATE})" when the date is of
format -MM-DD, in UTC timezone.

 



Can we drop this? Or, at least, relax this.


I think UTC makes a lot of sense in an international context like ours.


It does, but mostly if you also care about the hour and minute of the 
timestamp [1].




It also avoids flapping of the date between entries (i.e. a newer entry
having an older date than the previous one).


This appears to be mainly an esthetic issue (and probably also rarely 
happens).




the same time, I'd like us to standardise on UTC


But why? Which problem does it solve for p.mask entries?



I usually just enter my locale date here and like to avoid having to
think about that UTC is potentially in a different date. I also can
not remember any situation where the date being in UTC matters. Plus,
if you want accurate timestamps, then the git commit/author date is
here for you. :)


Users consume p.mask entries directly rather than via git.


Correct. But how many users reading p.mask entries thought about the 
timezone the date timestamp is in? I'd assume at most only a few, 
because it is not relevant to know if its +00 or +02 or -07.


Some, including me, consider timestamps without timezone specifiers to 
be in local time (either of the consumer or producer of the timestamp). 
Hence, if you really must have UTC here, then at least consider making 
it explicit my requiring the 'Z' timezone specifier (which, if you want 
to be ISO compatible, probably means that the timestamp must include 
HH:MM too).


- Flow


1: That is probably why I don't see a timezone specifier for calendar 
dates in ISO 8601.


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] Re: Standard parsable format for profiles/package.mask file

[Florian Schmaus]

On 21/09/2023 21.40, Arthur Zamarin wrote:

Hi all

I want to suggest a standard format for profiles/package.mask, for
multiple reasons:


Sounds sensible. +1


The first line of the "#"-prefixed explanation block must be of the
format "${AUTHOR_NAME} <${EMAIL}> (${SINGLE_DATE})" when the date is of
format -MM-DD, in UTC timezone.

 

Can we drop this? Or, at least, relax this.

I usually just enter my locale date here and like to avoid having to 
think about that UTC is potentially in a different date. I also can not 
remember any situation where the date being in UTC matters. Plus, if you 
want accurate timestamps, then the git commit/author date is here for 
you. :)


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] metadata.xml upstream docs as reference to scientific publications/papers

[Florian Schmaus]

On 17/09/2023 14.18, Alexander Neuwirth wrote:
Thanks. Instead of using the lang entry I can imagine these other 
approaches:


2. Adding something specific to GLEP 68, like `type="doi"> https...`. However that seems like a bit too much work for 
adding something that only a small subset of users (science) cares 
about.



  


sounds perfectly fine.

It would require (minor) adjustments to the schema and DTD. And besides 
that, packages that do not have a use for this information do not have 
to pay a cost. Hence, I am not sure why you assume its too much work.



Also integration of parsing with existing tools is an extra 
overhead.


Most XML parsers are non-strict. Which means that they ignore elements 
that they do not know. Therefore, the same argument as above can be 
made: tools that do not need to extract the information, should not 
require any adjustments.


- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] Last rites: dev-erlang/protobuffs

[Florian Schmaus]

# Florian Schmaus  (2023-09-03)
# Became a leaf package after dev-erlang/riak_pb was removed
# (989c0f55e0). Unmaintained in ::gentoo and upstream since multiple
# years. Alternative at
# https://github.com/tomas-abrahamsson/gpb.
# Two open bugs: #725584 and #793236.
# Removal on 2023-10-03.
dev-erlang/protobuffs


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/7] eclass/nuget.eclass: introduce new eclass

[Florian Schmaus]

On 31/07/2023 15.53, Michał Górny wrote:

On Mon, 2023-07-31 at 12:49 +0200, Florian Schmaus wrote:

On 31/07/2023 11.32, Sam James wrote:


Florian Schmaus  writes:


[[PGP Signed Part:Undecided]]
On 31/07/2023 07.02, Michał Górny wrote:

On Sun, 2023-07-30 at 22:19 +0200, Florian Schmaus wrote:

Which problem are we solving by moving away from this towards a slightly
more verbose construct?

The problem was that cargo.eclass ebuilds were taking significant
time
during cache regeneration and slowing down tools noticeably.  No fancy
loops required, contrary to your great theory.


Removing the $()/fork from go-modules.eclass reduced the source time
of a package from 2400 milliseconds to 236 milliseconds.

Changing, for example net-p2p/arti-1.1.6, to use _cargo_set_crate_uris
reduces the source time from 44 milliseconds to 24 milliseconds.

That is a win in relative reduction, but absolute its just 20
milliseconds. Cache regeneration is an embarrassingly parallel
problem. Therefore such a reduction should not matter much, assuming
you have some parallelism on the hardware level.


Consistency matters


Sure, I would be in favor of consistently using $(foo_uris).

Especially since the performance gains of the variable-setting approach
are even lower than I first assumed. The cargo.eclass runs the function
that computes CARGO_CRATE_URIS now twice, which adds significantly more
overhead than the fork of $(foo_uris). See my patch to the ML.



So, to summarize, your point is that after you've ignored the original
thread


I can not say that I have actively ignored the thread. I am simply not 
aware of such a thread where we discussed that the variable-setting 
pattern has to be used instead of $(foo_uris).


I am sorry, I must have missed it. And it appears that my search-foo is 
not strong enough to dig it up. Could you please point me to the thread?


Or, are you maybe referring to the thread with your cargo.eclass patches 
from June, 16th? That seemed to be only about cargo.eclass and not about 
a tree-wide policy.




and we've actually started switching stuff to ${xxx}, we should
reopen the discussion and start moving everything back to $(xxx),


I never demanded that anything should be moved back to $(foo_uris). It's 
up to the eclass maintainer to decide which reasoning the follow and how 
they weight the arguments.




even
though you've proven yourself that it's less optimal ("but only
a little!") and because... you prefer it? 


Optimality is not one-dimensional. Speed is not everything.

The $(foo_uri) pattern has slightly less code complexity and is slightly 
better readable. Moreover, given that the justification for moving away 
from it is negligible speedup, I prefer $(foo_uri).


Moving from $(foo_uri) to the variable-setting pattern ${foo_uri} appear 
to be a solution to a non-existing problem.


I have only expressed my opinion. I think that the variable-setting 
pattern is somewhat disadvantageous in this case.


However, this is not a hill for me to die on.



Yes, that certainly makes
sense.  It's surely a great way to run a distro is to undo optimizations
6 weeks later because you liked the old variant better.


Again, nobody asked for any undoing.

- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/3] multiprocessing.eclass: search also GNUMAKEFLAGS for --load-average

[Florian Schmaus]

On 31/07/2023 11.31, Sam James wrote:

Florian Schmaus  writes:

[[PGP Signed Part:Undecided]]
On 25/07/2023 10.32, Michał Górny wrote:

On Tue, 2023-07-25 at 08:26 +0200, Florian Schmaus wrote:

On 25/07/2023 06.50, Michał Górny wrote:

On Mon, 2023-07-24 at 20:57 +0200, Florian Schmaus wrote:

Since --load-average may not be found in other Make implementations
besides GNU MAKE, it is potentially found in GNUMAKEFLAGS and not in
MAKEOPTS.

Signed-off-by: Florian Schmaus 
---
eclass/multiprocessing.eclass | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
index e55be636a02c..6489ecbb44a6 100644
--- a/eclass/multiprocessing.eclass
+++ b/eclass/multiprocessing.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2022 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
   # @ECLASS: multiprocessing.eclass
@@ -86,7 +86,7 @@ makeopts_jobs() {
# @FUNCTION: makeopts_loadavg
# @USAGE: [${MAKEOPTS}] [${inf:-999}]
# @DESCRIPTION:
-# Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set
+# Searches the arguments (defaults to ${MAKEOPTS} ${GNUMAKEFLAGS}) and 
extracts the value set
# for load-average. For make and ninja based builds this will mean new jobs 
are
# not only limited by the jobs-value, but also by the current load - which 
might
# get excessive due to I/O and not just due to CPU load.
@@ -95,7 +95,7 @@ makeopts_jobs() {
# If no limit is specified or --load-average is used without a number, 
${inf}
# (defaults to 999) is returned.
makeopts_loadavg() {
-   [[ $# -eq 0 ]] && set -- "${MAKEOPTS}"
+   [[ $# -eq 0 ]] && set -- "${MAKEOPTS} ${GNUMAKEFLAGS}"
# This assumes the first .* will be more greedy than the second .*
# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
local lavg=$(echo " $* " | sed -r -n \


I'm pretty sure [GNU]MAKEFLAGS has incompatible format, in particular it
makes hyphens optional.


Yes, hyphens are optional in GNUMAKEFLAGS.

However, makeopts_loadavg() would still be able to extract the
hyphen-prefixed short (-l) and long (--load-average) options from
GNUMAKEFLAGS. Hence having makeopts_loadavg() also inspect GNUMAKEFLAGS
seems like an improvement over the current situation.


Also, shouldn't you handle MAKEFLAGS then as well?  If we're to
support
arbitrary variables used by build systems.


We could.

But GNUMAKEFLAGS was not arbitrary chosen. The idea is that portage
may set --load-average via GNUMAKEFLAGS if the user did not set
MAKEOPTS and GNUMAKEFLAGS.

See https://github.com/gentoo/portage/pull/1072

I first put --load-average into MAKEOPTS, but --load-average is not a
portable make option, that is, some Make implementation do not support
it. Adding it to GNUMAKEFLAGS, a variable already set by portage, we
avoid passing this option to a make implementation that does not
support it.

Hence, just adding GNUMAKEFLAGS is sufficient for the purpose of
propagating portage's potential new default into
multiprocessing.eclass & Co.


But this exposed a problem in that we're missing other variables
that make recognises, so we need to handle the other case too.


Sure, which one do you have in mind?

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/7] eclass/nuget.eclass: introduce new eclass

[Florian Schmaus]

On 31/07/2023 11.32, Sam James wrote:


Florian Schmaus  writes:


[[PGP Signed Part:Undecided]]
On 31/07/2023 07.02, Michał Górny wrote:

On Sun, 2023-07-30 at 22:19 +0200, Florian Schmaus wrote:

Which problem are we solving by moving away from this towards a slightly
more verbose construct?

The problem was that cargo.eclass ebuilds were taking significant
time
during cache regeneration and slowing down tools noticeably.  No fancy
loops required, contrary to your great theory.


Removing the $()/fork from go-modules.eclass reduced the source time
of a package from 2400 milliseconds to 236 milliseconds.

Changing, for example net-p2p/arti-1.1.6, to use _cargo_set_crate_uris
reduces the source time from 44 milliseconds to 24 milliseconds.

That is a win in relative reduction, but absolute its just 20
milliseconds. Cache regeneration is an embarrassingly parallel
problem. Therefore such a reduction should not matter much, assuming
you have some parallelism on the hardware level.


Consistency matters


Sure, I would be in favor of consistently using $(foo_uris).

Especially since the performance gains of the variable-setting approach 
are even lower than I first assumed. The cargo.eclass runs the function 
that computes CARGO_CRATE_URIS now twice, which adds significantly more 
overhead than the fork of $(foo_uris). See my patch to the ML.


> and I already raised the point last week as well.

Here on the mailing list, or somewhere else?

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] [PATCH] cargo.eclass: use CARGO_CRATE_URIS if already available

[Florian Schmaus]

With 59dbfb80f748 ("cargo.eclass: Add variable alternative to
$(cargo_crate_uris)") the _cargo_set_crate_uris function was
introduced. This function runs when the elcass is inherited and sets
CARGO_CRATE_URIS.

Ebuilds that use $(cargo_crate_uris) in SRC_URI will again invoke this
function, even though CARGO_CRATE_URIS is already set. Avoiding this
unnecessary computation reduces the ebuild source time of
app-shells/nushell-0.83.0 from 21 ms to 14 ms.

This is a significant reduction when compared to the variable-setting
alternative that the commit 59dbfb80f748 ("cargo.eclass: Add variable
alternative to $(cargo_crate_uris)") introduced. Using the
variable-setting approach would reduce the ebuild source time only by a
little bit more than one millisecond.

Using

pk pkg source --bench 10s '=app-shells/nushell-0.83.0'

as benchmark, yields

|   | Cached CARGO_CRATE_URIS | Non-Cached 
CARGO_CRATE_URIS |
|---+-+-|
| $(cargo_crate_uris)   | mean: 14.189ms  | mean: 21.445ms  
|
| variable-setting approach | mean: 12.822ms  | mean: 12.852ms  
|

full benchmark output

|   | Cached CARGO_CRATE_URIS (this commit) 
   | Non-Cached CARGO_CRATE_URIS
  |
|---+--+--|
| $(cargo_crate_uris)   | mean: 14.189ms, min: 13.646ms, max: 15.103ms, σ = 
149µs, N = 705 | mean: 21.445ms, min: 20.79ms,  max: 22.832ms, σ = 228µs, N = 
467 |
| variable-setting approach | mean: 12.822ms, min: 12.41ms,  max: 13.909ms, σ = 
165µs, N = 780 | mean: 12.852ms, min: 12.367ms, max: 15.437ms, σ = 227µs, N = 
779 |

Signed-off-by: Florian Schmaus 
---
 eclass/cargo.eclass | 8 
 1 file changed, 8 insertions(+)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 70b6008d9cd8..5d6911801097 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -240,6 +240,14 @@ _cargo_set_crate_uris "${CRATES}"
 # Constructs a list of crates from its arguments.
 # If no arguments are provided, it uses the CRATES variable.
 cargo_crate_uris() {
+   # Use already existing value for CARGO_CRATE_URIS, computed by
+   # _cargo_set_crate_uris, when this function is invoked without
+   # arguments.
+   if [[ $# -eq 0 && -n "${CARGO_CRATE_URIS}" ]]; then
+   echo "${CARGO_CRATE_URIS}"
+   return
+   fi
+
local crates=${*-${CRATES}}
if [[ -z ${crates} ]]; then
eerror "CRATES variable is not defined and nothing passed as 
argument"
-- 
2.41.0

Re: [gentoo-dev] [PATCH 1/7] eclass/nuget.eclass: introduce new eclass

[Florian Schmaus]

On 31/07/2023 07.02, Michał Górny wrote:

On Sun, 2023-07-30 at 22:19 +0200, Florian Schmaus wrote:

Which problem are we solving by moving away from this towards a slightly
more verbose construct?


The problem was that cargo.eclass ebuilds were taking significant time
during cache regeneration and slowing down tools noticeably.  No fancy
loops required, contrary to your great theory.


Removing the $()/fork from go-modules.eclass reduced the source time of 
a package from 2400 milliseconds to 236 milliseconds.


Changing, for example net-p2p/arti-1.1.6, to use _cargo_set_crate_uris 
reduces the source time from 44 milliseconds to 24 milliseconds.


That is a win in relative reduction, but absolute its just 20 
milliseconds. Cache regeneration is an embarrassingly parallel problem. 
Therefore such a reduction should not matter much, assuming you have 
some parallelism on the hardware level.


If switching to the variable-setting approach for something like 
cargo_crate_uris only motivated by reducing the cache regeneration a 
bit, then it is IMO borderline if its justified.


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/7] eclass/nuget.eclass: introduce new eclass

[Florian Schmaus]

On 30/07/2023 21.30, Michał Górny wrote:

On Sun, 2023-07-30 at 16:26 +0200, Maciej Barć wrote:

+# @FUNCTION: nuget_uris
+# @USAGE: 
+# @DESCRIPTION:
+# Generates the URIs to put in SRC_URI to help fetch dependencies.
+# If no arguments provided, uses the "NUGETS" variable.


Sounds like you're copying the old, horribly slow cargo.eclass API.
Don't do that, unless you have to.  Prefer variable-setting approach
used by modern cargo.eclass.


Should we make this a policy then?

On the other hand, I also wonder if its worth it.

You gave a similar comment when reviewing gradle.eclass and you changed 
the cargo.eclass to also provide the "variable-setting approach" [1].


However, while the $() construct in shells is relatively expensive due 
to the involved forking, it is usually only harmful when used in loops 
with a high iteration count. The loop in go-modules.eclass was a prime 
example of this. The loop was later fixed [2], and I can not help to 
think that this fix caused people to look for further usages of the fork 
construct.


Providing a better UX by increasing performance is good. Nevertheless, 
this should not become a which hunt.


The usage of cargo_carte_uris/ngutes_uris/gradle_src_uri is not in a 
large loop, therefore the situation is not comparable with the one in 
the go-modules.eclass [2].


Which problem are we solving by moving away from this towards a slightly 
more verbose construct?


- Flow


1: 59dbfb80f748 ("cargo.eclass: Add variable alternative to 
$(cargo_crate_uris)")

   https://github.com/gentoo/gentoo/commit/59dbfb80f748

2: 45e7aecd81e0 ("go-module.eclass: inline _go-module_gomod_encode()")
   https://github.com/gentoo/gentoo/commit/45e7aecd81e0


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/3] multiprocessing.eclass: search also GNUMAKEFLAGS for --load-average

[Florian Schmaus]

On 25/07/2023 10.32, Michał Górny wrote:

On Tue, 2023-07-25 at 08:26 +0200, Florian Schmaus wrote:

On 25/07/2023 06.50, Michał Górny wrote:

On Mon, 2023-07-24 at 20:57 +0200, Florian Schmaus wrote:

Since --load-average may not be found in other Make implementations
besides GNU MAKE, it is potentially found in GNUMAKEFLAGS and not in
MAKEOPTS.

Signed-off-by: Florian Schmaus 
---
   eclass/multiprocessing.eclass | 6 +++---
   1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
index e55be636a02c..6489ecbb44a6 100644
--- a/eclass/multiprocessing.eclass
+++ b/eclass/multiprocessing.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2022 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
   # Distributed under the terms of the GNU General Public License v2
   
   # @ECLASS: multiprocessing.eclass

@@ -86,7 +86,7 @@ makeopts_jobs() {
   # @FUNCTION: makeopts_loadavg
   # @USAGE: [${MAKEOPTS}] [${inf:-999}]
   # @DESCRIPTION:
-# Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set
+# Searches the arguments (defaults to ${MAKEOPTS} ${GNUMAKEFLAGS}) and 
extracts the value set
   # for load-average. For make and ninja based builds this will mean new jobs 
are
   # not only limited by the jobs-value, but also by the current load - which 
might
   # get excessive due to I/O and not just due to CPU load.
@@ -95,7 +95,7 @@ makeopts_jobs() {
   # If no limit is specified or --load-average is used without a number, ${inf}
   # (defaults to 999) is returned.
   makeopts_loadavg() {
-   [[ $# -eq 0 ]] && set -- "${MAKEOPTS}"
+   [[ $# -eq 0 ]] && set -- "${MAKEOPTS} ${GNUMAKEFLAGS}"
# This assumes the first .* will be more greedy than the second .*
# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
local lavg=$(echo " $* " | sed -r -n \


I'm pretty sure [GNU]MAKEFLAGS has incompatible format, in particular it
makes hyphens optional.


Yes, hyphens are optional in GNUMAKEFLAGS.

However, makeopts_loadavg() would still be able to extract the
hyphen-prefixed short (-l) and long (--load-average) options from
GNUMAKEFLAGS. Hence having makeopts_loadavg() also inspect GNUMAKEFLAGS
seems like an improvement over the current situation.



Also, shouldn't you handle MAKEFLAGS then as well?  If we're to support
arbitrary variables used by build systems.


We could.

But GNUMAKEFLAGS was not arbitrary chosen. The idea is that portage may 
set --load-average via GNUMAKEFLAGS if the user did not set MAKEOPTS and 
GNUMAKEFLAGS.


See https://github.com/gentoo/portage/pull/1072

I first put --load-average into MAKEOPTS, but --load-average is not a 
portable make option, that is, some Make implementation do not support 
it. Adding it to GNUMAKEFLAGS, a variable already set by portage, we 
avoid passing this option to a make implementation that does not support it.


Hence, just adding GNUMAKEFLAGS is sufficient for the purpose of 
propagating portage's potential new default into multiprocessing.eclass 
& Co.


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/3] multiprocessing.eclass: search also GNUMAKEFLAGS for --load-average

[Florian Schmaus]

On 25/07/2023 06.50, Michał Górny wrote:

On Mon, 2023-07-24 at 20:57 +0200, Florian Schmaus wrote:

Since --load-average may not be found in other Make implementations
besides GNU MAKE, it is potentially found in GNUMAKEFLAGS and not in
MAKEOPTS.

Signed-off-by: Florian Schmaus 
---
  eclass/multiprocessing.eclass | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
index e55be636a02c..6489ecbb44a6 100644
--- a/eclass/multiprocessing.eclass
+++ b/eclass/multiprocessing.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2022 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
  # Distributed under the terms of the GNU General Public License v2
  
  # @ECLASS: multiprocessing.eclass

@@ -86,7 +86,7 @@ makeopts_jobs() {
  # @FUNCTION: makeopts_loadavg
  # @USAGE: [${MAKEOPTS}] [${inf:-999}]
  # @DESCRIPTION:
-# Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set
+# Searches the arguments (defaults to ${MAKEOPTS} ${GNUMAKEFLAGS}) and 
extracts the value set
  # for load-average. For make and ninja based builds this will mean new jobs 
are
  # not only limited by the jobs-value, but also by the current load - which 
might
  # get excessive due to I/O and not just due to CPU load.
@@ -95,7 +95,7 @@ makeopts_jobs() {
  # If no limit is specified or --load-average is used without a number, ${inf}
  # (defaults to 999) is returned.
  makeopts_loadavg() {
-   [[ $# -eq 0 ]] && set -- "${MAKEOPTS}"
+   [[ $# -eq 0 ]] && set -- "${MAKEOPTS} ${GNUMAKEFLAGS}"
# This assumes the first .* will be more greedy than the second .*
# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
local lavg=$(echo " $* " | sed -r -n \


I'm pretty sure [GNU]MAKEFLAGS has incompatible format, in particular it
makes hyphens optional.


Yes, hyphens are optional in GNUMAKEFLAGS.

However, makeopts_loadavg() would still be able to extract the 
hyphen-prefixed short (-l) and long (--load-average) options from 
GNUMAKEFLAGS. Hence having makeopts_loadavg() also inspect GNUMAKEFLAGS 
seems like an improvement over the current situation.


But maybe I am missing something? I am not sure if your comment just a 
remark, or do you oppose the change?


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] [PATCH 3/3] meson.eclass: search also GNUMAKEFLAGS for --load-average

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/meson.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/meson.eclass b/eclass/meson.eclass
index 5aff3eb58930..65fc10b81e72 100644
--- a/eclass/meson.eclass
+++ b/eclass/meson.eclass
@@ -389,7 +389,7 @@ meson_src_compile() {
local mesoncompileargs=(
-C "${BUILD_DIR}"
--jobs "$(makeopts_jobs "${MAKEOPTS}" 0)"
-   --load-average "$(makeopts_loadavg "${MAKEOPTS}" 0)"
+   --load-average "$(makeopts_loadavg "${MAKEOPTS} 
${GNUMAKEFLAGS}" 0)"
)
 
case ${MESON_VERBOSE} in
-- 
2.41.0

[gentoo-dev] [PATCH 2/3] ninja-utils.eclass: search also GNUMAKEFLAGS for --load-average

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/ninja-utils.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/ninja-utils.eclass b/eclass/ninja-utils.eclass
index 5a211e81131d..3d15a32c5e6c 100644
--- a/eclass/ninja-utils.eclass
+++ b/eclass/ninja-utils.eclass
@@ -72,7 +72,7 @@ esac
 # Get the value of NINJAOPTS, inferring them from MAKEOPTS if unset.
 get_NINJAOPTS() {
if [[ -z ${NINJAOPTS+set} ]]; then
-   NINJAOPTS="-j$(makeopts_jobs "${MAKEOPTS}" 999) 
-l$(makeopts_loadavg "${MAKEOPTS}" 0)"
+   NINJAOPTS="-j$(makeopts_jobs "${MAKEOPTS}" 999) 
-l$(makeopts_loadavg "${MAKEOPTS} ${GNUMAKEFLAGS}" 0)"
fi
echo "${NINJAOPTS}"
 }
-- 
2.41.0

[gentoo-dev] [PATCH 1/3] multiprocessing.eclass: search also GNUMAKEFLAGS for --load-average

[Florian Schmaus]

Since --load-average may not be found in other Make implementations
besides GNU MAKE, it is potentially found in GNUMAKEFLAGS and not in
MAKEOPTS.

Signed-off-by: Florian Schmaus 
---
 eclass/multiprocessing.eclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
index e55be636a02c..6489ecbb44a6 100644
--- a/eclass/multiprocessing.eclass
+++ b/eclass/multiprocessing.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2022 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: multiprocessing.eclass
@@ -86,7 +86,7 @@ makeopts_jobs() {
 # @FUNCTION: makeopts_loadavg
 # @USAGE: [${MAKEOPTS}] [${inf:-999}]
 # @DESCRIPTION:
-# Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set
+# Searches the arguments (defaults to ${MAKEOPTS} ${GNUMAKEFLAGS}) and 
extracts the value set
 # for load-average. For make and ninja based builds this will mean new jobs are
 # not only limited by the jobs-value, but also by the current load - which 
might
 # get excessive due to I/O and not just due to CPU load.
@@ -95,7 +95,7 @@ makeopts_jobs() {
 # If no limit is specified or --load-average is used without a number, ${inf}
 # (defaults to 999) is returned.
 makeopts_loadavg() {
-   [[ $# -eq 0 ]] && set -- "${MAKEOPTS}"
+   [[ $# -eq 0 ]] && set -- "${MAKEOPTS} ${GNUMAKEFLAGS}"
# This assumes the first .* will be more greedy than the second .*
# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
local lavg=$(echo " $* " | sed -r -n \
-- 
2.41.0

Re: [gentoo-dev] [RFC PATCH] metadata: Add gnome package stabilization groups

[Florian Schmaus]

On 19/07/2023 18.10, Matt Turner wrote:

Signed-off-by: Matt Turner 
---
Feel free to bikeshed the location, structure, file-format, etc.

  metadata/stabilization-groups/gnome/evolution | 3 +++
  metadata/stabilization-groups/gnome/glib  | 3 +++
  metadata/stabilization-groups/gnome/gnome-shell   | 4 
  metadata/stabilization-groups/gnome/gobject-introspection | 2 ++
  metadata/stabilization-groups/gnome/sysprof   | 3 +++
  metadata/stabilization-groups/gnome/vala  | 2 ++
  metadata/stabilization-groups/gnome/vte   | 3 +++
  7 files changed, 20 insertions(+)
  create mode 100644 metadata/stabilization-groups/gnome/evolution
  create mode 100644 metadata/stabilization-groups/gnome/glib
  create mode 100644 metadata/stabilization-groups/gnome/gnome-shell
  create mode 100644 metadata/stabilization-groups/gnome/gobject-introspection
  create mode 100644 metadata/stabilization-groups/gnome/sysprof
  create mode 100644 metadata/stabilization-groups/gnome/vala
  create mode 100644 metadata/stabilization-groups/gnome/vte

diff --git a/metadata/stabilization-groups/gnome/evolution 
b/metadata/stabilization-groups/gnome/evolution
new file mode 100644
index ..21bbcf804e94
--- /dev/null
+++ b/metadata/stabilization-groups/gnome/evolution
@@ -0,0 +1,3 @@
+gnome-extra/evolution-data-server
+gnome-extra/evolution-ews
+mail-client/evolution
diff --git a/metadata/stabilization-groups/gnome/glib 
b/metadata/stabilization-groups/gnome/glib
new file mode 100644
index ..51a5659dd725
--- /dev/null
+++ b/metadata/stabilization-groups/gnome/glib
@@ -0,0 +1,3 @@
+dev-libs/glib
+dev-util/gdbus-codegen
+dev-util/glib-utils
diff --git a/metadata/stabilization-groups/gnome/gnome-shell 
b/metadata/stabilization-groups/gnome/gnome-shell
new file mode 100644
index ..ddf76f8f88f4
--- /dev/null
+++ b/metadata/stabilization-groups/gnome/gnome-shell
@@ -0,0 +1,4 @@
+gnome-base/gnome-shell
+gnome-extra/gnome-shell-extensions
+gnome-extra/gnome-shell-frippery
+x11-wm/mutter
diff --git a/metadata/stabilization-groups/gnome/gobject-introspection 
b/metadata/stabilization-groups/gnome/gobject-introspection
new file mode 100644
index ..8baf4ae59124
--- /dev/null
+++ b/metadata/stabilization-groups/gnome/gobject-introspection
@@ -0,0 +1,2 @@
+dev-libs/gobject-introspection
+dev-libs/gobject-introspection-common
diff --git a/metadata/stabilization-groups/gnome/sysprof 
b/metadata/stabilization-groups/gnome/sysprof
new file mode 100644
index ..66a338916039
--- /dev/null
+++ b/metadata/stabilization-groups/gnome/sysprof
@@ -0,0 +1,3 @@
+dev-util/sysprof
+dev-util/sysprof-capture
+dev-util/sysprof-common
diff --git a/metadata/stabilization-groups/gnome/vala 
b/metadata/stabilization-groups/gnome/vala
new file mode 100644
index ..2e4d5a33748d
--- /dev/null
+++ b/metadata/stabilization-groups/gnome/vala
@@ -0,0 +1,2 @@
+dev-lang/vala
+dev-libs/vala-common
diff --git a/metadata/stabilization-groups/gnome/vte 
b/metadata/stabilization-groups/gnome/vte
new file mode 100644
index ..ce25ab265262
--- /dev/null
+++ b/metadata/stabilization-groups/gnome/vte
@@ -0,0 +1,3 @@
+gui-libs/vte
+gui-libs/vte-common
+x11-libs/vte



Couldn't stabilization groups be package sets? For example, instead of

metadata/stabilization-groups/gnome/vte

you have

sets/stable-group-gnome-vte

with the exact same content (may requires that :;gentoo declares a 
sets.conf).


- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH v2] meson.eclass: allow disabling verbose compilation

[Florian Schmaus]

On 20/07/2023 17.00, Matt Turner wrote:

On Wed, Jul 19, 2023 at 3:23 AM Florian Schmaus  wrote:


On 18/07/2023 18.44, Matt Turner wrote:

From: Jonas Rakebrandt 

This works similar to cmake.eclass's ${CMAKE_VERBOSE}.

Closes: https://github.com/gentoo/gentoo/pull/28942
Signed-off-by: Jonas Rakebrandt 
Signed-off-by: Matt Turner 
---
   eclass/meson.eclass | 15 +--
   1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/eclass/meson.eclass b/eclass/meson.eclass
index 2c274b213191..3b30f66bf30a 100644
--- a/eclass/meson.eclass
+++ b/eclass/meson.eclass
@@ -55,6 +55,12 @@ BDEPEND=">=dev-util/meson-0.62.2
   # Build directory, location where all generated files should be placed.
   # If this isn't set, it defaults to ${WORKDIR}/${P}-build.

+# @ECLASS_VARIABLE: MESON_VERBOSE
+# @USER_VARIABLE
+# @DESCRIPTION:
+# Set to OFF to disable verbose messages during compilation
+: "${MESON_VERBOSE:=ON}"
+
   # @ECLASS_VARIABLE: EMESON_BUILDTYPE
   # @DESCRIPTION:
   # The buildtype value to pass to meson setup.
@@ -385,10 +391,15 @@ meson_src_compile() {
   -C "${BUILD_DIR}"
   --jobs "$(makeopts_jobs "${MAKEOPTS}" 0)"
   --load-average "$(makeopts_loadavg "${MAKEOPTS}" 0)"
- --verbose
- "$@"
   )

+ case ${MESON_VERBOSE} in
+ OFF) ;;
+ *) mesoncompileargs+=( --verbose ) ;;
+ esac


No strong opinion, just to educate myself, but is there an advantage of
using case/easc over if/fi here?

That is

if [[ ${MESON_VERBOSE} != off ]]; then
  mesoncompileargs+=( --verbose )
fi

or even the shell-style short idiom using ||.


No advantage as far as I'm aware. I was just copying the style used in
cmake.eclass.

I really wish bash just had boolean types :(


While the bash language has no boolean datatype, you can exploit the 
fact that 'true' and 'false' are usually shell builtins:


: "${MESON_VERBOSE:=true}"

and then later

if $MESON_VERBOSE; then
mesoncompileargs+=( --verbose )
fi

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH v2] meson.eclass: allow disabling verbose compilation

[Florian Schmaus]

On 18/07/2023 18.44, Matt Turner wrote:

From: Jonas Rakebrandt 

This works similar to cmake.eclass's ${CMAKE_VERBOSE}.

Closes: https://github.com/gentoo/gentoo/pull/28942
Signed-off-by: Jonas Rakebrandt 
Signed-off-by: Matt Turner 
---
  eclass/meson.eclass | 15 +--
  1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/eclass/meson.eclass b/eclass/meson.eclass
index 2c274b213191..3b30f66bf30a 100644
--- a/eclass/meson.eclass
+++ b/eclass/meson.eclass
@@ -55,6 +55,12 @@ BDEPEND=">=dev-util/meson-0.62.2
  # Build directory, location where all generated files should be placed.
  # If this isn't set, it defaults to ${WORKDIR}/${P}-build.
  
+# @ECLASS_VARIABLE: MESON_VERBOSE

+# @USER_VARIABLE
+# @DESCRIPTION:
+# Set to OFF to disable verbose messages during compilation
+: "${MESON_VERBOSE:=ON}"
+
  # @ECLASS_VARIABLE: EMESON_BUILDTYPE
  # @DESCRIPTION:
  # The buildtype value to pass to meson setup.
@@ -385,10 +391,15 @@ meson_src_compile() {
-C "${BUILD_DIR}"
--jobs "$(makeopts_jobs "${MAKEOPTS}" 0)"
--load-average "$(makeopts_loadavg "${MAKEOPTS}" 0)"
-   --verbose
-   "$@"
)
  
+	case ${MESON_VERBOSE} in

+   OFF) ;;
+   *) mesoncompileargs+=( --verbose ) ;;
+   esac


No strong opinion, just to educate myself, but is there an advantage of 
using case/easc over if/fi here?


That is

if [[ ${MESON_VERBOSE} != off ]]; then
mesoncompileargs+=( --verbose )
fi

or even the shell-style short idiom using ||.

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: [gentoo-dev-announce] Last rites: obsolete acct-* packages

[Florian Schmaus]

On 18/07/2023 11.56, Sam James wrote:


Mike Gilbert  writes:


On Mon, Jul 17, 2023 at 4:27 PM Sam James  wrote:

Haven't we been keeping these because we still need to decide on a
policy about what to do with dead acct-*/* packages?


Right. https://bugs.gentoo.org/781881 is still open. Flow could ping
the QA team and ask if it should be closed, given the opinion there
seems to be that there's no need to keep them, but I think it's wrong
to do this pre-empting a policy decision, given it essentially forces
the "don't keep them" path.


The bug has been open for several months without comment. If a policy
were going to materialize, I think it would have happened by now.

Forcing the issue by sending this last rites notice seems acceptable to me.


Pinging someone rather than "forcing the issue" as a first-step is customary.


I am sorry, but it seems that I have to clarify something.

First, I have "pinged someone."

As of writing this, I was the last to comment on the QA bug about five 
months ago, asking why we would want to keep unused acct-* packages [1]. 
Since then, this has not been answered, and there have been zero other 
replies. That signaled me that there was no interest in pursuing the 
matter further. In addition, we have already removed acct-* packages in 
the past.


Secondly, nobody immediately forces anything.

The removal date for the acct-* packages is one month from now. One 
month is hopefully enough time to decide whether we want such a policy.



Sam, I am afraid, but I believe that the situation is different from how 
you frame it.



The proponents of keeping obsolete acct-* packages have the inventive to 
establish their preferred policy.


Accusing me of not facilitating a QA bug that deals with establishing a 
policy I do not favor seems unfair.


Do you think that a QA bug that has not seen progress in nearly five 
months should be able to establish an illegitimate shadow policy?


Currently, acct-* packages are governed by our current policy regarding 
package removal. If we decide to change this, we can also revert the 
acct-* package removals.


- Flow


1: https://bugs.gentoo.org/781881#c7



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: [gentoo-dev-announce] Last rites: obsolete acct-* packages

[Florian Schmaus]

On 18/07/2023 10.42, Зураб Квачадзе wrote:

How do we handle this case, then.
Imagine we have a leaf package acct-user/foo, which has a reserved UID 
of 123. It gets last rited and its entry is removed from uid-gid.txt.


Nobody is proposing that the uid-gid.txt entry is removed. Ideally, it 
would be marked as 'historical', together with the date it went historical.



After a while appears a new package acct-user/bar, which takes the 123 
UID. Then a user, say Bob, updates their system, which haven't been 
updated for some time. What if they still have acct-user/foo, when > acct-user/bar with the same UID is installed?


If a UID/GID is in use, then acct-*.eclass will find the next suitable 
ID (unless, e.g. ACCT_USER_ENFORCE_ID is set, we is usually not the case).


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] Last rites: obsolete acct-* packages

[Florian Schmaus]

# Florian Schmaus  (2023-07-17)
# Obsolete acct-* packages which became leaf packages.
# Removal on 2023-08-17.
acct-user/artifactory
acct-group/artifactory
acct-user/cinder
acct-group/cinder
acct-user/glance
acct-group/glance
acct-user/heat
acct-group/heat
acct-user/keystone
acct-group/keystone
acct-user/litecoin
acct-group/litecoin
acct-user/logcheck
acct-group/logcheck
acct-user/minbif
acct-group/minbif
acct-user/minio
acct-group/minio
acct-user/netbox
acct-group/netbox
acct-user/neutron
acct-group/neutron
acct-user/nova
acct-group/nova
acct-user/placement
acct-group/placement
acct-user/quagga
acct-group/quagga
acct-user/rplayd
acct-group/rplayd
acct-user/rstudio-server
acct-group/rstudio-server
acct-user/rundeck
acct-group/rundeck
acct-user/sguil
acct-group/sguil
acct-user/sigh
acct-group/sigh
acct-user/smokeping
acct-group/smokeping
acct-user/sobby
acct-group/sobby
acct-user/spread
acct-group/spread
acct-user/stg
acct-group/stg
acct-user/swift
acct-group/swift
acct-user/thttpd
acct-group/thttpd
acct-group/gpio
acct-group/simplevirt
acct-group/spi

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] Re: Flow's Manifesto and questions for nominees (was: Re: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)

[Florian Schmaus]

On 14/07/2023 09.33, Sam James wrote:


Florian Schmaus  writes:


[[PGP Signed Part:Undecided]]
Posted to gentoo-dev@ since we are now entering a technical discussion
again.

For those who did not follow gentoo-project@, the previous posts include:

https://marc.info/?l=gentoo-project=168918875000738=2
https://marc.info/?l=gentoo-project=168881103930591=2

On 12/07/2023 21.28, Alec Warner wrote:

On Wed, Jul 12, 2023 at 12:07 PM Florian Schmaus  wrote:

Apologies for not replying to everyone individually.

I thank my fellow council candidates who took the time to reply to this
sensitive and obviously controversial matter. I understand that not
everyone feels comfortable taking a stance in this discussion.

I asked the other council candidates about their opinion on EGO_SUM.
Unfortunately, some replies included only a rather shallow answer. A few
focused on criticism of my actions and how I approach the issue. Which
is obviously fine. I read it all and have empathy for everyone who feels
aggravated. You may or may not share the complaints. But let us focus on
the actual matter for a moment.

Even the voices raised for a restricted reintroduction of EGO_SUM just
mention an abstract limit [1]. A concrete limit is not mentioned,
although I asked for it and provided my idea including specific limits.
Not knowing the concrete figures others have in mind makes it difficult
to find a compromise. For example, a fellow council candidate postulated
that it would be quicker for me to implement a limit-check in pkgcheck
than discuss EGO_SUM. I wish that were the case. Unfortunately it is


I think this misrepresents my point. All I said was that a bound should
be added matching what's in Portage right now.

Please in future respond to me directly if you're going to claim something 
about what I've said.


[...]
EGO_SUM affects two dimensions that could be limited/restricted:
A) the process environment, which may run into the Linux kernel
environment limit on exec(3)
B) the size of the package directory, where EGO_SUM affects the size of
ebuilds and the Manifest

[...]

A), however, is a different beast. There is undeniably a
kernel-enforced limit that we could hit due to an extremely large
EGO_SUM (among other things). However, the only bug report I know that
runs into this kernel limit was with texlive (bug #719202). The low
number of recorded bugs caused by the environment limit matches with
the fact that even the ebuild with the most EGO_SUM entries that I
ever analyzed, app-containers/cri-o-1.23.1 (2022-02-16) with 2052
EGO_SUM entries, does *not* run into the environment limit.



I thought I'd gave you a list before, but maybe it was someone else.

Anyway, a non-exhaustive list (I remember maybe two more but I got bored):
* https://bugs.gentoo.org/829545 ("app-admin/vault-1.9.1 - find: The environment is 
too large for exec().")
* https://bugs.gentoo.org/829684 ("app-metrics/prometheus-2.31.1 - find: The 
environment is too large for exec().")
* https://bugs.gentoo.org/830187 (you're CC'd on this) ("go lang ebuild: SRC_URI too long that 
it causes "Argument list too long" error")
* https://bugs.gentoo.org/831265 ("sys-cluster/minikube-1.24.0 - find: The 
environment is too large for exec().")
* a0be89b772474e3336d3de699d71482aa89d2444 ("app-emulation/nerdctl: drop 
0.14.0")


Thanks for providing this valuable information, Sam. I was indeed not 
aware of those bugs. They all seem to be fixed before 2022-02-16, that 
is the date of the ::gentoo tree I mostly analyzed (which was selected 
because it was just before EGO_SUM was deprecated).


Limiting the process environment to 90% of the kernel-enforced limit, as 
antarus also suggested (potentially by approximating the EGO_SUM 
entries) would have probably prevented those bugs. As I previously 
wrote, I would be happy to work on a pkgcheck for that, if the limit is 
only about the kernel's process environment limit (A).


However this still leaves us with some that seem to also demand a limit 
with regard to the package-directory size (B).




Other related bugs (as it's useful as a summary of where we are):
* https://bugs.gentoo.org/540146 ("sys-apps/portage: limit no of exported variables 
in EAPI 6")
* https://bugs.gentoo.org/720180 ("sys-apps/portage: add support to delay export of 
"A" variable until last moment")
* https://bugs.gentoo.org/721088 ("[Future EAPI] Don't export A")
* https://bugs.gentoo.org/833567 ("[Future EAPI] src_fetch_extra phase the runs 
after src_unpack")

I am not aware of a bug (yet?) for radhermit's suggestion wrt external
helpers which is related but different to exporting fewer variables.


Improving, that is, reducing, what portage exports to child processes of 
the ebuild is sensible. But it is only indirectly related to EGO_SUM and 
not a strict prerequisite to re-introduce EGO_SUM.


- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] Re: Flow's Manifesto and questions for nominees (was: Re: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)

[Florian Schmaus]

Posted to gentoo-dev@ since we are now entering a technical discussion 
again.


For those who did not follow gentoo-project@, the previous posts include:

https://marc.info/?l=gentoo-project=168918875000738=2
https://marc.info/?l=gentoo-project=168881103930591=2

On 12/07/2023 21.28, Alec Warner wrote:

On Wed, Jul 12, 2023 at 12:07 PM Florian Schmaus  wrote:

Apologies for not replying to everyone individually.

I thank my fellow council candidates who took the time to reply to this
sensitive and obviously controversial matter. I understand that not
everyone feels comfortable taking a stance in this discussion.

I asked the other council candidates about their opinion on EGO_SUM.
Unfortunately, some replies included only a rather shallow answer. A few
focused on criticism of my actions and how I approach the issue. Which
is obviously fine. I read it all and have empathy for everyone who feels
aggravated. You may or may not share the complaints. But let us focus on
the actual matter for a moment.

Even the voices raised for a restricted reintroduction of EGO_SUM just
mention an abstract limit [1]. A concrete limit is not mentioned,
although I asked for it and provided my idea including specific limits.
Not knowing the concrete figures others have in mind makes it difficult
to find a compromise. For example, a fellow council candidate postulated
that it would be quicker for me to implement a limit-check in pkgcheck
than discuss EGO_SUM. I wish that were the case. Unfortunately it is
potentially not trivial to implement if we want such a check to be
robust. But even worse, a specific limit must be known before
implementing such a check. And we currently have none.


I think my concern here is that I don't expect the Council to really
'vote on a specific limit.' The limit is an implementation detail, it
can change, it shouldn't require a council vote to change.

So my advice is "pick something reasonable that you think holds up to
scrutiny, and implement with that" and "expect the limit to change,
either because of the scrutiny, or because it might change in the
future" and implement your check accordingly (so e.g. the limit is
easily changeable.)


Please find below why this may not be enough.



But the real crux of an EGO_SUM reintroduction with a limit is the
following. Either the limit is too restrictive, and most packages are
affected by it and can not use EGO_SUM, which ultimately only
corresponds to the current state. Or the limit only affects a fraction
of the packages, so you should not bother having a limit.


Again the idea is there is already a limit ( the aforementioned
environment limit ) and one of the goals is to have a QA check that
says your ebuild is approaching that limit so you can do something
productive about it, as well as to avoid ebuilds that are not
installable. So just implement that. If you need a number, I think
"90% of the env limit" is defensible (but again, any reasonable number
will do fine.)


EGO_SUM affects two dimensions that could be limited/restricted:
A) the process environment, which may run into the Linux kernel
   environment limit on exec(3)
B) the size of the package directory, where EGO_SUM affects the size of
   ebuilds and the Manifest

I would be happy to put in any effort required to implement A) in 
pkgcheck, as I did for portage, if this check is the only thing that 
keeps us from reintroducing EGO_SUM.


Unfortunately, some argue that we need to limit B). Much of the effort I 
put into researching the EGO_SUM situation was analyzing how EGO_SUM's 
impact on package-directory size affects Gentoo. The result of the 
analysis strongly indicates that rather large package-directories can be 
sustained by ::gentoo in the foreseeable future. Especially since we are 
only talking about ~250 EGO_SUM packages currently, and a significant 
fraction of those packages will not have enormous package directories. 
And I also suggested that the policy is reconsidered at least every two 
years or once the number of EGO_SUM packages has doubled (whatever comes 
first).


My investigation of the history of EGO_SUM's deprecation has not 
surfaced any technical issue which justified EGO_SUM's deprecation with 
regard to B). It appears that technical issues do not drive the desire 
to limit B), but by esthetic preferences, which are highly subjective.


A), however, is a different beast. There is undeniably a kernel-enforced 
limit that we could hit due to an extremely large EGO_SUM (among other 
things). However, the only bug report I know that runs into this kernel 
limit was with texlive (bug #719202). The low number of recorded bugs 
caused by the environment limit matches with the fact that even the 
ebuild with the most EGO_SUM entries that I ever analyzed, 
app-containers/cri-o-1.23.1 (2022-02-16) with 2052 EGO_SUM entries, does 
*not* run into the environment limit.




The deprecation of EGO_SUM was and is unnecessary, a securit

Re: [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)

[Florian Schmaus]

On 30/06/2023 13.33, Eray Aslan wrote:

On Fri, Jun 30, 2023 at 03:38:11AM -0600, Tim Harder wrote:

Why do we have to keep exporting the related variables that generally
cause these size issues to the environment?


I really do not want to make a +1 response but this is an excellent
question that we need to answer before implementing EGO_SUM.


Could you please discuss why you make the reintroduction of EGO_SUM 
dependent on this question?


Portage will show you a warning message if the exported environment 
approaches the kernel limit, and it will show a detailed error message 
if executing an ebuild failed due to the limit being reached. There 
seems to be no reason why you should not be able to allow EGO_SUM again 
without first fixing, for example, https://bugs.gentoo.org/721088.


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)

[Florian Schmaus]

On 30/06/2023 10.22, Sam James wrote:

Florian Schmaus  writes:

[[PGP Signed Part:Undecided]]
[in reply to a gentoo-project@ post, but it was asked to continue this
on gentoo-dev@]
On 28/06/2023 16.46, Sam James wrote:

and questions remain unanswered on the
ML (why not implement a check in pkgcheck similar to what is in Portage,
for example)?


On 2023-05-30 [1], I proposed a limit in the range of 2 to 1.5 MiB for
the total package-directory size. I only care a little about the tool
that checks this limit, but pkgcheck is an obvious choice. I also
suggested that we review this policy once the number of Go packages
has doubled or two years after this policy was established (whatever
comes first).

But I fear you may be referring to another kind of check. You may be
talking about a check that forbids EGO_SUM in ::gentoo but allows it
overlays.


My position on this has been consistent:  > a check is needed to statically
determine when the environment size is too big. Copying the Portage
check into pkgcheck (in terms of the metrics) would satisfy this.


It is not as easy as merely copying existing portage code into pkgcheck 
(unless I am missing something).


I've talked to arthurzam, and there appears to be a .environment file 
created by pkgcheck, which we could use to approximate the exported 
environment.


Another option would be to have pkgcheck count the EGO_SUM entries. The 
tree-sitter API for Bash, which pkgcheck already uses, seems to allow 
for that. But that would be different from the check in portage. 
Although, IMHO, counting EGO_SUM entries would be sufficient.




That is, regardless of raw size, I'm asking for a calculation based on
the contents of EGO_SUM where, if exceeded, the package will not be
installable on some systems. You didn't have an issue implementing this
for Portage and I've mentioned this a bunch of times since, so I thought
it was clear what I was hoping to see.


So pkgcheck counting EGO_SUM entries would be sufficient for the purpose 
of having a static check that notices if the ebuild would likely run 
into the environment limit?


To find a common compromise, I would possibly invest my time in 
developing such a test. Even though I do not deem such a check a strict 
prerequisite to reintroduce EGO_SUM.




Intelligibly, EGO_SUM can be considered ugly. Compared to a
traditional Gentoo package, EGO_SUM-based ones are larger. The same is
true for Rust packages. However, looking at the bigger picture,
EGO_SUM's advantages outweigh its disadvantages.



Again, am on record as being fine with the general EGO_SUM approach,
even if I wish we didn't need it, as I see it as inevitable for things
like yarn, .NET, and of course Rust as we already have it.

Just ideally not huge ones, and certainly not huge ones which then
aren't even reliably installable because of environment size.


Talking about "reliably installable" makes it sound to me like there are 
cases where installing a EGO_SUM-based package sometimes works and 
sometimes not. But the kernel-limit is fixed and not even configurable, 
besides, of course patching the source (and in the absence of 
architectures with a page size below 4 KiB) [1].


Any developer testing whether or not an ebuild is installable would 
become immediately aware if the ebuild runs into the environment limit, 
or not.


That said, static code checks are always preferable over dynamic ones.

- Flow


1: 
https://elixir.bootlin.com/linux/v6.4.1/source/include/uapi/linux/binfmts.h#L15




OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)

[Florian Schmaus]

[in reply to a gentoo-project@ post, but it was asked to continue this 
on gentoo-dev@]


On 28/06/2023 16.46, Sam James wrote:

Florian Schmaus  writes:

On 17/06/2023 10.37, Arthur Zamarin wrote:

I also want to nominate people who I feel contribute a lot to Gentoo and
I have a lot of interaction with (ordered by name, not priority):
[…]
flow


I apologize for the late reply, and thank you for the nomination. I am
honored and accept.

As many of you know, I am spending a lot of time on the EGO_SUM
situation, as it is one of the most critical issues to solve.

I have used the last few days to carefully consider whether a seat on
the council is more harmful or beneficial to my efforts regarding
EGO_SUM. On the one hand, council work means I have less time to
improve the EGO_SUM situation. On the other hand, a seat in the
council increases the probability of positively influencing Gentoo's
future, also regarding EGO_SUM.



That's fine and it's great to see more people running!


Excellent that we share this view. :)



But with regard to EGO_SUM: you didn't appear at the meeting where we discussed
your previous EGO_SUM proposal,


Naively, as I am, I expected that the mailing list would be used for 
discussion and that the council meeting would be used chiefly for voting 
and intra-council discussion. And since the request to the council to 
vote on a concrete proposal was preceded by a multiple-week, if not 
month-long, mailing list discussion, I assumed that my presence in the 
council meeting was optional.


Had I known that my presence was required, or that the absence in the 
meeting would be blamed on me afterward, I would have appeared if possible.




and questions remain unanswered on the
ML (why not implement a check in pkgcheck similar to what is in Portage,
for example)?


On 2023-05-30 [1], I proposed a limit in the range of 2 to 1.5 MiB for 
the total package-directory size. I only care a little about the tool 
that checks this limit, but pkgcheck is an obvious choice. I also 
suggested that we review this policy once the number of Go packages has 
doubled or two years after this policy was established (whatever comes 
first).


But I fear you may be referring to another kind of check. You may be 
talking about a check that forbids EGO_SUM in ::gentoo but allows it 
overlays.


However, as stated before [2], this is not a viable approach. One reason 
why it is not practicable is auditability.




The blocker is not a council seat, it's about addressing people's
concerns...


Unfortunately, it appears that I am terrible at convincing everyone that 
the deprecation of EGO_SUM was a mistake. I tried to respond to every 
concern. Often, the response included arguments based on factual data. 
But eventually, I would only expect to convince some, as the EGO_SUM 
question touches the subjective realm of style.


I know that the EGO_SUM situation and the resulting discussion grew huge 
and left many understandably bored or confused, which then turned away. 
But that is a pity because it is a relevant discussion for Gentoo's 
long-term success.


The bottom line is that the EGO_SUM discussion yielded no evidence or 
even a slight indication that EGO_SUM was deprecated based on technical 
issues. Instead, it appears that EGO_SUM was deprecated because some 
deemed it unaesthetic.


Intelligibly, EGO_SUM can be considered ugly. Compared to a traditional 
Gentoo package, EGO_SUM-based ones are larger. The same is true for Rust 
packages. However, looking at the bigger picture, EGO_SUM's advantages 
outweigh its disadvantages.


- Flow


1: https://marc.info/?l=gentoo-dev=168546196902731 
<25308876-7ac4-8c90-8641-1034cc67c...@gentoo.org>
2: https://marc.info/?l=gentoo-dev=168569387514376 
<012fa74d-2910-ea90-6008-26cc23604...@gentoo.org>


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] [PATCH 2/2] dev-java/openjfx: switch to gradle.eclass

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 dev-java/openjfx/openjfx-11.0.11_p1.ebuild | 46 +++---
 1 file changed, 15 insertions(+), 31 deletions(-)

diff --git a/dev-java/openjfx/openjfx-11.0.11_p1.ebuild 
b/dev-java/openjfx/openjfx-11.0.11_p1.ebuild
index 7d61ff67f2c2..a99111598f7c 100644
--- a/dev-java/openjfx/openjfx-11.0.11_p1.ebuild
+++ b/dev-java/openjfx/openjfx-11.0.11_p1.ebuild
@@ -1,19 +1,19 @@
-# Copyright 2019-2021 Gentoo Authors
+# Copyright 2019-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
-EAPI=6
+EAPI=8
 
 MY_PV="${PV/_p/+}"
 SLOT="${MY_PV%%[.+]*}"
-EGRADLE_VER="4.10.3"
+EGRADLE_BUNDLED_VER="4.10.3"
 
-inherit flag-o-matic java-pkg-2 multiprocessing
+inherit flag-o-matic gradle java-pkg-2 multiprocessing
 
 DESCRIPTION="Java OpenJFX client application platform"
 HOMEPAGE="https://openjfx.io;
 
-SRC_URI="https://hg.openjdk.java.net/${PN}/${SLOT}-dev/rt/archive/${MY_PV}.tar.bz2
 -> ${P}.tar.bz2
-   https://downloads.gradle.org/distributions/gradle-${EGRADLE_VER}-bin.zip
+SRC_URI="
+   
https://hg.openjdk.java.net/${PN}/${SLOT}-dev/rt/archive/${MY_PV}.tar.bz2 -> 
${P}.tar.bz2

https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-sandbox/7.1.0/lucene-sandbox-7.1.0.jar

https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-grouping/7.1.0/lucene-grouping-7.1.0.jar

https://repo.maven.apache.org/maven2/org/apache/lucene/lucene-queryparser/7.1.0/lucene-queryparser-7.1.0.jar
@@ -22,6 +22,7 @@ 
SRC_URI="https://hg.openjdk.java.net/${PN}/${SLOT}-dev/rt/archive/${MY_PV}.tar.b

https://repo.maven.apache.org/maven2/org/antlr/gunit/3.5.2/gunit-3.5.2.jar

https://repo1.maven.org/maven2/org/antlr/antlr4/4.7.2/antlr4-4.7.2-complete.jar
https://repo.maven.apache.org/maven2/org/antlr/ST4/4.0.8/ST4-4.0.8.jar
+   $(gradle-src_uri)
 "
 
 LICENSE="GPL-2-with-classpath-exception"
@@ -84,28 +85,8 @@ PATCHES=(
 
 S="${WORKDIR}/rt-${MY_PV}"
 
-egradle() {
-   local GRADLE_HOME="${WORKDIR}/gradle-${EGRADLE_VER}"
-   local gradle="${GRADLE_HOME}/bin/gradle"
-   local gradle_args=(
-   --info
-   --stacktrace
-   --no-build-cache
-   --no-daemon
-   --offline
-   --gradle-user-home "${T}/gradle_user_home"
-   --project-cache-dir "${T}/gradle_project_cache"
-   )
-
-   export GRADLE_HOME
-
-   # FIXME: build.gradle believes $ANT_HOME/bin/ant shoud exist
-   unset ANT_HOME
-
-   einfo "gradle "${gradle_args[@]}" ${@}"
-   # TERM needed, otherwise gradle may fail on terms it does not know about
-   TERM="xterm" "${gradle}" "${gradle_args[@]}" ${@} || die "gradle failed"
-}
+# Fails to build if gradle is invoked with --parallel.
+EGRADLE_PARALLEL=false
 
 pkg_setup() {
JAVA_PKG_WANT_BUILD_VM="openjdk-${SLOT} openjdk-bin-${SLOT}"
@@ -148,7 +129,7 @@ pkg_setup() {
 
 src_unpack() {
unpack "${P}.tar.bz2"
-   unpack "gradle-${EGRADLE_VER}-bin.zip"
+   gradle-src_unpack
 
mkdir "${T}/jars" || die
 
@@ -171,7 +152,7 @@ src_prepare() {
java-pkg_jar-from --build-only --with-dependencies --into "${d}" 
stringtemplate
java-pkg_jar-from --build-only --with-dependencies --into "${d}" 
hamcrest-core
 
-   sed -i "s#__gentoo_swt_jar__#$(java-pkg_getjars swt-4.10)#" 
"${S}"/build.gradle || die
+   sed -i "s#__gentoo_swt_jar__#$(java-pkg_getjars swt-4.10)#" 
build.gradle || die
 }
 
 src_configure() {
@@ -189,7 +170,7 @@ src_configure() {
[[ -r ${jdk_doc}/element-list ]] || die "JDK Docs not found, 
terminating build early"
fi
 
-   cat <<- _EOF_ > "${S}"/gradle.properties
+   cat <<- _EOF_ > gradle.properties
COMPILE_TARGETS = linux
COMPILE_WEBKIT = false
COMPILE_MEDIA = $(usex media true false)
@@ -206,6 +187,9 @@ src_configure() {
 }
 
 src_compile() {
+   # FIXME: build.gradle believes $ANT_HOME/bin/ant shoud exist
+   unset ANT_HOME
+
egradle zips $(usex doc "" "--exclude-task javadoc")
 }
 
-- 
2.39.3

[gentoo-dev] [PATCH 1/2] gradle.eclass: add new eclass

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/gradle.eclass   | 208 +
 eclass/tests/gradle.sh |  62 
 2 files changed, 270 insertions(+)
 create mode 100644 eclass/gradle.eclass
 create mode 100755 eclass/tests/gradle.sh

diff --git a/eclass/gradle.eclass b/eclass/gradle.eclass
new file mode 100644
index ..91c8299d0c98
--- /dev/null
+++ b/eclass/gradle.eclass
@@ -0,0 +1,208 @@
+# Copyright 2021-2023 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: gradle.eclass
+# @MAINTAINER:
+# Gentoo Java Project 
+# @AUTHOR:
+# Florian Schmaus 
+# @BLURB: common ebuild functions for gradle-based packages.
+# @DESCRIPTION:
+# This eclass provides support for the gradle build system.  There
+# are currently two approaches to using gradle in ebuilds.  You can either
+# depend on a gradle system-wide installation from a gradle ebuild, typically
+# dev-java/gradle-bin, or, bundle gradle with the ebuild.
+#
+# To use a system-wide gradle installation, set EGRADLE_MIN and
+# EGRADLE_MAX_EXCLUSIVE and declare a BDEPEND on the gradle package.
+# @CODE
+# inherit gradle
+# EGRADLE_MIN=7.3
+# EGRADLE_MAX_EXCLUSIVE=8
+#
+# BDEPEND="|| (dev-java/gradle-bin:7.3 dev-java/gradle-bin:7.4)
+# @CODE
+#
+# To use a bundled gradle version, set EGRADLE_BUNDLED_VER and add
+# $(gradle_src_uri) to SRC_URI.
+# @CODE
+# inherit gradle
+# EGRADLE_BUNDLED_VER=7.6
+# SRC_URI="
+# ...
+# $(gradle_src_uri)
+# "
+# src_unpack() {
+#default
+#gradle-src_unpack
+# }
+# @CODE
+#
+# Afterwards, use egradle to invoke gradle.
+# @CODE
+# src_compile() {
+# egradle build
+# }
+# @CODE
+
+case ${EAPI} in
+   7|8) ;;
+   *) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
+esac
+
+if [[ -z ${_GRADLE_ECLASS} ]] ; then
+_GRADLE_ECLASS=1
+
+inherit edo
+
+# @ECLASS_VARIABLE: EGRADLE_MIN
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# Minimum required gradle version.
+
+# @ECLASS_VARIABLE: EGRADLE_MAX_EXCLUSIVE
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# First gradle version that is not supported.
+
+# @ECLASS_VARIABLE: EGRADLE_EXACT_VER
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# The exact required gradle version.
+
+# @ECLASS_VARIABLE: EGRADLE_BUNDLED_VER
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# The gradle version that will be bundled with this package.
+
+# @ECLASS_VARIABLE: EGRADLE_PARALLEL
+# @DESCRIPTION:
+# Set to the 'true', the default, to invoke gradle with --parallel. Set
+# to 'false' to disable parallel gradle builds.
+: "${EGRADLE_PARALLEL=true}"
+
+# @ECLASS_VARIABLE: EGRADLE_USER_HOME
+# @DESCRIPTION:
+# Directroy used as the user's home directory by gradle. Defaults to
+# ${T}/gradle_user_home
+: "${EGRADLE_USER_HOME="${T}/gradle_user_home"}"
+
+# @ECLASS_VARIABLE: EGRADLE_OVERWRITE
+# @USER_VARIABLE
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# User-specified overwrite of the used gradle binary.
+
+# @FUNCTION: gradle-set_EGRADLE
+# @DESCRIPTION:
+# Set the EGRADLE environment variable.
+gradle-set_EGRADLE() {
+   [[ -n ${EGRADLE} ]] && return
+
+   if [[ -n ${EGRADLE_OVERWRITE} ]]; then
+   EGRADLE="${EGRADLE_OVERWRITE}"
+   return
+   fi
+
+   if [[ -n ${EGRADLE_BUNDLED_VER} ]]; then
+   EGRADLE="${WORKDIR}/gradle-${EGRADLE_BUNDLED_VER}/bin/gradle"
+   return
+   fi
+
+   local candidate selected selected_ver ver
+
+   for candidate in "${BROOT}"/usr/bin/gradle-; do
+   if [[ ${candidate} != */gradle?(-bin)-+([.0-9]) ]]; then
+   continue
+   fi
+
+   ver=${candidate##*-}
+
+   if [[ -n ${EGRADLE_EXACT_VER} ]]; then
+   ver_test "${ver}" -ne "${EGRADLE_EXACT_VER}" && continue
+
+   selected="${candidate}"
+   break
+   fi
+
+   if [[ -n ${EGRADLE_MIN} ]] \
+  && ver_test "${ver}" -lt "${EGRADLE_MIN}"; then
+   # Candidate does not satisfy EGRADLE_MIN condition.
+   continue
+   fi
+
+   if [[ -n ${EGRADLE_MAX_EXCLUSIVE} ]] \
+  && ver_test "${ver}" -ge "${EGRADLE_MAX_EXCLUSIVE}"; 
then
+   # Candidate does not satisfy EGRADLE_MAX_EXCLUSIVE 
condition.
+   continue
+   fi
+
+   if [[ -n ${selected_ver} ]] \
+  && ver_test "${selected_ver}" -gt "${ver}"; then
+   # Candidate is older than the currently selected 
candidate.
+   continue
+   fi
+
+   selected="${candidate}"
+   selected_ver="${ver}"
+   done

[gentoo-dev] [PATCH 0/2] new gradle.eclass

[Florian Schmaus]

I would like to propose the gradle.eclass for ::gentoo.

Multiple people have shown interest in an eclass for Gradle, as it would
make it easier to move Gradle-based projects into ::gentoo. For exmaple,
ghidra from ::pentoo. And, as a nice bonus, the addition of the gradle
eclass to ::gentoo would make it easier for overlays to use it. This, in
turn, reduces the friction when migrating Gradle-based projects from
overlays into ::gentoo.

The second patch shows how gradle.eclass can be used in the openfjx ebuild.

PR at https://github.com/gentoo/gentoo/pull/28986


Florian Schmaus (2):
  gradle.eclass: add new eclass
  dev-java/openjfx: switch to gradle.eclass

 dev-java/openjfx/openjfx-11.0.11_p1.ebuild |  46 ++---
 eclass/gradle.eclass   | 208 +
 eclass/tests/gradle.sh |  62 ++
 3 files changed, 285 insertions(+), 31 deletions(-)
 create mode 100644 eclass/gradle.eclass
 create mode 100755 eclass/tests/gradle.sh

-- 
2.39.3

Re: [gentoo-dev] [PATCH] acct-user.eclass: improve error message when usermod fails

[Florian Schmaus]

On 24/06/2023 19.01, Mike Gilbert wrote:

usermod refuses to update the home directory for a user with running
processes. Output a more helpful message and avoid calling die for this.

For other usermod failures, output stderr as an eerror message and die.

Example output:

  * Failed to update user portage
  * This user currently has one or more running processes.
  * Please update this user manually with the following command:
  *   usermod '--comment' 'System user; portage' '--home' 
'/var/lib/portage/home' '--shell' '/bin/bash' '--gid' 'portage' '--groups' '' 
portage

Bug: https://bugs.gentoo.org/888189
Signed-off-by: Mike Gilbert 
---
  eclass/acct-user.eclass | 28 +++-
  1 file changed, 23 insertions(+), 5 deletions(-)

diff --git a/eclass/acct-user.eclass b/eclass/acct-user.eclass
index f8a51ebf9c6b..88d7d354c8e6 100644
--- a/eclass/acct-user.eclass
+++ b/eclass/acct-user.eclass
@@ -432,12 +432,30 @@ acct-user_pkg_postinst() {
fi
  
  	elog "Updating user ${ACCT_USER_NAME}"

-   if ! usermod "${opts[@]}" "${ACCT_USER_NAME}" 
2>"${T}/usermod-error.log"; then
-   # usermod outputs a warning if unlocking the account would 
result in an
-   # empty password. Hide stderr in a text file and display it if 
usermod
-   # fails.
+   # usermod outputs a warning if unlocking the account would result in an
+   # empty password. Hide stderr in a text file and display it if usermod 
fails.
+   usermod "${opts[@]}" "${ACCT_USER_NAME}" 2>"${T}/usermod-error.log"
+   local status=$?
+   if [[ ${status} -ne 0 ]]; then
cat "${T}/usermod-error.log" >&2
-   die "usermod failed"
+   if [[ ${status} -eq 8 ]]; then
+   # usermod refused to update the home directory
+   # for a uid with active processes.
+   eerror "Failed to update user ${ACCT_USER_NAME}"
+   eerror "This user currently has one or more running 
processes."
+   eerror "Please update this user manually with the following 
command:"
+
+   # Surround opts with quotes.
+   # With bash-5 (EAPI 8), we can use "${opts[@]@Q}" 
instead.
+   local q="'"
+   local optsq=( "${opts[@]/#/$q}" )
+   optsq=( "${optsq[@]/%/$q}" )
+
+   eerror "  usermod ${optsq[@]} ${ACCT_USER_NAME}"
+   else
+   eerror "$(<"${T}/usermod-error.log")"
+   die "usermod failed"


No strong opinion, but it is often a good idea to include the exit 
status. For example:


   die "usermod failed (${status})"

Otherwise, lgtm. Thanks!

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Eselect repository feature request

[Florian Schmaus]

On 21/06/2023 19.59, TOMAS FABRIZIO ORSI wrote:

I think flow's idea to make the sync command configurable somehow
would be sufficient, assuming there is demand for it.

I agree.
I'm glad Flow has their ideas straight.

By the way Flow, could you share the links of the repo/PR so that we can
follow the development along? I would love to see it! ^_^


https://github.com/projg2/eselect-repository/pull/23

Unlike I previously assumed, this currently shells out twice to portage:

1. to query the list of currently available repositories, via "portageq 
get_repos /"

2. to sync a repository, via "emaint sync -r ${repo}"

Feedback welcomed. :)

- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Eselect repository feature request

[Florian Schmaus]

On 21/06/2023 17.56, Mike Gilbert wrote:

On Wed, Jun 21, 2023 at 11:41 AM Florian Schmaus  wrote:


On 20.06.23 19:26, Mike Gilbert wrote:

On Tue, Jun 20, 2023 at 1:08 PM Florian Schmaus  wrote:


On 20.06.23 16:41, TOMAS FABRIZIO ORSI wrote:

  Isn't that duplicating the information of metadata/layout.conf's
  'master' key-value pair [1]?


Yes, I agree that it would be duplicating that information. As a matter
of fact, Michał Górny pointed the same thing out.
However, Michał also added, quote: "What's really lacking here is
support for specifying dependencies via |repositories.xml|


Do we need to duplicate the information in repositories.xml, with all
the drawbacks of duplication?

Can't eselect repository add the new repository, then read the 'masters'
value from layout.conf, and add the missing repositories recursively?


That would be a significant change in behavior for eselect repository.


Right, but it seems to be a desirable behaviour. Cases where the user
wants to add a repo but not immediately sync it are probably rare.

Furthermore, it would avoid duplicating the information, which avoids
the typical drawbacks of duplication (e.g., the two sets getting out of
sync).

I've looked at the eselect-repository code, and it seems not hard to
change the behaviour of "eselect repository add" to add and sync a
repository and then, recursively, add and sync further required
repositories.

I may give it a shot, but ideally I'd know if it has a chance to be
accepted upstream first. Or maybe there is a good reason why
eselect-repository behaves as it currently does that I am missing?


I can't speak for "upstream", but here are my concerns:

1. As a developer, I might just want to create the repos.conf config
snippet and sync the repo manually.
2. As a user, I might have any arbitrary reason for not wanting to
sync immediately.


Would an opt-out switch be enough to alleviate those concerns of you?



3. eselect-repository does not currently depend on any particular
package manager. It writes config files intended for Portage, but it
does not actually invoke any Portage commands. That feels like a
significant distinction to me.
4. If you start invoking Portage commands, you then have to deal with
the possibility of people using alternate package managers. pkgcore
can also utilize Portage's repos.conf, and the user might prefer to
use pmaint instead of emaint or emerge --sync.


Those two points seem to be based on the same fundamental concern.

The only portage specific code would be the call to "emaint sync -r 
$repo" (remember that "emerge --sync" is just a wrapper for "emaint sync 
--auto"). I think it would be easy to add later 1. add support for 
different package managers (if the need arises), and 2. make the "sync 
command" user configurable.


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Eselect repository feature request

[Florian Schmaus]

On 20.06.23 19:26, Mike Gilbert wrote:

On Tue, Jun 20, 2023 at 1:08 PM Florian Schmaus  wrote:


On 20.06.23 16:41, TOMAS FABRIZIO ORSI wrote:

 Isn't that duplicating the information of metadata/layout.conf's
 'master' key-value pair [1]?


Yes, I agree that it would be duplicating that information. As a matter
of fact, Michał Górny pointed the same thing out.
However, Michał also added, quote: "What's really lacking here is
support for specifying dependencies via |repositories.xml|


Do we need to duplicate the information in repositories.xml, with all
the drawbacks of duplication?

Can't eselect repository add the new repository, then read the 'masters'
value from layout.conf, and add the missing repositories recursively?


That would be a significant change in behavior for eselect repository.


Right, but it seems to be a desirable behaviour. Cases where the user 
wants to add a repo but not immediately sync it are probably rare.


Furthermore, it would avoid duplicating the information, which avoids 
the typical drawbacks of duplication (e.g., the two sets getting out of 
sync).


I've looked at the eselect-repository code, and it seems not hard to 
change the behaviour of "eselect repository add" to add and sync a 
repository and then, recursively, add and sync further required 
repositories.


I may give it a shot, but ideally I'd know if it has a chance to be 
accepted upstream first. Or maybe there is a good reason why 
eselect-repository behaves as it currently does that I am missing?


- Flow

Re: [gentoo-dev] Eselect repository feature request

[Florian Schmaus]

On 20.06.23 16:41, TOMAS FABRIZIO ORSI wrote:

Isn't that duplicating the information of metadata/layout.conf's
'master' key-value pair [1]?


Yes, I agree that it would be duplicating that information. As a matter 
of fact, Michał Górny pointed the same thing out.
However, Michał also added, quote: "What's really lacking here is 
support for specifying dependencies via |repositories.xml|


Do we need to duplicate the information in repositories.xml, with all 
the drawbacks of duplication?


Can't eselect repository add the new repository, then read the 'masters' 
value from layout.conf, and add the missing repositories recursively?


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Eselect repository feature request

[Florian Schmaus]

On 18.06.23 22:39, TOMAS FABRIZIO ORSI wrote:
Hello gentoo devs. The other day I made a feature suggestion to the 
eselect repository github page. (Here's the link: 
https://github.com/projg2/eselect-repository/issues/20 
).
Michał Górny suggested that I should contact the gentoo-dev mailing list 
with my suggestion, so here it is: My suggestion is to make it possible 
for eselect repository to manage overlay dependencies.


As it stands, and as far as I'm concerned, there is no "proper" way of 
having an ebuild dependency from another overlay. So overlay writers 
have to copy ebuilds from other overlays or rewrite them. This implies 
synchronization issues (where the "copied" ebuilds get out of sync from 
the original ebuild) and time loss (people writing the same ebuild more 
than once).


Michał Górny suggested that I should make an edit to the 
repositories.xml file in order to tackle the issue.


This is my general idea:
(I am using the github template as an example, but the idea should apply 
to all other templates. I got this file from 
https://github.com/gentoo/api-gentoo-org/blob/master/files/overlays/repositories.xml ).


   * GITHUB TEMPLATE
     
       X
       XX
       https://github.com// 


       
         X
         X
       
       https://github.com//.git 

       git+ssh://g...@github.com//.git 

       https://github.com///commits/master.atom 


       
           overlayName
       
     


Isn't that duplicating the information of metadata/layout.conf's 
'master' key-value pair [1]?


- Flow

1: 
https://wiki.gentoo.org/wiki/Repository_format/metadata/layout.conf#masters


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 2/2] kernel-build.eclass: add USE="modules-sign"

[Florian Schmaus]

On 15.06.23 11:50, Andrew Ammerlaan wrote:

 From fc8894ff62b45cc7a4148a9f6ba51f1afe7b920a Mon Sep 17 00:00:00 2001
From: Andrew Ammerlaan 
Date: Thu, 8 Jun 2023 20:44:58 +0200
Subject: [PATCH] sys-kernel/gentoo-kernel: add USE="modules-sign"

- Enable module signing configure options if requested by the user.
- Respect the linux-mod-r1.eclass variables MODULES_SIGN_HASH and 
MODULES_SIGN_KEY,

- Warn the user if we are letting the kernel build system generate
the signing key. This key will end up binary packages. Plus external 
modules will have to be resigned if gentoo-kernel is re-emerged (i.e. a 
new key was generated).


Bug: https://bugs.gentoo.org/881651
Bug: https://bugs.gentoo.org/814344
Signed-off-by: Andrew Ammerlaan 
---
  ...8.ebuild => gentoo-kernel-6.3.8.ebuild} | 45 ++-
  1 file changed, 44 insertions(+), 1 deletion(-)
  rename sys-kernel/gentoo-kernel/{gentoo-kernel-6.3.8.ebuild => 
gentoo-kernel-6.3.8-r1.ebuild} (71%)


diff --git a/sys-kernel/gentoo-kernel/gentoo-kernel-6.3.8.ebuild 
b/sys-kernel/gentoo-kernel/gentoo-kernel-6.3.8-r1.ebuild

similarity index 71%
rename from sys-kernel/gentoo-kernel/gentoo-kernel-6.3.8.ebuild
rename to sys-kernel/gentoo-kernel/gentoo-kernel-6.3.8-r1.ebuild
index fd81855a6140a..4bc03564efbe0 100644
--- a/sys-kernel/gentoo-kernel/gentoo-kernel-6.3.8.ebuild
+++ b/sys-kernel/gentoo-kernel/gentoo-kernel-6.3.8-r1.ebuild
@@ -44,7 +44,7 @@ S=${WORKDIR}/${MY_P}

  LICENSE="GPL-2"
  KEYWORDS="~amd64 ~arm ~arm64 ~hppa ~ppc ~ppc64 ~riscv ~x86"
-IUSE="debug hardened"
+IUSE="debug hardened modules-sign"
  REQUIRED_USE="arm? ( savedconfig )
  hppa? ( savedconfig )
  riscv? ( savedconfig )"
@@ -136,5 +136,48 @@ src_prepare() {
  merge_configs+=( "${dist_conf_path}/big-endian.config" )
  fi

+    if use modules-sign; then
+    : "${MODULES_SIGN_HASH:=sha512}"
+    cat <<-EOF > "${WORKDIR}/modules-sign.config" || die
+    ## Enable module signing
+    CONFIG_MODULE_SIG=y
+    CONFIG_MODULE_SIG_ALL=y
+    CONFIG_MODULE_SIG_FORCE=y
+    CONFIG_MODULE_SIG_${MODULES_SIGN_HASH^^}=y
+    EOF
+    if [[ -n "${MODULES_SIGN_KEY}" ]]; then
+    if [[ -e "${MODULES_SIGN_KEY}" ]]; then
+    echo "CONFIG_MODULE_SIG_KEY=\"${MODULES_SIGN_KEY}\"" \
+    >> "${WORKDIR}/modules-sign.config"
+    else
+    die "MODULES_SIGN_KEY=${MODULES_SIGN_KEY} not found!"
+    fi
+    fi
+    merge_configs+=( "${WORKDIR}/modules-sign.config" )
+    fi
+
  kernel-build_merge_configs "${merge_configs[@]}"
  }
+
+pkg_postinst() {
+    kernel-build_pkg_postinst
+    if use modules-sign; then
+    if [[ -z "${MODULES_SIGN_KEY}" ]]; then
+    ewarn ""


You can drop the empty string argument and simply just write ewarn.

And I am not sure if we really need a leading empty ewarn line, but this 
appears to be a common idiom.


+    ewarn "MODULES_SIGN_KEY was not set, this means the kernel 
build system"
+    ewarn "automatically generated the signing key. This key 
was installed"
+    ewarn "in 
${EROOT}/usr/src/linux-${PV}${KV_LOCALVERSION}/certs"

+    ewarn "and will also be included in any binary packages."
+    ewarn "Please take appropriate action to protect the key!"
+    ewarn ""
+    ewarn "Recompiling this package causes a new key to be 
generated. As"
+    ewarn "a result any external kernel modules will need to be 
resigned."
+    ewarn "Use emerge @module-rebuild, or manually sign the 
modules as"

+    ewarn "described on the wiki [1]"
+    ewarn ""
+    ewarn "Consider using the MODULES_SIGN_KEY variable to use 
an external key."

+    ewarn ""
+    ewarn "[1]: 
https://wiki.gentoo.org/wiki/Signed_kernel_module_support;

+    fi
+    fi
+}


- Flow

Re: [gentoo-dev] [PATCH] go-module.eclass: inline _go-module_gomod_encode()

[Florian Schmaus]

On 12/06/2023 16.12, Sam James wrote:


Florian Schmaus  writes:


The only call site of _go-module_gomod_encode() was using $() in a loop
over EGO_SUM. This caused bash to fork() for every loop iteration, which
significantly affected the time it takes to "source" an ebuild using
EGO_SUM.

For example, "pkg pkg source =sys-cluster/k3s-1.23.3_p1" previously took
2.4 seconds. Inlining _go-module_gomod_encode() reduces this to
236 milliseconds.

This also adds missing 'local' declarations for some variables.


Nice one & lgtm, thanks!


Thanks. :)

Hopefully it gets applied soon.



But please remember to CC eclass maintainers (done now). If you did and
I missed it, apologies.


Right, I resend the mail with only williamw in 'to' right after I send 
the first mail after I noticed that I did not include the eclass 
maintainer. And then I wondered if a get_maintainer script would be a 
nice thing for ::gentoo. But that's a different discussion :)


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] [PATCH] go-module.eclass: inline _go-module_gomod_encode()

[Florian Schmaus]

The only call site of _go-module_gomod_encode() was using $() in a loop
over EGO_SUM. This caused bash to fork() for every loop iteration, which
significantly affected the time it takes to "source" an ebuild using
EGO_SUM.

For example, "pkg pkg source =sys-cluster/k3s-1.23.3_p1" previously took
2.4 seconds. Inlining _go-module_gomod_encode() reduces this to
236 milliseconds.

This also adds missing 'local' declarations for some variables.

Signed-off-by: Florian Schmaus 
---
 eclass/go-module.eclass | 44 +++--
 1 file changed, 16 insertions(+), 28 deletions(-)

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
index f97b69f591c8..6c58d7f26f07 100644
--- a/eclass/go-module.eclass
+++ b/eclass/go-module.eclass
@@ -262,7 +262,22 @@ go-module_set_globals() {
continue
fi
 
-   _dir=$(_go-module_gomod_encode "${module}")
+   # Encode the name(path) of a Golang module in the format 
expected by Goproxy.
+   # Upper letters are replaced by their lowercase version with a 
'!' prefix.
+   # The transformed result of 'module' is stored in the '_dir' 
variable.
+   #
+   ## Python:
+   # return re.sub('([A-Z]{1})', r'!\1', s).lower()
+   ## Sed:
+   ## This uses GNU Sed extension \l to downcase the match
+   # echo "${module}" |sed 's,[A-Z],!\l&,g'
+   local re _dir lower
+   _dir="${module}"
+   re='(.*)([A-Z])(.*)'
+   while [[ ${_dir} =~ ${re} ]]; do
+   lower='!'"${BASH_REMATCH[2],}"
+   _dir="${BASH_REMATCH[1]}${lower}${BASH_REMATCH[3]}"
+   done
 
for _ext in "${exts[@]}" ; do
# Relative URI within a GOPROXY for a file
@@ -496,33 +511,6 @@ go-module_live_vendor() {
popd >& /dev/null || die
 }
 
-# @FUNCTION: _go-module_gomod_encode
-# @DEPRECATED: none
-# @DESCRIPTION:
-# Encode the name(path) of a Golang module in the format expected by Goproxy.
-#
-# Upper letters are replaced by their lowercase version with a '!' prefix.
-#
-_go-module_gomod_encode() {
-   ## Python:
-   # return re.sub('([A-Z]{1})', r'!\1', s).lower()
-
-   ## Sed:
-   ## This uses GNU Sed extension \l to downcase the match
-   #echo "${module}" |sed 's,[A-Z],!\l&,g'
-   #
-   # Bash variant:
-   debug-print-function "${FUNCNAME}" "$@"
-   #local re input lower
-   re='(.*)([A-Z])(.*)'
-   input="${1}"
-   while [[ ${input} =~ ${re} ]]; do
-   lower='!'"${BASH_REMATCH[2],}"
-   input="${BASH_REMATCH[1]}${lower}${BASH_REMATCH[3]}"
-   done
-   echo "${input}"
-}
-
 fi
 
 if [[ ! ${GO_OPTIONAL} ]]; then
-- 
2.39.3

Re: [gentoo-dev] EGO_SUM

[Florian Schmaus]

On 01/06/2023 21.55, William Hubbs wrote:

The EGO_SUM alternatives
- do not have the same level of trust and therefore have a negative
impact on security (a dubious tarball someone put somewhere, especially
when proxy-maint)


For this, I would argue that vetting the tarball falls to the developer
who is proxying. If you don't trust the proxy maintainer you
are pushing for, it is easy to make a dependency tarball yourself and
add it to your dev space.


- are not easily verifiable


I don't have a response to this other than to say that go does its
own verification of modules with the dependency tarballs that it can't
do with vendor tarballs.


Yes, go has "go mod verify", which was added to the go-mod eclass after 
I asked on 2022-10-21 in #gentoo-dev if the eclass verifies the 
dependency tarball. robbat2 was so kind to provide a proof of concept of 
the security issue I was pointing out, which is available under 
https://gist.github.com/robbat2/82f4c208b6674e707081eda689096d55. This 
demonstration of the issue triggered 
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=733b4944c1a061269f96219cc96530f89d8f439e, 
which made the go-module.eclass run "go mod verify".


Unfortunately, a malicious contributor can trivially sidestep this 
verification step, rendering it ineffective. First, neither portage [1] 
nor PMS require that a later (source) archive can not override an 
existing file. This looseness allows, for example, the (non-upstream) 
dependency tarball, to override (upstream's) go.sum. Secondly, a 
dependency tarball could create the vendor/ directory, preventing the 
condition under which the go-module.eclass runs "go mod verify". Both 
approaches allow the dependency tarball to inject malicious code. With 
the first approach, "go mod verify" completes successfully; with the 
second, "go mod verify" is simply not invoked.


The verification, as is, is ineffective.



Last but not least, we have the same situation in the Rust ecosystem,
but we allow the EGO_SUM "equivalent" there.


I'm not sure it is quite the same because Rust projects tend to have
much smaller numbers of dependencies.


I am curious to know of any specific reason why Rust projects generally 
get by with fewer dependencies. This impression may be deceiving, caused 
by the fact that the Go-lang ecosystem hosts several projects with a 
more significant number of dependencies. If you look at the analysis 
[2], you find that under the top 10 Go packages by EGO_SUM entry count 
are cri-o, prometheus, k3s, and k3d, among others. If someone rewrites 
any of those in Rust, they would probably end up with the same number of 
dependencies.




Another thing to consider is that using EGO_SUM adds a significant
amount of processing to the go-module eclass.
I was advised recently that this isn't a good idea since bash is
slow, so I am considering moving most of that processing into
get-ego-vendor by having it generate the contents of SRC_URI directly
instead of using the eclass code to do that.


Was this analyzed and quantified? Is this hurting us? The cache 
regeneration of an ebuild tree is an embarrassingly parallel operation, 
so this would need to be exponentially complex [3] to be of any 
significance.


It may be possible to tune the existing EGO_SUM handling. We should keep 
EGO_SUM if viable, as it directly maps Go's go.sum and makes developing 
Go-lang ebuilds as frictionless as possible.


- Flow



1: https://github.com/gentoo/portage/pull/1030
2: 
https://dev.gentoo.org/~flow/gentoo-tree-analysis-results/2023-05-17T100838-gentoo-at-2022-02-16-60dc7a03ff2f/post-processed-ego-sum.txt
3: something similar to what was recently found in the latex ebuilds, 
see 
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6ee282f0645dcfccf1836b9cc7ae6629eb8b

Re: [gentoo-dev] Re: EGO_SUM

[Florian Schmaus]

On 02/06/2023 10.31, Michał Górny wrote:

On Fri, 2023-06-02 at 10:17 +0200, Florian Schmaus wrote:

On 30/05/2023 18.35, Arthur Zamarin wrote:

On 30/05/2023 18.52, Florian Schmaus wrote:

To prevent harm from Gentoo, we should reach an agreement that everyone
can live with. To achieve a consensus, and since I can not rule out that
I missed a post that includes specific numbers, please share your ideas
on how EGO_SUM could be reinstated in ::gentoo by replying to this mail.


I still want to ask why in ::gentoo should it be enabled? I'm trying to
understand why?


In short: Auditability
[…]
A Gentoo developer, Gentoo user, or, anyone can look at the ebuild and
immediately tell that it will likely not inject malicious code into the
resulting binary image. Furthermore, the only input is from upstream,
and while you may not look at every line of source code, you assign a
certain trust level to upstream and probably assume that the input is
also likely non-malicious.



This reasoning is seriously flawed.  A "typical" EGO_SUM ebuilds
contains dozens to hundreds of different packages from dozens of
different authors.  You can't seriously expect anyone to be able to
reasonably establish trust to all of them.


I am sorry. I was unable to get my point across.

The security impact is unrelated to what you describe. You always have a 
certain degree of trust in upstream. Regardless if upstream is consumed 
by 100 Gentoo packages or if there are 100 entries in EGO_SUM.


The point was and is about *non-upstream input* in the ebuild. While 
EGO_SUM fetches its artifacts from upstream, a dependency tarball does 
typically not originate from upstream.


Even if we would not trust EGO_SUM upstream, consuming inputs via 
EGO_SUM would still be better from a security perspective because 
EGO_SUM upstream is consumed by Gentoo and all of Go's ecosystem. Hence, 
if something gets compromised, it will likely be detected quickly. 
Compared to dependency tarballs, which are usually only consumed by Gentoo.



> In the end, gentoo.git security model is entirely reliant
> on the developer verifying the final product and signing on it.
> Everything else is untrustworthy noise.

How do you verify the output, that is, final product? This is hard, for 
example, reproducible builds are far from trivial to achieve.


On the other hand, ensuring that the input matches what upstream 
provides and expects is far more manageable.


- Flow


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: EGO_SUM

[Florian Schmaus]

Hi Arthur,

thanks for your mail.

On 30/05/2023 18.35, Arthur Zamarin wrote:

On 30/05/2023 18.52, Florian Schmaus wrote:

To prevent harm from Gentoo, we should reach an agreement that everyone
can live with. To achieve a consensus, and since I can not rule out that
I missed a post that includes specific numbers, please share your ideas
on how EGO_SUM could be reinstated in ::gentoo by replying to this mail.


I still want to ask why in ::gentoo should it be enabled? I'm trying to
understand why? 


In short: Auditability

Let me try to explain with a simplified example.

Gentoo's ebuilds contain the instructions to transform source code 
(input) via a compilation process (transformation) into a binary image 
(output).


A pseudo-example ebuild may contain the following

foo-1.0.ebuild:
```
# Input
SRC_URI="https://foo-soft.org/foo/1.0/foo-1.0.tar.gz;

# Transformation
src_compile() {
emake foo
}

# Output into imagedir $D
src_install() {
emake DESTDIR="${D}" foo-install
}
```

A Gentoo developer, Gentoo user, or, anyone can look at the ebuild and 
immediately tell that it will likely not inject malicious code into the 
resulting binary image. Furthermore, the only input is from upstream, 
and while you may not look at every line of source code, you assign a 
certain trust level to upstream and probably assume that the input is 
also likely non-malicious.


That changes fundamentally with dependency tarballs. Now you have

foo-1.0.ebuild:
```
# Input
SRC_URI="
https://foo-soft.org/foo/1.0/foo-1.0.tar.gz
https://some-random.dude/on/the/internet/foo-1.0-deps.tar.gz
"

# Transformation
src_compile() {
emake foo
}

# Output into imagedir $D
src_install() {
emake DESTDIR="${D}" foo-install
}
```

Now you need to look into foo-1.0-deps.tar.gz if you want the keep the 
level of trust as before. And here, "look into foo-1.0-deps.tar.gz" 
means to ideally apply the same steps the creator of the tarball 
supposedly did and compare your foo-1.0-deps.tar.gz tarball with the one 
from the ebuild. To make matters worse, you can not simply compare the 
two tarballs bytewise, but you have to compare the archives for 
structural identity.


In the case of ::gentoo, this is especially problematic for 
proxy-maintained packages. See 
https://github.com/gentoo/gentoo/pull/27050 for an actual example.


Assuming that every developer will accurately audit the non-upstream 
inputs, a proxied maintainer provides, creates considerable wiggle room 
for a highly security-sensitive matter. And even if we would establish a 
firm policy, we still would need the tools to verify the non-upstream 
inputs (which we do not have currently). Furthermore, Gentoo lacks 
manpower, not only in the proxy-maint project, and verifying 
non-upstream inputs introduces additional effort maintaining ::gentoo.


Last but not least, this also affects non-proxied packages in ::gentoo.

Even if every one of my fellow Gentoo developers is trustworthy, the 
fact that most ebuilds are easily auditable by simply looking at them is 
a huge advantage. Of course, some ebuilds pull in a lot of third-party 
patches (Xen, for example), which makes it hard to verify those. But not 
having EGO_SUM means that *all Go-packages* are immediately more 
challenging to verify because of the non-upstream input that the 
dependency tarball presents. Regardless if a Gentoo developer created 
the tarball or not.




Also please remember the issue of scale. Look at the amount of packages
under dev-python. There are a lot of tools written in Go.


We currently have around 250 Go-packages in ::gentoo and dev-python/* 
alone contains 1600 packages. So the package-count numbers of the two 
programming languages are not yet comparable. But note that I suggested 
to review the EGO_SUM policy once the number of Go packages has doubled 
or in two years (whatever comes first) in my previous mail.


- Flow



OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] Re: EGO_SUM

[Florian Schmaus]

On 24/04/2023 18.11, Florian Schmaus wrote:
I like to ask the Gentoo council to vote on whether EGO_SUM should be 
reinstated ("un-deprecated") or not.


I am thankful that the council considered my request to vote on the 
topic. However, the council decided not to vote on this in its last 
session and to return the issue to the mailing lists.


Some see the requirement of some limitations as necessity it comes to 
reinstating EGO_SUM. Unfortunately, I could not see specific numbers 
mentioned since June 2022 in the three EGO_SUM threads [1, 2, 3] I am 
aware of.


To prevent harm from Gentoo, we should reach an agreement that everyone 
can live with. To achieve a consensus, and since I can not rule out that 
I missed a post that includes specific numbers, please share your ideas 
on how EGO_SUM could be reinstated in ::gentoo by replying to this mail.


Having EGO_SUM would significantly increase the security of Gentoo's 
users (amongst other benefits).


Personally, I do not see that we currently need any form of limitation 
to reinstate EGO_SUM. I substantiated this with data based on a two-year 
history analysis of gentoo.git. The summary is that the

- size increase of ::gentoo is unproblematic for users
- additional sync delta of ::gentoo is unproblematic for users
- higher rate of gentoo.git's increase is unproblematic for developers
when we reinstate EGO_SUM in ::gentoo.

Therefore, we could (and IMHO should) simply un-deprecate EGO_SUM. 
However, I would review this decision once the number of Go packages has 
doubled or in two years (whatever comes first).


Many share the concerns of an EGO_SUM-less world. I know that some seek 
a compromise by reinstating EGO_SUM with some limitations. The ::gentoo 
repository is able to handle packages (at least) up to the range of 2 to 
1.5 MiB total package-directory size. Therefore I propose a limit in 
that range.


- Flow


1: https://www.mail-archive.com/gentoo-dev@lists.gentoo.org/msg95175.html
2: https://www.mail-archive.com/gentoo-dev@lists.gentoo.org/msg95279.html
3: https://www.mail-archive.com/gentoo-dev@lists.gentoo.org/msg97310.html


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: EGO_SUM

[Florian Schmaus]

On 08/05/2023 14.03, Michał Górny wrote:

On Mon, 2023-05-08 at 09:53 +0200, Florian Schmaus wrote:

Furthermore, both numbers, 256 MiB and 410 MiB, are based on the
over-approximation that every EGO_SUM package uses 1.6 MiB, which is
almost certainly not the case. The mean package-directory size of a
EGO_SUM using package at 2022-02-16 was 280 KiB.


Please extend this analysis to Manifest changes over time, and how they
are going to impact total gentoo.git size.


Gladly.

The average daily change caused by Manifests of EGO_SUM packages from 
2020-02-16 to 2022-02-16 was at most 80 KiB. (See below for the 
methodology used to obtain this number.)


In other words, a daily syncing user had at most 80 KiB traffic on 
average per day to sync the Manifests of all EGO_SUM that existed on 
2022-02-16.


Even in lesser developed regions of the world, 80 KiB a day are 
manageable. And, this would still be the case if we double, quadruple or 
octuple this number.


I note that this number does not include ebuilds and metadata. However, 
one can easily over-approximate that the additional ebuilds and metadata 
delta, that comes with the observed Manifest changes, is smaller than 
the Manifest changes themselves. Therefore, a pessimistic approximation 
is twice 80 KiB.


But then again, the 80 KiB are not considering transport compression. 
And, as we have learned, Manifests roughly compress to 50% of their 
original size. So the average EGO_SUM-generated network traffic, 
assuming that it is compressed, remains in the region of hundred 
kilobytes per day.


We can also use this number to over-approximate the growth rate of 
gentoo.git due to EGO_SUM.


Assume that 120 EGO_SUM packages cause a daily growth rate of 160 KiB, 
that is 2x 80 KiB and the number we have used above. Doubling this 
number would yield the estimated rate of the current number of Go 
packages in ::gentoo. This rate amounts to 320 KiB daily, increasing 
gentoo.git by 114 MiB per year. Please double this number for a bit of 
future safety.


In summary, this and the previous analysis finds not data-size-based 
arguments against EGO_SUM's usage.


Using EGO_SUM is fine for users and developers. The ::gentoo increase, 
even if it would quadruple the current size, does not entail any issues. 
The expected average daily delta that EGO_SUM would cause today is also 
no threat, even for users with low-bandwidth connections. The size 
increase which EGO_SUM causes to gentoo.git is also within manageable 
bounds. If an ebuild developer has 1-2 gigabytes free on their disk, 
they will not need to buy a larger disk in the coming years if we start 
using EGO_SUM again in ::gentoo.


- Flow


# Appendix: Methodology

We took gentoo.git at 2022-02-16 at the commit 60dc7a03ff2f. From there, 
we created the numstat log (git log --numstat) of each Manifest of every 
EGO_SUM package. We configured the numstat log to go back at most two 
years in time, that is, till 2020-02-16. The numstat log contains the 
changed lines (added/removed) of the Manifest in the target period. An 
awk script calculated the total sum of added and removed lines. Note 
that this treats removed lines equal to added lines, even though the 
removed lines should cause significantly less network traffic. We also 
extracted the date of the oldest commit in the observed period. This 
date was used to calculate the total number of days in the period, which 
accounts for packages that came to life after 2020-02-16 and would 
otherwise skew the analysis towards smaller results.


Dividing the total number of changed lines by the number of days yields 
the average number of lines changed per day per package.


We further determined the worst-observed line length of EGO_SUM packages 
manifests, which was 404 bytes.


Summarizing the average number of lines changed over all packages 
yielded 195.58093724672614. Multiplying this number by the maximal 
observed line length of 404 bytes gives 79014.69 bytes per day or, in 
other words, roughly 80 KiB per day.


The raw and post-processed results of this analysis are available at

https://dev.gentoo.org/~flow/gentoo-tree-analysis-results/2023-05-17T100838-gentoo-at-2022-02-16-60dc7a03ff2f/

The code used to carry out this analysis is available at

https://gitlab.gentoo.org/flow/gentoo-tree-analysis

for everyone to study the code, reproduce the results, and check for 
issues and bugs.


As always, I appreciate any feedback.


OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: EGO_SUM

[Florian Schmaus]

On 02.05.23 21:45, Sam James wrote:

Florian Schmaus  writes:

On 27/04/2023 23.16, Sam James wrote:

Florian Schmaus  writes:

On 26/04/2023 18.12, Matt Turner wrote:

On Wed, Apr 26, 2023 at 11:31 AM Florian Schmaus  wrote:

The discussion would be more productive if someone who is supporting the
EGO_SUM deprecation could rationally summarize the main arguments why we
deprecated EGO_SUM.

You're requesting the changes. It's on you to read the previous
threads and try to understand. It's not others' responsibilities to
justify the status quo to you, but tl;dr is Manifest files grew to
insane sizes for golang packages with many dependencies, and the
Manifest size is a cost all Gentoo users pay regardless of whether
they use the package.


I am sorry. I did try to understand the reasoning in the previous
threads. However, I do not conclude that the "cost" users must pay for
EGO_SUM justifies EGO_SUM's deprecation. It is the other way around:
EGO_SUM's advantages do not explain its deprecation, even if users
have to pay a cost.

You write that the "Manifest sizes grew to insane sizes"?

At which boundary does a package size, the total size of the package's
directory, become insane?

Disk space is cheap. Currently, ::gentoo, without metadata, is around
470 MiB. If you add 10 Go packages with a whopping 200 KiB each, then
this adds 2 MiB to that. I need someone to explain how this
constitutes an issue with disk space. Even if we add 100 Go packages,
probably roughly the number of Go packages we have in ::gentoo, then
those 20 MiB are not significant. Needless to say that the average
size of a Go package is less than the 200 KiB uses in this
calculation.

The numbers you've used here suggest you've missed some of the
big problematic cases from the past:
- https://bugs.gentoo.org/833478 (1.1MB manifest)
- https://bugs.gentoo.org/833477 (1.6MB manifest)


Thanks for pointing those bugs out.

But please allow me to clarify that I did not miss those "problematic"
cases from the past.


This kind of phrasing is the sort of thing which makes it seem like you
don't appreciate/acknowledge others' concerns.


I am genuinely sorry if my usage of "problematic" made it appear that I 
do not appreciate the other's concerns. Like most people on this mailing 
list, I appreciate everyone who cares about Gentoo and raises concerns.


I do, however, not share the concerns regarding EGO_SUM.

It is hard to share concerns based on rather abstract reasons—for 
example, the portrayal of EGO_SUM as unfair.


It would be easier to share concerns if somebody gave concrete reasons 
against EGO_SUM. For example, use cases that are no longer possible. Or 
developers or users who are restricted in their work by EGO_SUM in a 
relevant way.


But actual problems that currently speak against the use of EGO_SUM have 
not surfaced.




I said problematic because it was clearly beyond what your worst-case
estimates were, i.e. far more than what you were saying would be a
large amount for the purposes of calculations.


Using the term "worst-case", even if I put it in quotes, probably got 
people on the wrong track. I am sorry for that; my bad. It is, in 
general, impossible even to approximate the worst-case size-increase of 
::gentoo.


Our best chance is to use historical data to interpolate the future.

My back-of-the-envolope calculation was 256 Go-packages, with each 
having 1 MiB. An analysis of the three on 2022-02-16, at the commit 
right before Minikube and k3s were cleaned, showed that only five 
packages out of 120 had larger package-directory sizes than one MiB.


256 Go-packages is roughly the number of Go-packages we have right now. 
Assuming they all have a package-directory size of 1.6 MiB, the most 
extensive EGO_SUM package the analysis yielded so far, we end up with 
410 MiB.


The point you criticize was that a system able to handle the current 
size of ::gentoo would also be able to manage an additional 256 MiB. The 
point still stands if we exchange the 256 MiB with 410 MiB.


Furthermore, both numbers, 256 MiB and 410 MiB, are based on the 
over-approximation that every EGO_SUM package uses 1.6 MiB, which is 
almost certainly not the case. The mean package-directory size of a 
EGO_SUM using package at 2022-02-16 was 280 KiB.


- Flow