Re: [gentoo-dev] [PATCH 1/4] autotools.eclass: don't inject -I${SYSROOT} to aclocal

2022-01-19 Thread Sam James


> On 19 Jan 2022, at 06:35, Mike Frysinger  wrote:
> 
> On 17 Jan 2022 11:09, Sam James wrote:
>> When -I${SYSROOT} is injected, it'll override the default of -Im4, which
>> results in trying to install macros to ${SYSROOT} (a sandbox violation)
>> when they can't be found.
>> 
>> From aclocal(1):
>> ```
>>   -I DIR add directory to search list for .m4 files
>> 
>>   --install
>>  copy third-party files to the first -I directory
>> ```
>> 
>> The first directry is normally -Im4 if anything, whereas when injected
>> (when ${SYSROOT} is defined), it ends up being ${SYSROOT}, not m4 (so
>> we try to copy macros to somewhere outside of the build directory).
> 
> we should define the semantics we want and bring it upstream to get into
> automake.  although it seems like ACLOCAL_PATH might work well enough
> for us now to switch to that.
> 
> as a stop gap, it seems like the use of --install is pretty low ?  we're
> cross-compiling about ~2.5k packages in CrOS every day and never seen a
> failure here.  so the few packages which are running into troubles can
> workaround it by setting AT_SYS_M4DIR right ?

I've only seen it in the wild with:
- app-crypt/tpm2-tss (https://bugs.gentoo.org/756211 
<https://bugs.gentoo.org/756211>)
- another package which I hit during "normal" use but I'm afraid I can't
recall what. I suspect I hit it before Python grew a BDEPEND on autoconf-archive
so we're less likely to hit it now.

But I accept it's niche. See below though, I think we agree that AT_SYS_M4DIR /
system acdir should be satisfactory here.

I don't mind keeping the old logic for < EAPI 7 if that'll help you in CrOS 
though.

> 
>> In EAPI 7+, this is almost always the case! We don't generally expect
>> to find macros (particularly things like autoconf-archive) in ${SYSROOT}
>> because that's for DEPEND-class dependencies, then they end up being
>> copied in unnecessarily and wrongly.
> 
> i think this optimism is misplaced.  libraries often install m4 files
> which is precisely why this logic is in here.
> https://bugs.gentoo.org/677002#c10 <https://bugs.gentoo.org/677002#c10>
> 
> deleting this check will break things.  prob more than we're fixing.

Sorry, you're absolutely right here!

But I think it's addressed by the system-acdir commit that follows it up.

i.e. the commit message is totally wrong (and I'll fix this), but the change is 
correct
given we follow it up with seemingly a better way of handling
the original case.

Does that sound right?

Best,
sam



signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] [PATCH 8/8] flag-o-matic.eclass: allow -frecord-command-line

2022-01-18 Thread Sam James
In Clang, -frecord-gcc-switches does the same as this anyway.

Signed-off-by: Sam James 
---
 eclass/flag-o-matic.eclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/eclass/flag-o-matic.eclass b/eclass/flag-o-matic.eclass
index 062bd04e2e0bd..50caa401bacfb 100644
--- a/eclass/flag-o-matic.eclass
+++ b/eclass/flag-o-matic.eclass
@@ -80,6 +80,7 @@ _setup-allowed-flags() {
# Cosmetic/output related, see e.g. bug #830534
-fno-diagnostics-color '-fmessage-length=*'
-fno-ident -fpermissive -frecord-gcc-switches
+   -frecord-command-line
'-fdiagnostics*' '-fplugin*'
'-W*' -w
 
-- 
2.34.1




[gentoo-dev] [PATCH 7/8] flag-o-matic.eclass: allow -fstack-clash-protection, -fcf-protection=*

2022-01-18 Thread Sam James
-fstack-clash-protection suggested by Arfrever.

Reported-by: Arfrever Frehtes Taifersar Arahesis 
Signed-off-by: Sam James 
---
 eclass/flag-o-matic.eclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/eclass/flag-o-matic.eclass b/eclass/flag-o-matic.eclass
index 7ac4f4a7791d1..062bd04e2e0bd 100644
--- a/eclass/flag-o-matic.eclass
+++ b/eclass/flag-o-matic.eclass
@@ -53,6 +53,8 @@ _setup-allowed-flags() {
# Hardening flags
'-fstack-protector*'
'-fstack-check*' -fno-stack-check
+   -fstack-clash-protection
+   '-fcf-protection=*'
-fbounds-check -fbounds-checking
-fno-PIE -fno-pie -nopie -no-pie
# Spectre mitigations, bug #646076
-- 
2.34.1




[gentoo-dev] [PATCH 6/8] flag-o-matic.eclass: allow -ffixed-x18 for arm64

2022-01-18 Thread Sam James
Needed for shadow stack bits on ARM64.

Closes: https://bugs.gentoo.org/800533
Thanks-to: Jannik Glückert 
Signed-off-by: Sam James 
---
 eclass/flag-o-matic.eclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/eclass/flag-o-matic.eclass b/eclass/flag-o-matic.eclass
index a2e37b89b6f08..7ac4f4a7791d1 100644
--- a/eclass/flag-o-matic.eclass
+++ b/eclass/flag-o-matic.eclass
@@ -109,6 +109,9 @@ _setup-allowed-flags() {
-mfix-rm7000 -mno-fix-rm7000 -mfix-r1 -mno-fix-r1
'-mr10k-cache-barrier=*' -mthumb -marm
 
+   # needed for arm64 (and in particular SCS)
+   -ffixed-x18
+
# gcc 4.5
-mno-fma4 -mno-movbe -mno-xop -mno-lwp
# gcc 4.6
-- 
2.34.1




[gentoo-dev] [PATCH 5/8] flag-o-matic.eclass: allow -glldb

2022-01-18 Thread Sam James
We already allow -ggdb for GDB and this is the analogue for LLDB.

Bug: https://bugs.gentoo.org/800533
Reported-by: Jannik Glückert 
Signed-off-by: Sam James 
---
 eclass/flag-o-matic.eclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/eclass/flag-o-matic.eclass b/eclass/flag-o-matic.eclass
index d6590a2e52dfd..a2e37b89b6f08 100644
--- a/eclass/flag-o-matic.eclass
+++ b/eclass/flag-o-matic.eclass
@@ -73,6 +73,7 @@ _setup-allowed-flags() {
-gdwarf '-gdwarf-*'
-gstabs -gstabs+
-gz
+   -glldb
 
# Cosmetic/output related, see e.g. bug #830534
-fno-diagnostics-color '-fmessage-length=*'
-- 
2.34.1




[gentoo-dev] [PATCH 4/8] flag-o-matic.eclass: allow Spectre mitigation flags

2022-01-18 Thread Sam James
Closes: https://bugs.gentoo.org/646076
Signed-off-by: Sam James 
---
 eclass/flag-o-matic.eclass | 5 +
 1 file changed, 5 insertions(+)

diff --git a/eclass/flag-o-matic.eclass b/eclass/flag-o-matic.eclass
index 37577209281a1..d6590a2e52dfd 100644
--- a/eclass/flag-o-matic.eclass
+++ b/eclass/flag-o-matic.eclass
@@ -55,6 +55,11 @@ _setup-allowed-flags() {
'-fstack-check*' -fno-stack-check
-fbounds-check -fbounds-checking
-fno-PIE -fno-pie -nopie -no-pie
+   # Spectre mitigations, bug #646076
+   '-mindirect-branch=*'
+   -mindirect-branch-register
+   '-mfunction-return=*'
+   -mretpoline
 
# Misc
-fno-unit-at-a-time -fno-strict-overflow
-- 
2.34.1




[gentoo-dev] [PATCH 3/8] flag-o-matic.eclass: restructure comments a bit

2022-01-18 Thread Sam James
No functional change.

Signed-off-by: Sam James 
---
 eclass/flag-o-matic.eclass | 15 +++
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/eclass/flag-o-matic.eclass b/eclass/flag-o-matic.eclass
index 38ad14d8f5fe8..37577209281a1 100644
--- a/eclass/flag-o-matic.eclass
+++ b/eclass/flag-o-matic.eclass
@@ -49,13 +49,20 @@ setup-allowed-flags() {
 _setup-allowed-flags() {
ALLOWED_FLAGS=(
-pipe -O '-O[12sg]' '-mcpu=*' '-march=*' '-mtune=*'
+
+   # Hardening flags
'-fstack-protector*'
-   '-fsanitize*' '-fno-sanitize*'
'-fstack-check*' -fno-stack-check
-   -fbounds-check -fbounds-checking -fno-strict-overflow
-   -fno-PIE -fno-pie -nopie -no-pie -fno-unit-at-a-time
+   -fbounds-check -fbounds-checking
+   -fno-PIE -fno-pie -nopie -no-pie
+
+   # Misc
+   -fno-unit-at-a-time -fno-strict-overflow
+
+   # Sanitizers
+   '-fsanitize*' '-fno-sanitize*'
 
-   # debugging symbols should generally be very safe to add
+   # Debugging symbols should generally be very safe to add
-g '-g[0-9]'
-ggdb '-ggdb[0-9]'
-gdwarf '-gdwarf-*'
-- 
2.34.1




[gentoo-dev] [PATCH 2/8] flag-o-matic.eclass: allow -fno-diagnostics-color -fmessage-length=0

2022-01-18 Thread Sam James
Both of these options are useful for automated reports and should
be harmless.

Closes: https://bugs.gentoo.org/830534
Reported-by: Agostino Sarubbo 
Signed-off-by: Sam James 
---
 eclass/flag-o-matic.eclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/eclass/flag-o-matic.eclass b/eclass/flag-o-matic.eclass
index f8181a17e911a..38ad14d8f5fe8 100644
--- a/eclass/flag-o-matic.eclass
+++ b/eclass/flag-o-matic.eclass
@@ -62,6 +62,8 @@ _setup-allowed-flags() {
-gstabs -gstabs+
-gz
 
+   # Cosmetic/output related, see e.g. bug #830534
+   -fno-diagnostics-color '-fmessage-length=*'
-fno-ident -fpermissive -frecord-gcc-switches
'-fdiagnostics*' '-fplugin*'
'-W*' -w
-- 
2.34.1




[gentoo-dev] [PATCH 1/8] flag-o-matic.eclass: strip-flags: Fix logic to properly support "=" in patterns.

2022-01-18 Thread Sam James
From: Arfrever Frehtes Taifersar Arahesis 

Signed-off-by: Arfrever Frehtes Taifersar Arahesis 
Signed-off-by: Sam James 
---
 eclass/flag-o-matic.eclass | 23 +++
 1 file changed, 11 insertions(+), 12 deletions(-)

diff --git a/eclass/flag-o-matic.eclass b/eclass/flag-o-matic.eclass
index 32119cb9a526f..f8181a17e911a 100644
--- a/eclass/flag-o-matic.eclass
+++ b/eclass/flag-o-matic.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2021 Gentoo Authors
+# Copyright 1999-2022 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: flag-o-matic.eclass
@@ -48,7 +48,7 @@ setup-allowed-flags() {
 # Note: shell globs and character lists are allowed
 _setup-allowed-flags() {
ALLOWED_FLAGS=(
-   -pipe -O '-O[12sg]' -mcpu -march -mtune
+   -pipe -O '-O[12sg]' '-mcpu=*' '-march=*' '-mtune=*'
'-fstack-protector*'
'-fsanitize*' '-fno-sanitize*'
'-fstack-check*' -fno-stack-check
@@ -70,7 +70,7 @@ _setup-allowed-flags() {
'-[DUILR]*' '-Wl,*'
 
# Linker choice flag
-   '-fuse-ld'
+   '-fuse-ld=*'
)
 
# allow a bunch of flags that negate features / control ABI
@@ -80,19 +80,19 @@ _setup-allowed-flags() {
-fno-omit-frame-pointer '-fno-builtin*'
)
ALLOWED_FLAGS+=(
-   -mregparm -mno-app-regs -mapp-regs -mno-mmx -mno-sse
+   '-mregparm=*' -mno-app-regs -mapp-regs -mno-mmx -mno-sse
-mno-sse2 -mno-sse3 -mno-ssse3 -mno-sse4 -mno-sse4.1 -mno-sse4.2
-mno-avx -mno-aes -mno-pclmul -mno-sse4a -mno-3dnow -mno-popcnt
-mno-abm -mips1 -mips2 -mips3 -mips4 -mips32 -mips64 -mips16 
-mplt
-   -msoft-float -mno-soft-float -mhard-float -mno-hard-float -mfpu
-   -mieee -mieee-with-inexact -mschedule -mfloat-gprs -mspe 
-mno-spe
+   -msoft-float -mno-soft-float -mhard-float -mno-hard-float 
'-mfpu=*'
+   -mieee -mieee-with-inexact '-mschedule=*' -mfloat-gprs -mspe 
-mno-spe
-mtls-direct-seg-refs -mno-tls-direct-seg-refs -mflat -mno-flat
-   -mno-faster-structs -mfaster-structs -m32 -m64 -mx32 -mabi
-   -mlittle-endian -mbig-endian -EL -EB -fPIC -mlive-g0 -mcmodel
-   -mstack-bias -mno-stack-bias -msecure-plt '-m*-toc' -mfloat-abi
+   -mno-faster-structs -mfaster-structs -m32 -m64 -mx32 '-mabi=*'
+   -mlittle-endian -mbig-endian -EL -EB -fPIC -mlive-g0 
'-mcmodel=*'
+   -mstack-bias -mno-stack-bias -msecure-plt '-m*-toc' 
'-mfloat-abi=*'
-mfix-r4000 -mno-fix-r4000 -mfix-r4400 -mno-fix-r4400
-mfix-rm7000 -mno-fix-rm7000 -mfix-r1 -mno-fix-r1
-   -mr10k-cache-barrier -mthumb -marm
+   '-mr10k-cache-barrier=*' -mthumb -marm
 
# gcc 4.5
-mno-fma4 -mno-movbe -mno-xop -mno-lwp
@@ -452,9 +452,8 @@ strip-flags() {
local new=()
 
for x in ${!var} ; do
-   local flag=${x%%=*}
for y in "${ALLOWED_FLAGS[@]}" ; do
-   if [[ -z ${flag%%${y}} ]] ; then
+   if [[ ${x} == ${y} ]] ; then
new+=( "${x}" )
break
fi
-- 
2.34.1




Re: [gentoo-dev] [RFC] making rust-bin ordered first in virtual/rust

2022-01-18 Thread Sam James


> On 17 Jan 2022, at 23:24, Georgy Yakovlev  wrote:
> 
> Hi,
> 
> I've been approached multiple times with that request, and a lot of
> time I see new users completely destroyed by rust build time and disk
> space requirements.
> 

I'll out myself as being one of these people!

> WDYT about switching order of rusts in a virtual?
> 
> RDEPEND="|| (
>~dev-lang/rust-${PV}
>~dev-lang/rust-bin-${PV}
> )"
> 
> 
> becomes
> 
> RDEPEND="|| (
>~dev-lang/rust-bin-${PV}
>   ~dev-lang/rust-${PV}
> )"
> 
> 
> Existing installs should be unaffected ofc.
> But portage may prefer to depclean rust and not rust-bin if both are
> present.
> Users who wish to use source version at all times can just add it to
> world file.
> 
> I see both positives and negatives of doing that, but would like to
> reach out to community first.

I'd like to do -bin first to be consistent with OpenJDK, IcedTea,
and to improve first-install experience.

Not that I'm advocating for removing source builds or anything
like that, but I dare say Rust is generally not something that people
can customise much anyway.

As Ionen has noted, we did already switch in desktop stages.

Best,
sam



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH 2/4] autotools.eclass: use --system-acdir for aclocal

2022-01-17 Thread Sam James


> On 17 Jan 2022, at 11:09, Sam James  wrote:
> 
> We need to instruct aclocal that it might find macros in both
> ${BROOT} _and_ ${SYSROOT}.
> 
> - A classic example within BROOT is autoconf-archive.
> 
> - A classic example within SYSROOT is, say, libogg. A fair amount of
>  codec software installs its own macro to help locating it (but this
>  is in no ways limited to that genre/area).
> 
>  The correct position for a dependency like libogg is DEPEND, and yet
>  the status quo doesn't mean that aclocal is obligated to check in ${ESYSROOT}
>  which is where DEPEND-class dependencies are guaranteed to be installed.
> 
>  We can't rely on these being in BDEPEND -- in fact, most of the time,
>  they won't be. If we wanted to rely on macros always being provided by
>  BDEPEND, we'd have to duplicate a considerable number of dependencies
>  in both BDEPEND + DEPEND, with the unnecessary cross-compilation that would
>  entail too: it makes far more sense to just tell aclocal to look in the
>  right place (an extra location).
> 
> Bug: https://bugs.gentoo.org/710792
> Closes: https://bugs.gentoo.org/677002
> Closes: https://bugs.gentoo.org/738918
> Thanks-to: David Michael  (for the suggestion)
> Thanks-to: James Le Cuirot  (rubberducking & sounding board)
> Signed-off-by: Sam James 
> ---
> eclass/autotools.eclass | 16 +++-
> 1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/eclass/autotools.eclass b/eclass/autotools.eclass
> index e2572290f0cbe..2cf7c076d01ed 100644
> --- a/eclass/autotools.eclass
> +++ b/eclass/autotools.eclass
> @@ -332,8 +332,22 @@ eaclocal_amflags() {
> # They also force installing the support files for safety.
> # Respects AT_M4DIR for additional directories to search for macros.
> eaclocal() {
> + # Feed in a list of paths:
> + # - ${BROOT}/usr/share/aclocal
> + # - ${ESYSROOT}/usr/share/aclocal
> + # See bug #677002
> + if [[ ! -f "${T}"/aclocal/dirlist ]] ; then
> + mkdir "${T}"/aclocal || die
> + cat <<- EOF > "${T}"/aclocal/dirlist || die
> + ${BROOT}/usr/share/aclocal
> + ${ESYSROOT}/usr/share/aclocal
> + EOF
> + fi
> +
> + local system_acdir=" --system-acdir=${T}/aclocal"
> +
>   [[ ! -f aclocal.m4 || -n $(grep -e 'generated.*by aclocal' aclocal.m4) 
> ]] && \
> - autotools_run_tool --at-m4flags aclocal "$@" $(eaclocal_amflags)
> + autotools_run_tool --at-m4flags aclocal "$@" 
> $(eaclocal_amflags) ${system_acdir}
> }

I've changed this locally to just skip the add-system-acdir-logic for older 
EAPIs. Forgot to amend the commit,
Chewi had already pointed this out privately.

best,
sam



signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] [PATCH 4/4] autotools.eclass: update for autoconf 2.71

2022-01-17 Thread Sam James
Closes: https://bugs.gentoo.org/827852
Signed-off-by: Sam James 
---
 eclass/autotools.eclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/eclass/autotools.eclass b/eclass/autotools.eclass
index 5250b28042ee2..5568cca505d78 100644
--- a/eclass/autotools.eclass
+++ b/eclass/autotools.eclass
@@ -95,7 +95,7 @@ if [[ -n ${WANT_AUTOCONF} ]] ; then
none)   _autoconf_atom="" ;; # some packages don't require 
autoconf at all
2.1)_autoconf_atom="~sys-devel/autoconf-2.13" ;;
# if you change the "latest" version here, change also 
autotools_env_setup
-   latest|2.5) _autoconf_atom=">=sys-devel/autoconf-2.69" ;;
+   latest|2.5) _autoconf_atom=">=sys-devel/autoconf-2.71" ;;
*)  die "Invalid WANT_AUTOCONF value 
'${WANT_AUTOCONF}'" ;;
esac
export WANT_AUTOCONF
@@ -524,7 +524,7 @@ autotools_env_setup() {
[[ ${WANT_AUTOMAKE} == "latest" ]] && \
die "Cannot find the latest automake! Tried 
${_LATEST_AUTOMAKE[*]}"
fi
-   [[ ${WANT_AUTOCONF} == "latest" ]] && export WANT_AUTOCONF=2.5
+   [[ ${WANT_AUTOCONF} == "latest" ]] && export WANT_AUTOCONF=2.71
 }
 
 # @FUNCTION: autotools_run_tool
-- 
2.34.1




[gentoo-dev] [PATCH 3/4] autotools.eclass: update for latest automake 1.16.4

2022-01-17 Thread Sam James
Signed-off-by: Sam James 
---
 eclass/autotools.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/autotools.eclass b/eclass/autotools.eclass
index 2cf7c076d01ed..5250b28042ee2 100644
--- a/eclass/autotools.eclass
+++ b/eclass/autotools.eclass
@@ -74,7 +74,7 @@ inherit gnuconfig libtool
 # Do NOT change this variable in your ebuilds!
 # If you want to force a newer minor version, you can specify the correct
 # WANT value by using a colon:  :
-_LATEST_AUTOMAKE=( 1.16.2-r1:1.16 )
+_LATEST_AUTOMAKE=( 1.16.4:1.16 )
 
 _automake_atom="sys-devel/automake"
 _autoconf_atom="sys-devel/autoconf"
-- 
2.34.1




[gentoo-dev] [PATCH 2/4] autotools.eclass: use --system-acdir for aclocal

2022-01-17 Thread Sam James
We need to instruct aclocal that it might find macros in both
${BROOT} _and_ ${SYSROOT}.

- A classic example within BROOT is autoconf-archive.

- A classic example within SYSROOT is, say, libogg. A fair amount of
  codec software installs its own macro to help locating it (but this
  is in no ways limited to that genre/area).

  The correct position for a dependency like libogg is DEPEND, and yet
  the status quo doesn't mean that aclocal is obligated to check in ${ESYSROOT}
  which is where DEPEND-class dependencies are guaranteed to be installed.

  We can't rely on these being in BDEPEND -- in fact, most of the time,
  they won't be. If we wanted to rely on macros always being provided by
  BDEPEND, we'd have to duplicate a considerable number of dependencies
  in both BDEPEND + DEPEND, with the unnecessary cross-compilation that would
  entail too: it makes far more sense to just tell aclocal to look in the
  right place (an extra location).

Bug: https://bugs.gentoo.org/710792
Closes: https://bugs.gentoo.org/677002
Closes: https://bugs.gentoo.org/738918
Thanks-to: David Michael  (for the suggestion)
Thanks-to: James Le Cuirot  (rubberducking & sounding board)
Signed-off-by: Sam James 
---
 eclass/autotools.eclass | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/eclass/autotools.eclass b/eclass/autotools.eclass
index e2572290f0cbe..2cf7c076d01ed 100644
--- a/eclass/autotools.eclass
+++ b/eclass/autotools.eclass
@@ -332,8 +332,22 @@ eaclocal_amflags() {
 # They also force installing the support files for safety.
 # Respects AT_M4DIR for additional directories to search for macros.
 eaclocal() {
+   # Feed in a list of paths:
+   # - ${BROOT}/usr/share/aclocal
+   # - ${ESYSROOT}/usr/share/aclocal
+   # See bug #677002
+   if [[ ! -f "${T}"/aclocal/dirlist ]] ; then
+   mkdir "${T}"/aclocal || die
+   cat <<- EOF > "${T}"/aclocal/dirlist || die
+   ${BROOT}/usr/share/aclocal
+   ${ESYSROOT}/usr/share/aclocal
+   EOF
+   fi
+
+   local system_acdir=" --system-acdir=${T}/aclocal"
+
[[ ! -f aclocal.m4 || -n $(grep -e 'generated.*by aclocal' aclocal.m4) 
]] && \
-   autotools_run_tool --at-m4flags aclocal "$@" $(eaclocal_amflags)
+   autotools_run_tool --at-m4flags aclocal "$@" 
$(eaclocal_amflags) ${system_acdir}
 }
 
 # @FUNCTION: _elibtoolize
-- 
2.34.1




[gentoo-dev] [PATCH 1/4] autotools.eclass: don't inject -I${SYSROOT} to aclocal

2022-01-17 Thread Sam James
When -I${SYSROOT} is injected, it'll override the default of -Im4, which
results in trying to install macros to ${SYSROOT} (a sandbox violation)
when they can't be found.

>From aclocal(1):
```
   -I DIR add directory to search list for .m4 files

   --install
  copy third-party files to the first -I directory
```

The first directry is normally -Im4 if anything, whereas when injected
(when ${SYSROOT} is defined), it ends up being ${SYSROOT}, not m4 (so
we try to copy macros to somewhere outside of the build directory).

In EAPI 7+, this is almost always the case! We don't generally expect
to find macros (particularly things like autoconf-archive) in ${SYSROOT}
because that's for DEPEND-class dependencies, then they end up being
copied in unnecessarily and wrongly.

We could drop this just for < EAPI 7, but I'm not sure there's much
point there either. As Chewi observed in bug 677002, you can't
assume they'll be present in ${SYSROOT} anyway, and frankly,
the cross-compilation (and --root, --sysroot, and so on) situation
is rather bleak for earlier EAPIs, which is why we did all that
work for 7.

Bug: https://bugs.gentoo.org/710792
Closes: https://bugs.gentoo.org/677002
Closes: https://bugs.gentoo.org/738918
Thanks-to: James Le Cuirot 
Signed-off-by: Sam James 
---
 eclass/autotools.eclass | 8 +---
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/eclass/autotools.eclass b/eclass/autotools.eclass
index 95c92cc6df8ca..e2572290f0cbe 100644
--- a/eclass/autotools.eclass
+++ b/eclass/autotools.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2021 Gentoo Authors
+# Copyright 1999-2022 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: autotools.eclass
@@ -666,12 +666,6 @@ autotools_m4sysdir_include() {
# First try to use the paths the system integrator has set up.
local paths=( $(eval echo ${AT_SYS_M4DIR}) )
 
-   if [[ ${#paths[@]} -eq 0 && -n ${SYSROOT} ]] ; then
-   # If they didn't give us anything, then default to the SYSROOT.
-   # This helps when cross-compiling.
-   local path="${SYSROOT}/usr/share/aclocal"
-   [[ -d ${path} ]] && paths+=( "${path}" )
-   fi
_autotools_m4dir_include "${paths[@]}"
 }
 
-- 
2.34.1




[gentoo-dev] Last rites: dev-db/oracle-instanclient-*

2022-01-15 Thread Sam James
# Marco Genasci  (2022-01-15)
# Removed in favor of unified package dev-db/oracle-instantclient
# Removal on 2022-02-15. Bug #589146
dev-db/oracle-instantclient-basic
dev-db/oracle-instantclient-jdbc
dev-db/oracle-instantclient-odbc
dev-db/oracle-instantclient-sqlplus


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] Maintainer needed: dev-db/sqlite

2022-01-14 Thread Sam James


> On 14 Jan 2022, at 23:10, Peter Stuge  wrote:
> 
> Mike Gilbert wrote:
>> The current (proxied) maintainer is somewhat difficult to work with
> 
> Why is Arfrever being treated so bad here? To me, it looks like
> you're the one who is difficult to work with. :\
> 

floppym is not obligated to work with somebody if he finds it
difficult.

> 
> Jakov Smolić wrote:
>> From what I've investigated, other major distributions don't apply
>> any similar patches which means that we are likely to stop carrying
>> most (or even all) of the current patches.
> 
> What kind of silly groupthink is this? I expect Gentoo to champion choice.

Adding in a huge heap of patches which exceed the tree limits, have
no justification within them, and nobody else needs is a good reason
to dump them.

If someone actually wants them, that's another matter.

Even if they are being kept, justification for them should be made
so that others know why we're doing it, why it's worth rebasing them,
why we're changing the default behaviour of SQLite, ...

(This is all worth doing anyway, but Gentoo, if we're going to do
tropes, also doesn't like to deviate from upstream without
justification.)

Please don't bring out this cliched "choice" trope just because
we're discussing something. Obviously if they're actually useful
in an application, we can talk about keeping them.

> 
> 
> Mike Gilbert wrote:
>> There is an open QA bug [1] regarding the large set of undocumented
>> patches that are being applied in the stable ebuilds.
> 
> Arfrever is active in the bug you linked, has provided explanations
> for the patches and prepared to restructure the patches so that they
> can be gated by local USE flags, has made several different concrete
> suggestions for possible implementations and requested feedback, but
> has received no reply in the bug and instead there's now this
> backstabbing discussion on this list.
> 

You've missed discussions on IRC and some of the bugs _have_
gone unanswered (in particular https://bugs.gentoo.org/825278 

which started this all off).

Also, we were waiting several months for new SQLite which
blocked security bumps for e.g. seamonkey.

He has also received replies on the bug.

> Really?
> 

You've intervened in something where you don't know all the
circumstances, including the history of the contributor,
with an aggressive tone. Really?

sam



signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Mailing list for ebuild patches? (Was: Re: [PATCH] media-libs/freetype: fix GCC usage during configure)

2022-01-07 Thread Sam James


> On 7 Jan 2022, at 13:08, Adrian Ratiu  wrote:
> 
> If $CC_BUILD is not set, configure defaults to GCC for some
> of its tests causing clang builds to use a mixture of the
> two compilers instead of using just clang consistently.
> [snip]

Thanks!

Looks like Polynomial-C applied this as 
https://github.com/gentoo/gentoo/commit/355c5b5715ffcf787c421d03209642d2823cf1f7
 
.

FWIW, normally we don't post individual package patches
to this ML, but it's a good question as to.. where they should go
if people want to use git send-email/a ML workflow.

Right now, sometimes people send them to gentoo-proxy-maint
(the list) which the proxy maintainers team that handles
most user contributions looks at, but I'll be honest and say
our workflow isn't really optimised for it given it's used
pretty infrequently.

Makes me wonder if we should rename the list
or have a separate one (gentoo-patches?).

(Or just use that list and make sure people CC
maintainers as you did here?)

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

2022-01-05 Thread Sam James


> On 5 Jan 2022, at 17:51, Alec Warner  wrote:
> On Tue, Jan 4, 2022 at 3:03 PM Sam James  wrote:
>> 
>> On 4 Jan 2022, at 22:58, Sam James  wrote:
>> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
>> amount of RAM available (uses amount declared as needed
>> in the ebuild). Typically should be ~2GB per job.
>> [snip]
> 
> I'm still not sure I grasp why we cannot make OOMs easier to discover
> from portage.
> 
> Most packages don't even use check-reqs, so your solution is very
> narrow (and I get why, because you get a lot of bug reports from the
> big packages that do use it.)
> 

Yeah, you've got the thinking here :)

> Can we write a build log analyzer?

I think this is a good idea, even if it's just a few hand-gathered regexes,
we can try improve the situation a bit, even for non-OOM (like
some common Perl-related failures).

> 
> -A
> 
> PS: If this was a global change I'd downvote it. It's only for
> check-reqs though and most packages don't use check-reqs; I don't
> really care. I'd be concerned about adopting this kind of approach
> wider; its very much a bandaid.
> 

Yep, I don't deny that, and I can't really say I'd support this
if it was global. The specific intent here is that check-reqs
consumers are generally big beasts and hence it's a direct,
targeted action to reduce bad bug reports and improve
the user experience, especially for newcomers who
end up hitting OOMs early on.

Best,
sam



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

2022-01-05 Thread Sam James


> On 5 Jan 2022, at 19:02, Roy Bamford  wrote:
> 
> Sam,
> 
> Do users with FEATURES=distcc still have to opt out of this
> MAKEOPTS clamping?
> 

Great point! I think we could add an exemption for that and make it a noop or 
warning-only.

Best,
sam



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

2022-01-05 Thread Sam James


> On 5 Jan 2022, at 19:18, Kai Krakow  wrote:
> 
> Am Mi., 5. Jan. 2022 um 19:22 Uhr schrieb Ulrich Mueller  >:
>> 
>>> [...]
>> That applies to all parallel builds though, not only to ebuilds
>> inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
>> telling the user that the --jobs setting in their make.conf is wrong,
>> in the first place.
> 
> Well, I'm using a safe combination of jobs and load-average, maybe the
> documentation should be tweaked instead.
> 

I think "safe" is doing some heavy lifting here...

> I'm using
> [...]

> 
> The "--jobs" parameter is mostly a safe-guard against "make" or
> "emerge" overshooting the system resources which would happen if
> running unconstrained without "--load-average". The latter parameter
> OTOH tunes the parallel building processes automatically to the
> available resources. If the system starves of memory, thus starts to
> swap, load will increase, and make will reduce the jobs. It works
> pretty well.
> 
> I've chosen the emerge loadavg limit slightly higher so a heavy ebuild
> won't starve emerge from running configure phases of parallel ebuilds.
> 

... because it's quite hard for this logic to work correctly enough
of the time without jobserver integration (https://bugs.gentoo.org/692576 
).

But indeed, I'd say you're not the target audience for this (but I appreciate
the input).

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

2022-01-05 Thread Sam James


> On 5 Jan 2022, at 08:28, Ulrich Mueller  wrote:
> 
>>>>>> On Tue, 04 Jan 2022, Sam James wrote:
> 
>> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
>> amount of RAM available (uses amount declared as needed
>> in the ebuild). Typically should be ~2GB per job.
> 
> Where does this number 2 GB come from? The amount of RAM strongly
> depends on the programming language and other factors, so I don't
> believe that there's one number that can be used for everything.
> (If only considering C and C++ 2 GB seems to be excessive, i.e. it
> will limit parallel build more than necessary. When linking, the number
> may be _much_ larger.)
> 

This is essentially "common law" (or "common lore" if you like!) and
is well-accepted as a good rule of thumb.

The number being larger doesn't really make any difference here;
the point is to help in cases where we're pretty sure there's going
to be a problem (only users of check-reqs.eclass, and we can
Introduce a variable to give ~RAM per job too if needed).

> Also not sure if I understand the arithmetic. Shouldn't it use
> CHECKREQS_MEMORY as the basis of the calculation?

As noted, I was sort of on the fence about whether to hardcode
or use the CHECKREQS_MEMORY value and I'll think about this
again. I went for the current approach partly to keep things simple
(to avoid needing min/max bits).

> 
>> +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
>> +# @USER_VARIABLE
>> +# @DESCRIPTION:
>> +# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) 
>> jobs
>> +# to lower memory usage.
>> +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
> 
> If anything, the feature should be opt-in rather than opt-out.

That would defeat the purpose. If it's opt-in, then people are already aware
of their settings being inappropriate, and should probably fix them.

This is also about reducing the number of support queries about
builds which failed for trivial reasons, as someone who has to handle
quite a lot of those (until such a time as we can implement better
detection, but this is also a generally nice UX improvement -- as
per what Florian/flow said).



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

2022-01-05 Thread Sam James


> On 5 Jan 2022, at 19:53, Ulrich Mueller  wrote:
> 
>> On Wed, 05 Jan 2022, Florian Schmaus wrote:
> 
>>> That applies to all parallel builds though, not only to ebuilds
>>> inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
>>> telling the user that the --jobs setting in their make.conf is wrong,
>>> in the first place.
> 
>> Yes, exactly. And it is a bandaid solution. But I believe that it will
>> do more good than evil. And it is probably only used until portage is
>> able to report to the user that the emerge failed due to OOM (which I
>> believe to be non-trivial to implement, but I am happy to be proven
>> otherwise).
> 
> Obviously I disagree. Tweaking the value for a subset of packages isn't
> a solution to the problem.
> 
> MAKEOPTS applies to all parallel builds, and users should set it to a
> value suitable for their system (i.e. number of CPUs, available memory,
> etc.). Maybe our documentation needs to be improved? I see that
> make.conf.example says "The suggested number for parallel makes is
> CPUs+1" which may not be the best possible advice.
> 

As you've noted, the memory requirements vary per package,
and it was somewhat of an oversight/error to not include
the memory limit from the eclass variable instead here
(although I recall kind of deciding to go with this approach
for simplicity).

It's therefore nontrivial to come up with a good value for their
whole system :)

Our documentation has mostly been updated but apparently
make.conf.example hasn't been.



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH v3] kernel-2.eclass: Respect portage toolchain variables

2022-01-04 Thread Sam James


> On 3 Jan 2022, at 18:23, Mike Gilbert  wrote:
> On Mon, Jan 3, 2022 at 12:49 PM Adrian Ratiu  > wrote:
>> 
>> Starting with kernel>=v5.7 the build system can override the
>> tools vars by setting LLVM=1 [1], but older kernels still use
>> the default GNU tools, so to be able to use a full LLVM/Clang
>> build, CC & co should be set to their respective portage values.
>> 
>> [1] a0d1c951ef08 kbuild: support LLVM=1 to switch the default tools to 
>> Clang/LLVM
>> 
>> Co-authored-by: Manoj Gupta 
>> Signed-off-by: Adrian Ratiu 
>> [snip]
> 
> This seems ok to me, at least given the way the eclass currently works.
> 
> At some point, we should really convert xmakeopts into an array. Any
> of these variables might contain spaces, and that would break the
> current implementation.


agreed, but lgtm


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

2022-01-04 Thread Sam James


> On 4 Jan 2022, at 22:58, Sam James  wrote:
> 
> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
> amount of RAM available (uses amount declared as needed
> in the ebuild). Typically should be ~2GB per job.
> 
> Bug: https://bugs.gentoo.org/570534
> Signed-off-by: Sam James 
> ---
> eclass/check-reqs.eclass | 42 +---
> 1 file changed, 39 insertions(+), 3 deletions(-)

Note that we discussed this on GitHub a bit when I just posted it there
for some rough feedback: https://github.com/gentoo/gentoo/pull/23311 
<https://github.com/gentoo/gentoo/pull/23311>.

I think this is valuable for reducing invalid bug reports from OOM and
easing user experience.

Still kind of a WIP/rough draft, but may be ready in this state. Need
more testing, so not planning on pushing yet or anything.


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

2022-01-04 Thread Sam James
Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.

Bug: https://bugs.gentoo.org/570534
Signed-off-by: Sam James 
---
 eclass/check-reqs.eclass | 42 +---
 1 file changed, 39 insertions(+), 3 deletions(-)

diff --git a/eclass/check-reqs.eclass b/eclass/check-reqs.eclass
index 2130e2e349141..8c4adc8b4f121 100644
--- a/eclass/check-reqs.eclass
+++ b/eclass/check-reqs.eclass
@@ -43,6 +43,8 @@ case ${EAPI} in
*) die "${ECLASS}: EAPI=${EAPI:-0} is not supported" ;;
 esac
 
+inherit multiprocessing
+
 EXPORT_FUNCTIONS pkg_pretend pkg_setup
 
 if [[ ! ${_CHECK_REQS_ECLASS} ]]; then
@@ -53,6 +55,13 @@ _CHECK_REQS_ECLASS=1
 # @DESCRIPTION:
 # How much RAM is needed? Eg.: CHECKREQS_MEMORY=15M
 
+# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
+# @USER_VARIABLE
+# @DESCRIPTION:
+# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) 
jobs
+# to lower memory usage.
+: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
+
 # @ECLASS-VARIABLE: CHECKREQS_DISK_BUILD
 # @DEFAULT_UNSET
 # @DESCRIPTION:
@@ -346,9 +355,36 @@ _check-reqs_memory() {
eend 0
else
eend 1
-   _check-reqs_unsatisfied \
-   ${size} \
-   "RAM"
+
+   # Has the user allowed us to mangle their MAKEOPTS?
+   if [[ ${CHECKREQS_MEMORY_MANGLE_JOBS} == "yes" ]] ; then
+   local jobs=$(makeopts_jobs)
+
+   local 
estimated_max_memory=$((${actual_memory}/$(_check-reqs_get_kibibytes 1G)))
+   if [[ $((jobs*2)) -gt ${estimated_max_memory} 
]] ; then
+   # Number of jobs exceeds RAM/2GB, so 
clamp it.
+   local 
new_jobs=$(($(_check-reqs_get_number ${estimated_max_memory}G)*10/20))
+
+   # This might _still_ be too big on 
small machines. Give up in such cases.
+   # (Users can still set the do nothing 
variable which is independent of this.)
+   if [[ $((new_jobs*2)) -gt 
${estimated_max_memory} ]] ; then
+   _check-reqs_unsatisfied \
+   ${size} \
+   "RAM"
+   else
+   # The clamped jobs seem to be 
enough to satisfy the check-reqs requirement from the ebuild.
+   ewarn "Clamping MAKEOPTS jobs 
to -j${new_jobs} to reduce memory usage"
+   ewarn "Compiler jobs may use 
around ~2GB each: https://wiki.gentoo.org/wiki/MAKEOPTS;
+   ewarn "To disable this, set 
CHECKREQS_MEMORY_MANGLE_JOBS=no."
+
+   MAKEOPTS+=" -j${new_jobs}"
+   fi
+   fi
+   else
+   _check-reqs_unsatisfied \
+   ${size} \
+   "RAM"
+   fi
fi
else
eend 1
-- 
2.34.1




Re: [gentoo-dev] [PATCH 1/2] linux-mod.eclass: drop unnecessary IUSE="kernel_linux"

2022-01-04 Thread Sam James


> On 4 Jan 2022, at 21:54, Michał Górny  wrote:
> 
> On Tue, 2022-01-04 at 11:17 -0500, Mike Gilbert wrote:
>> On Tue, Jan 4, 2022 at 5:23 AM Sam James  wrote:
>>> 
>>> It's already an implicit IUSE, so we don't need this.
>> 
>> I think it is better to declare it explicitly rather than relying on
>> the IUSE_IMPLICIT setting in profiles.
> 
> I agree.  Perhaps we should go even further and remove them from
> implicit flags.
> 

(Note: I did push this change already to coalesce this
with FreeBSD and other removals.)

I don't think declaring it explicitly is helpful given
in most cases (and even more now after recent
cleanups), it's relied upon without declaring it anyway.

Use within IUSE while it's implicit also means
IUSE seems to regularly get out of sync with
actual use within the ebuilds.

As for removing implicit use entirely: while you might
argue this could be okay for the Linux kernel, it would cause
unnecessary rebuilds indefinitely whenever we add a new dep
to an ebuild for e.g. Darwin, or if we did it for libcs,
for say, musl/non-glibc.

best,
sam


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] [PATCH 1/2] linux-mod.eclass: drop unnecessary IUSE="kernel_linux"

2022-01-04 Thread Sam James
It's already an implicit IUSE, so we don't need this.

Signed-off-by: Sam James 
---
 eclass/linux-mod.eclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/eclass/linux-mod.eclass b/eclass/linux-mod.eclass
index 9f4ae64f6b55..496b9c98b526 100644
--- a/eclass/linux-mod.eclass
+++ b/eclass/linux-mod.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2021 Gentoo Authors
+# Copyright 1999-2022 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: linux-mod.eclass
@@ -170,7 +170,7 @@ esac
0) die "EAPI=${EAPI} is not supported with 
MODULES_OPTIONAL_USE_IUSE_DEFAULT due to lack of IUSE defaults" ;;
 esac
 
-IUSE="kernel_linux dist-kernel
+IUSE="dist-kernel

${MODULES_OPTIONAL_USE:+${_modules_optional_use_iuse_default}}${MODULES_OPTIONAL_USE}"
 SLOT="0"
 RDEPEND="
-- 
2.34.1




[gentoo-dev] [PATCH 2/2] linux-info.eclass: drop unnecessary IUSE="kernel_linux"

2022-01-04 Thread Sam James
It's already an implicit IUSE, so we don't need this.

Signed-off-by: Sam James 
---
 eclass/linux-info.eclass | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/eclass/linux-info.eclass b/eclass/linux-info.eclass
index 568f7a1a2832..a0942f0e554c 100644
--- a/eclass/linux-info.eclass
+++ b/eclass/linux-info.eclass
@@ -1,4 +1,4 @@
-# Copyright 1999-2021 Gentoo Authors
+# Copyright 1999-2022 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 # @ECLASS: linux-info.eclass
@@ -137,8 +137,6 @@ inherit toolchain-funcs
 
 EXPORT_FUNCTIONS pkg_setup
 
-IUSE="kernel_linux"
-
 # Bug fixes
 # fix to bug #75034
 case ${ARCH} in
-- 
2.34.1




Re: [gentoo-dev] Rationalizing USE flags by narrowing the scope of them.

2022-01-03 Thread Sam James


> On 3 Jan 2022, at 17:16, Alec Warner  wrote:
>> [snip]
> 
> I'm trying to understand your principles here. Like on what basis do
> you remove or add flags (in general).
> 
> I want to remove:
> - bash-completion

FWIW, I've managed to remove basically all instances of {bash,zsh}-completion
and made upstream PRs (all of one of which have been merged!) for fixing
`./configure` behaviour accordingly.

This is in align with our small files policy: 
https://projects.gentoo.org/qa/policy-guide/installed-files.html#pg0301 
.

> - acl
> - ldap

ACL is kind of similar to what Ionen said for PAM, i.e. sometimes
people may want to turn it off and it makes sense to expose
this option for those who do, but we don't need to try support it.

I think some of this kind of comes back to "how do we better
make clear what is/isn't okay (supported)to customise?"

LDAP is a fun one because IIRC we had it enabled by default
for too long and it didn't really break anything when we turned
It off.

Overall, I think we kind of come back to this idea of
trying to just set better IUSE defaults rather than
in profiles, so that it's per-package where possible.

> - policykit

This is a reasonable flag to keep given the heavy polkit
dependency of spidermonkey (for now) and it's somewhat
heavy-handed as a concept anyway.

> - readline

readline is a tricky one because of its relation with libedit too.

> - sound
> 
> (Part of this is just to have a meta discussion so we settle on some
> driving principles on why we keep one flag over the other.)
> 
> I can easily craft a narrative for getting rid of ipv6, for example,
> but I cannot really craft a good narrative for getting rid of pam, or
> policykit, or ldap as flags. So why do we keep some and remove others?
> 
> 

It's very hard to quantify :(


Best,
sam



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] Rationalizing USE flags by narrowing the scope of them.

2022-01-03 Thread Sam James


> On 4 Jan 2022, at 00:29, Michael Orlitzky  wrote:
> 
> On Mon, 2022-01-03 at 21:29 +0100, Piotr Karbowski wrote:
>> 
>> Perhaps the 'pam' example was extreme, but ipv6, or threads as Sam
>> shared, does not make sense to be togglable.
>> 
> 
> Many packages need their ipv6 code disabled if the kernel has no ipv6
> support, and enabling ipv6 in the kernel is a pointless security risk
> for pretty much anyone in the United States.
> 

Obviously for such cases, the flag should be kept. But that's not
the case for plenty of packages.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] Rationalizing USE flags by narrowing the scope of them.

2022-01-01 Thread Sam James


> On 2 Jan 2022, at 00:03, Scott Ellis  wrote:
> 
> Your `ipv6` USE flag hits home - I don't use IPv6, nor do I want to have IPv6 
> support built into things (just another potential "thing" that I have to 
> secure, or errors/warnings I need to suppress since I run an IPv6-less 
> kernel).
> 
> If there needs to be a path to culling USE flags, perhaps looking to which 
> flags actually cause packages to pull in additional dependencies (vs solely 
> enable/disable a feature) would be a better place to start?
> 

Yep, that's a good idea, but please do see my other comments on IPv6. USE=ipv6 
actually has some problematic properties
aside from people wanting (or not wanting) IPv6.

One reason being that it's often not even a supported configuration upstream, 
but there's others I mentioned too.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] Rationalizing USE flags by narrowing the scope of them.

2022-01-01 Thread Sam James


> On 2 Jan 2022, at 04:28, Blake Bartenbach  wrote:
> 
> On Sat Jan 1, 2022 at 4:21 PM CST, Piotr Karbowski wrote:
>> The thing is, it's 2022, and it does not make any sense to *not* support
>> IPv6, even if one does not connect to any network with IPv6, there's no
>> harm to just have it there.
>> 
> 
> This kind of logic goes down a slippery slope very quickly though.
> "There's no harm to just have it there" kind of defeats the purpose of a
> configurable operating system.

Yeah, agreed on that part. We can't really deny that Gentoo is
the home of tweakers and ricers, even if we have other types of user too.

The main aim should be to avoid complexity in ebuilds and invalid
bug reports. Masking/forcing on flags as appropriate avoids
most of these issues.

But on IPv6, please see my other post. If we have to use an autoconf
cache variable to force IPv6 off, that suggests it's probably not even
a supported configuration upstream.

> 
>> Beside 'ipv6', there are other USE flags that I have on mind. 'pam'
>> being another of them.
>> 
> 
> Well, I'm not sure about the pam one. The only USE flag that
> consistently baffles me is 'X'. It really does not seem to have a well
> defined definition, and it seems to do different things with different
> packages. For the longest time, I had that flag globally disabled, but
> used X and almost every package worked totally fine.
> 

Somewhat related: we've started moving to USE=gui to help
the situation a bit. See 
https://projects.gentoo.org/qa/policy-guide/use-flags.html#pg0802 
.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] Rationalizing USE flags by narrowing the scope of them.

2022-01-01 Thread Sam James


> On 1 Jan 2022, at 22:21, Piotr Karbowski  wrote:
> 
> Hi,
> 
> I'd like to get some insight how others see the concept of narrowing the 
> scope of USE flags in Gentoo.
> 
> Taking a quote from devmanual:
> 
>  > USE flags are to control optional dependencies and settings which the user 
> may reasonably want to select.
> 
> I'd like to focus on the 'reasonably want' here. While it is commonly agreed 
> on that we interface as USE flags only things that make sense to be 
> togglable, it is not always the case. It is not uncommon to see packages that 
> puts every possible option as USE flag which hardly benefit anyone in some 
> cases.
> 
> It creates artificial choice of USE flag that makes as much sense as building 
> and trying to use solar-powered night vision googles. Possible to be 
> engineered, but makes absolute no sense to exist, yet, there will be someone 
> who will go with it and then things will not work in desired way, bugs will 
> be reported, effort will be wasted on investigation and patching things up.
> 
> As example I'd like to use 'ipv6' USE flag, at the moment of writing this 
> email there's 351 ebuilds in tree that expose ipv6 as USE flag, allow it to 
> be disabled.
> 
> The thing is, it's 2022, and it does not make any sense to *not* support 
> IPv6, even if one does not connect to any network with IPv6, there's no harm 
> to just have it there.

There's two flags which come to mind:

- USE=ipv6:

IPv6 is a good example of a flag we may want to kill. The reason being that 
usually the flag
is ineffective -- we don't actually expose it consistently across all packages 
that may
support it in the build system, and even when there's technically a method for 
such
In the build system, it's fragile (autoconf cache variables, usually).

A good example of this recently was app-editors/vim. I looked into exactly
what "disabling" IPv6 for an editor meant. _All_ it did was force use of
older POSIX socket functions which lack support for IPv6. These newer
functions which were disabled *were not* IPv6-only.

If you don't want IPv6, you should really disable it in the kernel and
ensure you don't have any IPv6 interfaces setup.

I genuinely don't think USE=ipv6 adds any value most of the time
for the aforementioned reasons. If it really does enable/disable
a huge chunk of code, maybe it makes sense to keep it for such
a package, but in general, every time I've checked, it's
been nebulous.

TL;DR: The flag doesn't exist in plenty of packages
and usually it's rather fragile. I don't think we should keep
it where it toggles something simple like Vim (I removed it
for that reason).

- USE=threads:

Rather dangerous flag at this point, I reckon. We should really
default to on/off as appropriate per-upstream. Usually upstream
roll with one supported configuration (it's either recommended
you have it on or off) and that's it.

If someone puts "USE=threads" in make.conf expecting a speedup
or something good to come out of it, they'd be mistaken.

I'd prefer to only keep it for where there's some API change
or meaningful reason to want it. Otherwise we should just default to
upstream behaviour.


> 
> While I am all for choice, I am for choice on things that do make sense. For 
> instance, Linux kernel can be built with CONFIG_MULTIUSER=n, someone could 
> argue that since Linux kernel, that is user-configured in Gentoo, can be 
> built without support for other than UID 0, then Gentoo should support it. 
> One of the extreme examples of not supporting something that does not make 
> sense to be supported.
> 
> Beside 'ipv6', there are other USE flags that I have on mind. 'pam' being 
> another of them.

Note that having USE flags for things, even if forced on/masked (for the 
opposite case) is useful for building embedded systems. So, if you wanted to go 
this route, a sensible
first step would actually be forcing PAM on. But I don't think PAM is a 
candidate for dropping.

While I think it's hard to run a modern desktop system without it, there are a 
fair number of people who do still run -pam and I don't think
breaking it for the sake of it is a good idea. They already know there be 
dragons.

> 
> Whats your view on it?

I think I broadly agree, although PAM is a mildly controversial example.

I'd like to discuss specific examples of flags like USE=threads, 
some-if-not-all instances of USE=ipv6,
And others which people raise if any others come up.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] toolchain.eclass: fix crossdev powerpc-*-musl builds

2021-12-26 Thread Sam James


> On 27 Dec 2021, at 05:24, Georgy Yakovlev  wrote:
> 
> otherwise initial build may fail with:
> unknown long double size, cannot define BFP_FMT

If possible, would you mind filing a bug with the build log
of it failing (and brief steps on how to get it) just so
we can easily test if we can drop it in future?

(We have a lot of hacks like this where I worry
we'll never be able to drop them with confidence)

If it takes a huge amount of work to get there, then don't
bother, as it's negligible gain.

> 
> Signed-off-by: Georgy Yakovlev 
> ---
> eclass/toolchain.eclass | 5 +
> 1 file changed, 5 insertions(+)
> 
> diff --git a/eclass/toolchain.eclass b/eclass/toolchain.eclass
> index fd03ba176276..1102c4fc5d56 100644
> --- a/eclass/toolchain.eclass
> +++ b/eclass/toolchain.eclass
> @@ -1099,6 +1099,11 @@ toolchain_src_configure() {
>   # Set up defaults based on current CFLAGS
>   is-flagq -mfloat-gprs=double && confgcc+=( --enable-e500-double 
> )
>   [[ ${CTARGET//_/-} == *-e500v2-* ]] && confgcc+=( 
> --enable-e500-double )
> + if [[ ${CTARGET} == powerpc-*-musl ]]; then
> + # fix: unknown long double size, cannot define BFP_FMT
> + confgcc+=( --disable-decimal-float )
> + export gcc_cv_target_ldbl128=no
> + fi
>   ;;
>   ppc64)
>   # On ppc64 big endian target gcc assumes elfv1 by default,
> --
> 2.34.1
> 
> 

Looks fine otherwise though.

best,
sam


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Up for grabs: media-tv/plex-media-server

2021-12-25 Thread Sam James
media-tv/plex-media-server is up for grabs as the result of a proxied 
maintainer retiring.

It's a popular package. There are some active overlays with Plex in them,
it'd be nice to pull in their work / approach their authors and ask if they'd
be willing to share their work in ::gentoo.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] flag-o-matic.eclass: filter-lfs-flags: remove -D_TIME_BITS=64

2021-12-18 Thread Sam James


> On 18 Dec 2021, at 18:27, Mike Gilbert  > wrote:
> 
> glibc only allows _TIME_BITS=64 when _FILE_OFFSET_BITS=64.
> 
> Signed-off-by: Mike Gilbert mailto:flop...@gentoo.org>>
> ---
> eclass/flag-o-matic.eclass | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/eclass/flag-o-matic.eclass b/eclass/flag-o-matic.eclass
> index d262a60b6bb..32119cb9a52 100644
> --- a/eclass/flag-o-matic.eclass
> +++ b/eclass/flag-o-matic.eclass
> @@ -193,7 +193,8 @@ filter-lfs-flags() {
>   # _LARGEFILE_SOURCE: enable support for new LFS funcs (ftello/etc...)
>   # _LARGEFILE64_SOURCE: enable support for 64bit variants 
> (off64_t/fseeko64/etc...)
>   # _FILE_OFFSET_BITS: default to 64bit variants (off_t is defined as 
> off64_t)
> - filter-flags -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 
> -D_LARGEFILE64_SOURCE
> + # _TIME_BITS: default to 64bit time_t (requires _FILE_OFFSET_BITS=64)
> + filter-flags -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 
> -D_LARGEFILE64_SOURCE -D_TIME_BITS=64
> }
> 
> # @FUNCTION: filter-ldflags
> --
> 2.34.1
> 
> 


LGTM and please go ahead so we can move forward with our planning for this (see 
https://wiki.gentoo.org/wiki/Project:Toolchain/time64_migration 
 for others).

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] toolchain-funcs.eclass: add detection for loongarch

2021-12-13 Thread Sam James


> On 30 Aug 2021, at 04:25, WANG Xuerui  wrote:
> 
> From: WANG Xuerui 
> 
> The Linux port currently under review has arch/loongarch, and should
> almost certainly remain that way till merge; meanwhile it's ARCH=loong
> on the Gentoo side, per mailing list discussion[1] and eselect
> adaptation[2]. This architecture is little-endian-only according to the
> manual[3].
> 
> [1]: 
> https://archives.gentoo.org/gentoo-dev/message/388a4b7428461660e89c8eae8c292f32
> [2]: 
> https://gitweb.gentoo.org/proj/eselect.git/commit/?id=a49477f39d3f000cc2ca57f18aafbd66656aba05
> [3]: 
> https://github.com/loongson/LoongArch-Documentation/blob/2021.08.17/docs/LoongArch-Vol1-EN/basic-integer-instructions/programming-model-of-basic-integer-instructions/endian.adoc
> 
> Signed-off-by: WANG Xuerui 
> ---
> eclass/toolchain-funcs.eclass | 2 ++
> 1 file changed, 2 insertions(+)
> 

Merged.



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH v2] eclass/dune.eclass: fixes

2021-12-09 Thread Sam James


> On 9 Dec 2021, at 22:10, Maciej Barć  wrote:
> 
> bump to EAPI 8

Say "support EAPI 8" instead.

May be worth splitting the commit into a few different changes
so the summary can be more informative than "fixes"
(try git add -p).

lgtm otherwise, thanks for doing this!

> drop support for EAPI 5
> set DUNE_PKG_NAME to PN by default
> move "Move docs to the appropriate place" block to dune-install
> to make dune-install now handle a list of subpackages correctly
> 
> Signed-off-by: Maciej Barć 
> ---
> eclass/dune.eclass | 50 --
> 1 file changed, 31 insertions(+), 19 deletions(-)
> [snip]

Best,
sam



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] rfc: allow -1 for ACCT_USER_ID and ACCT_GROUP_ID in ::gentoo

2021-11-28 Thread Sam James


> On 29 Nov 2021, at 00:06, Michael Orlitzky  wrote:
> 
> On Sun, 2021-11-28 at 23:39 +0000, Sam James wrote:
>> 
>> Whissi and others raised some points that I think you may have some views on
>> (and I'm interested in hearing them).
>> 
> 
> I don't want to put words in his mouth, but I think Whissi takes issue
> with using the package manager to manage users, period. Not
> specifically with our use of a UID/GID hint.
> 
> I didn't respond to the first thread because I didn't want to pick a
> fight when the correct conclusion (IMO) was already reached. In the
> first thread I see only hypothetical problems raised (and a bunch of
> people who didn't realize the numbers are only a hint). If any of those
> problems are real and solved by allowing ACCT_USER_ID=-1 in ::gentoo,
> you'll have to point them out.
> 

Yeah, that seems like a fair interpretation (and matches my understanding).

I don't really see the problem with people who want manual administration
just setting the relevant variables in make.conf.

What I wish we had done (and there's still time to do, albeit belated --
it's still useful for the remaining big bits like Apache and nginx) is
write a news item explaining the implications and linked to a page
like https://wiki.gentoo.org/wiki/Practical_guide_to_the_GLEP_81_migration 
<https://wiki.gentoo.org/wiki/Practical_guide_to_the_GLEP_81_migration>
(which ConiKost created after we discussed how to inform users better)
that explains how to work around/express their preferences/give their own hints.

Sorry, I should've been explicit. The main thing I'd like to understand better
from your POV is:

this isn't new, but you're quite clear you feel that the UID/GID range 
limitations
are completely arbitrary and without merit(?).

Whissi essentially says the opposite: 
https://archives.gentoo.org/gentoo-dev/message/17a22877f5f18dae44a2f0859d807450 
<https://archives.gentoo.org/gentoo-dev/message/17a22877f5f18dae44a2f0859d807450>.

I'd like to understand if this is just a result of beliefs about what the PM 
should/shouldn't do
or if there's genuine problems with continuing to extend the range?

I think I'd like to see sources on various UID ranges being hardcoded in places 
as
I suspect any such software may have dubious quality anyway, but that's on him,
not you.

It still seems like in terms of interoperability, there's little impact:
folks can force whatever UIDs/GIDs they want. It's not like the situation was
any better with dynamic allocation unless you installed in exactly the right 
order
(so some precise setup wasrequired in the past anyway, the difference is now you
explicitly state what you want if you need it).

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] go-module.eclass: Add GO_OPTIONAL flag

2021-11-28 Thread Sam James


> On 28 Nov 2021, at 19:23, Zac Medico  wrote:
> 
>> [snip]
> 
> How about if we also add a GO_DEPEND variable, so that eclasshi consumers can 
> do something like BDEPEND="go? ( ${GO_DEPEND} )" ?
> --

My preference is to go with what we've been doing more recently (do _OPTIONAL) 
so that
consumers handle things properly and we don't have to try account for various 
cases
(this was the problem with eclass variables which add a USE flag or depend only
if a USE flag is set).

But that said, I can't immediately think of any such annoying edge
cases with this idea. It'd be additional to GO_OPTIONAL and it'd
ensure a sane baseline Go version is chosen. So, why not?



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] Clang/LLVM profile

2021-11-28 Thread Sam James


> On 29 Nov 2021, at 01:45, 2b57 <2...@protonmail.com> wrote:
> 
> Sorry all, it seems that I've confused the lists. I'll forward this to user
> 
> ‐‐‐ Original Message ‐‐‐
> On Monday, November 29th, 2021 at 2:42 AM, 2b57 <2...@protonmail.com> wrote:
>> Hello all,
>> 
>> I'm in the middle of developing a proof-of-concept "native" Clang/LLVM 
>> profile – such, that stage3 built using this profile will not even contain 
>> GCC and binutils. Why? Well, because Clang is in pretty good shape lately, 
>> you can compile kernel and elfutils even. Also just for fun!
>> 
>> The approach I've decided to take is to create virtual/toolchain and 
>> virtual/binutils packages with RDEPEND attributes set to gcc || clang and 
>> binutils || llvm. I've reached a point where modifications to 
>> ::gentoo/scripts/bootstrap.sh are needed, and currently I've solved it with 
>> making an OverlayFS overlay, which combines both ::gentoo repo and my custom 
>> script. General idea is that once LLVM toolchain is in place (stage1), 
>> custom profile for stage2 masks gcc/binutils and virtuals get resolved by 
>> LLVM stuff. However, that is not the case; something goes wrong and it seems 
>> that binutils package is deeply embedded somewhere else...
>> 
>> Anyway, I'd appreciate any feedback and suggestions, since I'm sure I'm not 
>> the only one interested in this topic.
>> 
>> Grab the src here: https://github.com/2b57/toolchain-clang 
>> 
> 

Honestly, I think this is pretty on-topic for gentoo-dev given a lot of us are 
quite interested in this.

That said, a few notes:
- I'm not sure why you would need virtual/toolchain or virtual/binutils unless 
it's just for usage within bootstrapping scripts? Seems more like you could 
just remove gcc from @system and such?

- _personally_, I'd prefer to do experimentation using libstdc++ from GCC and 
try libcxx later on as a fair amount of things still fail to build with LLVM's 
libcxx. But that doesn't mean others have the same view
or that it's invalid to try! They're still bugs nonetheless.

Please do let us know via Bugzilla if there's some quirks we need to add to 
ebuilds.

We also have a group of us interested in using Clang in #gentoo-arm on 
libera.chat (IRC) -- the channel is not super restricted to ARM chat.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] rfc: allow -1 for ACCT_USER_ID and ACCT_GROUP_ID in ::gentoo

2021-11-28 Thread Sam James


> On 28 Nov 2021, at 23:26, Michael Orlitzky  wrote:
> [sinp]
> The only problem that anyone has put forth is one that does not exist.
> UIDs and GIDs are still assigned dynamically in Gentoo. The number you
> type in the ebuild is only a hint: it's the first number that will be
> tried during the dynamic assignment. There is no limit on the number of
> hints, and we will never run out because a conflict is never possible,
> because the damned things are assigned dynamically.
> 
> Is there an actual problem?
> 

Could you look at this thread?

https://archives.gentoo.org/gentoo-dev/message/5bad6853520e3ab182f6399a53198fec 


Whissi and others raised some points that I think you may have some views on
(and I'm interested in hearing them).

best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] rfc: java-vm-2.eclass eapi 8 support

2021-11-27 Thread Sam James


> On 27 Nov 2021, at 09:54, Miroslav Šulc  wrote:
> 
> hello devs,
> 
> please find attached patch for upgrade of java-vm-2.eclass to support eapi 8. 
> eapi 5 is not used by any package inheriting this eclass so i dropped it.
> 
> i also attach update of the ebuilds of packages that use this eclass and can 
> be updated to eapi 8 (dev-java/icedtea can't atm, it also inherits another 
> java eclass that does not support eapi 8 yet, and dev-java/gcj-jdk is masked 
> in profiles/releases/17.0/package.mask). all works fine except 
> dev-java/openjdk (all slots) where the configuration phase fails or does not 
> finish correctly. all those packages merge fine with eapi 6 (in-tree ebuilds).
> 
> openjdk:8 - configuration finishes but no configuration file is created (and 
> hence compilation fails) as during the configuration phase it complains with 
> this:
> configure: error: Could not find freetype!
> 
> openjdk:11 and openjdk:17 - configuration fails with this error (i was told 
> by sam it is a known bug):
> configure: error: unrecognized options: --disable-static
> 

When b.g.o is back, let's make sure we update the existing bug about it / file 
a new one.

> i'd like to merge the patches asap (except the broken openjdk) so please let 
> me know if you find anything that could be improved or all's ok.
> 

lgtm. I'd send with git send-email in future to ease review.

I don't see a point in nitpicking the Java eclasses right now, especially as we 
might be looking at future improvements anyway for how we handle deps.

Thanks for doing this!

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

2021-11-25 Thread Sam James


> On 25 Nov 2021, at 17:07, Thomas Deutschmann  wrote:
> 
> On 2021-11-25 18:01, Piotr Karbowski wrote:
>>> https://github.com/gentoo/gentoo/blob/master/sys-libs/glibc/glibc-2.34-r2.ebuild#L643
>> Would you see something like this on more ebuilds, postgres, mysql, 
>> elasticsearch, or have proper FEATURE flag for it instead?
>> It's all cool and giggles until you realize that even such random variable 
>> is not even prefixed with PORTAGE_ or anything, meaning it could be taken 
>> out of shell and meant for entirely different thing.
> 
> Yeah, sounds like the I_KNOW_WHAT_I_AM_DOING thing which you end up having 
> enabled globally for various reasons.

Just like updating in a cron job is a not-great idea, setting this globally and 
not per-package via /etc/portage/env sounds rather cavalier.


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

2021-11-25 Thread Sam James


> On 25 Nov 2021, at 03:21, Thomas Deutschmann  wrote:
> 
> Bug: https://bugs.gentoo.org/825234
> Signed-off-by: Thomas Deutschmann 
> ---
> [snip]

> +On 2021-11-21, a member of the QA project accidentially de-keyworded
> +MariaDB 10.6 to address a file collision, users, who also had latest
> +dev-db/mariadb-connector-c installed, experienced (NOTE: The default
> +MySQL connector in Gentoo Linux is provided by
> +dev-db/mysql-connector-c) [Link 1].

There's no reason to refer to MySQL here (nor am I sure that we're
really "defaulting" to either MariaDB or MySQL).

> +
> +However, downgrades are not supported in MySQL/MariaDB [Link 2].

There's no reason to refer to MySQL here.

> +Link 1: https://bugs.gentoo.org/825234#c8

No need to link to the comment. Also, please use the standard [0] and [1] 
reference
notation.



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

2021-11-25 Thread Sam James


> On 25 Nov 2021, at 13:59, Thomas Deutschmann  wrote:
> If you do a normal world upgrade, this is the default portage behavior, not? 
> I.e. package manager will downgrade if you don't stop. And especially on 
> servers, people tend to use cronjobs/scripts to do that...

I would really not recommend doing this and I say this regularly. We expect 
users to:
- read elog output;
- read news items first;
- read pkg_pretend / pkg_postinst output which may be critical and needed to be 
acted on soon (even if it's accompanied by a news item, although it isn't 
always).

Should we try to improve that situation? Yeah, I think so, but that's really 
not the case right now.

> And forcefully here refers to the undesirable result (at least that was my 
> intention). Something the user doesn't want.

As we've discussed at length during the "Bitcoin consent" issue (wrt Taproot), 
users are expected to read and verify world upgrade output before proceeding.

We have similar issues with e.g. new major glibc versions (you shouldn't do 
this unless you're in a position to restart at least several services).


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

2021-11-25 Thread Sam James

> On 25 Nov 2021, at 13:37, Thomas Deutschmann  wrote:
> For the records:
> I hope that this news item won't get delayed by the recent ComRel bug someone 
> filled against me regarding this news item.

No need to reference that here. Please keep that to e.g. -core.

> While we often rush with news items and don't wait the 72h listed in GLEP 42, 
> this one should go out as soon as possible.
> Why?
> Because every minute we wait will increase the chance that someone affected 
> by this will be unable to recover. This is (for those not familiar with 
> database engines) because bin logs are about to expire (getting overwritten) 
> in typical setups after 3-5 days. And in case someone will learn about this 
> not before next week and has to do a full restore, that user will lose about 
> one week of data...

While it seems like the news item may be worth pursuing out of caution, I would 
expect people operating sophisticated database setups to do at least one of:
- slot the DB version in their world file;
- mask newer versions and unmask when ready;
- upgrade all machines to the same version at the same time (for clustering);
- read the world upgrade output and not permit the downgrade?

I also echo the other requests to add a downgrade warning based on 
${REPLACING_VERSIONS} given this is a
common issue with databases.

Also, while this does not affect the rationale for this news item, I would note 
that I would expect
anybody using Gentoo in production to be using stable or at the very least to 
have backups
of a server running on ~arch that they're upgrading without reading the 
upgrades/downgrades
carefully for software they rely on.

thanks,
sam


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: dev-tex/pdfannotextractor

2021-11-24 Thread Sam James
# Volkmar W. Pogatzki  (2021-11-23)
# Does not support updated dev-java/pdfbox-2.0.24, Bug #803488
# Blocks (CVE-2018-11797, CVE-2021-{27807,27906,31811,31812})
dev-tex/pdfannotextractor


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: dev-java/jackson

2021-11-15 Thread Sam James
# Volkmar W. Pogatzki  (2021-11-15)
# Library without consumers,
# bug #771693 (multiple CVEs); Removal in 30 days.
dev-java/jackson


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: dev-db/db-je

2021-11-12 Thread Sam James
# Volkmar W. Pogatzki  (2021-11-13)
# java package without consumers, bug #787338. Removal in 30 days.
dev-db/db-je


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: net-analyzer/neti

2021-11-12 Thread Sam James
# Volkmar W. Pogatzki  (2021-11-13)
# depends on deprecated jdk / jre, bug #787449
# stuck with EAPI 5, latest release in 2005
# Removal in 30 days.
net-analyzer/neti



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [RFC] Proposed dates for Python 3.10 switch and Python 3.8 removal

2021-11-12 Thread Sam James

[Resending due to mail client nonsense...]

> On 11 Nov 2021, at 18:44, Patrick McLean  wrote:
> On Thu, 11 Nov 2021 09:55:48 -0800
> Patrick McLean mailto:chutz...@gentoo.org>> wrote:
>> On Thu, 11 Nov 2021 17:05:45 +0100
>> Michał Górny  wrote:
>>> I'd like to add some dates regarding 3.8 removal and 3.10 switch to
>>> the implementation timeline [1].
>>> 
>>> Unless I'm mistaken, CPython is following a yearly release cycle these
>>> days.  I think it would make sense to also aim for a yearly cycle
>>> in Gentoo, i.e. roughly switch to the next minor version every year.
>>> 
>>> Hence my proposal would be to:
>>> 
>>> a. ASAP: send "please port your packages to py3.9" mail
>>> 
>>> b. 2022-06-01: remove py3.8 target
>> 
>> Could we please push this back, I would prefer some time in 2023 if
>> possible.
> To clarify, I am just asking for it to be pushed back to the start of
> 2023, not June 2023. 2023-01-01 would be fine.


I think this is fair but if you do need something for work, please say that
so we don't need to guess / think it might be for your personal capacity.

I also think it's probably worthwhile to offer upfront help if needed if you
can, especially if it is for that purpose.

That said, I agree!

I think we need to push it back a little bit to ease upgrades (see
the recent thread) but also to help downstreams like Flatcar OS
which are still migrating away from Python 3.8.

I don't mind chipping in to help.

[There is a separate issue with stabilising the Python 3.10
interpreter. I'd like this done as soon as it's ready (and it works
fine here), but there's a problem with circular dependencies
and python-any-r1 dragging it in on upgrades and causing issues with libxcrypt.]

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [RFC] Proposed dates for Python 3.10 switch and Python 3.8 removal

2021-11-11 Thread Sam James
>
>
> On 11 Nov 2021, at 18:40, Patrick McLean  wrote:
>
> On Thu, 11 Nov 2021 12:58:24 -0500
> "Wolfgang E. Sanyer"  wrote:
>
>> El jue, 11 de nov. de 2021 12:56 p. m., Patrick McLean 
>> escribió:
>>
>>> On Thu, 11 Nov 2021 17:05:45 +0100
>>> Michał Górny  wrote:
 I'd like to add some dates regarding 3.8 removal and 3.10 switch to
 the implementation timeline [1].

 Unless I'm mistaken, CPython is following a yearly release cycle these
 days.  I think it would make sense to also aim for a yearly cycle
 in Gentoo, i.e. roughly switch to the next minor version every year.

 Hence my proposal would be to:

 a. ASAP: send "please port your packages to py3.9" mail

 b. 2022-06-01: remove py3.8 target
>>>
>>> Could we please push this back, I would prefer some time in 2023 if
>>> possible.
>>>
>>
>> Is there a particular reason you wish to push it back, or is it purely
>> personal preference?
>>
> Professional preference? My employer (which employs several Gentoo
> developers, and funds a large amount of Gentoo work) needs a bit more
> time to migrate all the internal stuff to 3.10.
>
>

I think this is fair but if you do need something for work, please say that
so we don't need to guess / think it might be for your personal capacity.

I also think it's probably worthwhile to offer upfront help if needed if you
can.

That said, I agree!

I think we need to push it back a little bit to ease upgrades (see
the recent thread) but also to help downstreams like Flatcar OS
which are still migrating away from Python 3.8.

I don't mind chipping in to help.

[There is a separate issue with stabilising the Python 3.10
interpreter. I'd like this done as soon as it's ready (and it works
fine here), but there's a problem with circular dependencies
and python-any-r1 dragging it in on upgrades and causing issues with libxcrypt.]

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [RFC] Un-slotting LLVM

2021-11-08 Thread Sam James


> On 8 Nov 2021, at 11:18, Michał Górny  wrote:
> Hi,
> A few years back I've slotted LLVM and Clang to make the life with
> revdeps easier.  Long story short, every major LLVM release (which
> happens twice a year) breaks API and it takes some time for revdeps to
> adjust.  Slotting made it possible to install multiple versions
> simultaneously, and therefore let "faster" packages use newer LLVM
> without being blocked by "slower" packages on the user's system.
>
> Unfortunately, this ended up pretty bothersome to maintain.  Besides
> making ebuilds quite complex (and prone to mistakes), I'm hearing more
> and more reports of programs being broken through getting multiple LLVM
> versions in the link chain.

I think this might just be Blender and friends which are especially fragile.

We may be able to get away with just coordinating those together.

> WDYT?

If we can help it, I'd really really prefer we don't. Being able to test
various different various of Clang quickly (just like gcc) is really helpful.

(especially given one day, we might dare to dream of using Clang
for the system toolchain. It becomes a lot easier to check for
regressions if you can just flip the version.)

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] You currently cannot smoothly upgrade a 4 months old Gentoo system

2021-11-03 Thread Sam James
On 4 Nov 2021, at 00:02, Sam James  wrote:
>> On 3 Nov 2021, at 23:53, Aaron Bauman  wrote:
>> Is that where the policy belongs?
>> If so, shouldn't the council update it based on their decisions?
>> "patches are welcome" doesn't fit every scenario.
> Got to agree here. If there's a gap in the documentation,
> let's file a bug -- irrespective of if someone is going to give
> a patch.
> Just commenting this on the ML means it'll get lost
> and we'll forget about it...

Filed https://bugs.gentoo.org/821553. Please
feel free to clarify it.


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] You currently cannot smoothly upgrade a 4 months old Gentoo system

2021-11-03 Thread Sam James


> On 3 Nov 2021, at 23:53, Aaron Bauman  wrote:
> Is that where the policy belongs?
> If so, shouldn't the council update it based on their decisions?
> "patches are welcome" doesn't fit every scenario.

Got to agree here. If there's a gap in the documentation,
let's file a bug -- irrespective of if someone is going to give
a patch.

Just commenting this on the ML means it'll get lost
and we'll forget about it...

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] You currently cannot smoothly upgrade a 4 months old Gentoo system

2021-11-03 Thread Sam James

> On 3 Nov 2021, at 20:16, Rich Freeman  wrote:
> It probably would be good to get these policies documented someplace
> OTHER than the council decision logs.  I do remember the discussion
> from the distant past but it is worth having it in a more prominent
> place, especially since a one year stable update path is not going to
> happen by accident.

+1

> I was thinking that it matters way more that @system is updatable than
> world in general, since @system can be used to bootstrap everything
> else.  However, I think this distinction really doesn't make much of a
> difference.  If your system can't be smoothly updated, it is unlikely
> to be due to some leaf package like a browser/MUA/etc.  Most likely it
> is due to a widely-used dependency.
> So, by all means require @world to be updatable, but I think that if
> you focus on the packages in @system you'll basically get the rest for
> free.

This isn't quite true because it's very possible that plenty of things
will have e.g. subslot deps on @system packages and therefore
are holding them back if a change happens (you want to rebuild
everything together).

> EAPI 8 came up in a later email and to consolidate replies I'll just
> say that the issue isn't so much adopting EAPIs in newer packages, but
> the rush to tidy up old ones that creates the problems.

Right. I think we could really try to just not cleanup so aggressively
unless we know there's some nasty < dep in the ebuild.

There's another really good reason for this: stabilisation and such
doesn't catch everything, and you might find you just got upgraded
to a newly-stabled ebuild which doesn't work, and now you have
to fight to downgrade because cleanup was immediately done!

> Having QA tools detect this would be ideal, but right now they could
> only reliably detect things like newer EAPIs in @system.  Since we
> don't require putting @system dependencies in packages there is no way
> for a QA tool to tell what is or isn't needed to update portage.  Then
> you have more complex situations like PYTHON_TARGETS, and probably
> others as well.  I had one host that struggled a bit with the xcrypt
> update for some reason (some weird preserved libs issue or something -
> libcrypt was still showing up as owned by glibc after updating it and
> I had to override collisions to get xcrypt to install).  When
> upgrading a system that is completely up to date requires careful
> following of news items it is going to be painful to execute a whole
> bunch of updates at once.

(We did work very hard on this to make it as smooth as possible
and we've also dropped the workaround which caused the intentional
collision (which we mentioned in the news item + glibc's pkg_postinst)
which Portage should've ignored with default FEATURES b/c the file
was orphaned, but it should be better now.)

best,
sam



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] You currently cannot smoothly upgrade a 4 months old Gentoo system

2021-11-03 Thread Sam James


> On 3 Nov 2021, at 22:15, Joshua Kinard  wrote:
> Actually, it is possible to manage dependency errors like those.  It just
> takes a *lot* of elbow grease, and and long, long time time.  Especially if
> you have museum-grade hardware that these errors are happening on.
>
> For Perl, I've usually just uninstall everything under virtual/* first, then
> try to let it upgrade.  Sometimes that "unsticks" something in perl-core
> enough to let the upgrades apply, pulling back in any needed items from
> virtual/.  If that doesn't solve the problem enough to let emerge do an
> upgrade cycle, I'll try using just the @system target, or start yanking
> things out from perl-core/* one-by-one until emerge shuts up and does what
> it is told.

For what it's worth, you really really shouldn't need to do this.

Perl is great at consuming backtracking cycles (largely because
of all of the || ( ...  ) deps in the virtuals) and cranking that
up helps a lot.

But something which we're not really clear enough on is that
users should genuinely never have to use emerge -C / --unmerge.

Trying just @system shouldn't be needed either and is in fact
likely to be more problematic:

https://wiki.gentoo.org/wiki/Project:Portage/FAQ#Why_is_there_a_dependency_conflict_when_I_attempt_to_upgrade_a_single_package.3F

> Also, *always* check for libperl-www being in the package list.  It's
> usually sucked in by way of dev-util/intltool and is responsible for ~35-40
> perl packages alone being pulled in.  If that's in the list, try
> uninstalling just that one, then run a depclean to remove all of its
> dependencies and then see if the upgrade will work.  If the upgrade tries to
> drag intltool or libperl-www back in, use --exclude to hold it out for later.
>

Aye, we fixed eudev to not need it anymore and there's an avahi
bug for the same I'll look at shortly.

> That all said, am I alone in thinking that the way Portage emits error
> messages about dependency resolution problems is extremely messy and
> border-line unreadable at times?  The current way it outputs depgraph errors
> feels like something I'd expect from a --debug switch.  We've got a
> reputation for being playful and colorful on the command line with our
> tooling, so I would wonder if that depgraph output couldn't be made to
> looknicer?
>

Yeah, I think one of our biggest weaknesses is that there's
usually a LOT of output and figuring out what matters isn't easy.

A good rule of thumb is that the "most fatal" error is _usually_
at the bottom but this isn't always true.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] You currently cannot smoothly upgrade a 4 months old Gentoo system

2021-11-03 Thread Sam James


> On 3 Nov 2021, at 15:03, Thomas Deutschmann  wrote:
>
> Hi,
>
> it is currently not possible to smoothly run a world upgrade on a 4 months 
> old system which doesn't even have a complicated package list:
> [snip]
>
> This is not about finding solution to upgrade the system (in this case it was 
> enough to force PYTHON_TARGETS=python3_8 for portage). This is about raising 
> awareness that Gentoo is a rolling distribution and that we guarantee users 
> to be able to upgrade their system when they do world upgrades just once a 
> year (remember: in my case the last world upgrade is just 4 months old!). If 
> they cannot upgrade their system without manual intervention, we failed to do 
> our job.
>
> Situations like this will disqualify Gentoo for any professional environment 
> like this will break automatic upgrades and you cannot roll individual fixes 
> for each possible situation via CFM tools like Salt, Ansible, Puppet or Chef.
>
> It would be very appreciated if everyone will pay more attention to this in 
> future. We can do better. In most cases we can avoid problems like this by 
> keeping older ebuilds around much longer for certain key packages to help 
> with upgrades.


I agree wholeheartedly with this and thank you for raising it.

## Remark on some previous discussion

First, let me just mention that I think it's been on some of our minds but we 
need to go a bit further with formalising matters. It was brought up at the end 
of the September 2021 council meeting as a footnote:
```
[21:16:56] <@sam_> I'd like to consider "upgrade lifcycles" at some point but I 
don't have notes ready for now. Mainly just about formalising efforts to 
support upgrades for X period and to try document a procedure for e.g. new EAPI 
versions and bootstrap packages not having new EAPIs for a while, and such.
[21:17:09] <@sam_> So, no, not right now, but I'd welcome any thoughts 
post-meeting while I consider it more
[21:17:33] <@sam_> The gist is to have a checklist so that we don't "get 
excited" like with EAPI 8 and end up making upgrades hard for people
[21:17:43] <@sam_> I think the GLEP we recently approved helps with that
```

I started working on some notes too on possible improvements: 
https://wiki.gentoo.org/wiki/User:Sam/TODO#Improving_upgrades. (I wanted to 
mention all of this here because
it's easy to lose track of e.g. council meeting references on a topic, so it's 
easy to find it in the thread now.)

## Summary of the two common cases

Now, in terms of the common issues regarding upgrades, I think we have two (to 
be clear, not trying to "fix your problem" -- just bring to bear some of the
support experience I've had from #gentoo and so on):

1) World upgrades which can't complete due to new EAPIs (one's Portage lacks 
support for e.g. EAPI 8 and hence cannot read ebuilds)

I'm open to more broad measures about usage of new EAPIs in ~arch / stable 
(say, e.g. the first Portage supporting EAPI N should sit in
~arch for 4/6/??? months before any ebuilds should use it?), but I think this 
is a drastic measure we might be able to avoid. Let's keep it
in mind in case we do need it though.

My general thinking on this is that it doesn't matter _too much_(?) as long as 
one can upgrade Portage without hassle. A lot of our
users seem to know to try upgrade Portage if they can't upgrade their system 
due to new EAPIs, but they then fall down due to
cryptic errors (see my next point). We could also improve the "unknown EAPI" 
error if necessary to make this more clear.

TL;DR: We might be able to leverage a more drastic option, but my hope is we 
can avoid any direct action in handling 1) if we deal
with the next point I'm about to make (2)).

2) Portage often can't upgrade itself when there's "pending global 
PYTHON_TARGETS changes" (e.g. when we change the default value of
PYTHON_TARGETS in the profiles (like from Python 3.8 to Python 3.9))

This one is far trickier. I've started documenting common hacks/methods at 
https://wiki.gentoo.org/wiki/User:Sam/Portage_help/Upgrading_Portage#Solution
which has been rather useful in #gentoo and on the forums (it's been nice to 
see links on those and other similar pages pop up on /r/gentoo).

Portage is written in Python and has dependencies in Python. A lot of them are 
optional (which is why in the wiki page
I linked to, I suggest emerge --syncing and then turning off USE=rsync-verify 
temporarily to reduce dependencies), but
I don't think this is particularly comforting to a user who just wants to 
upgrade Portage. They don't necessarily realise
they need to toggle one or *several* flags on Portage to make it work.

dilfridge has been advocating for some time that we try look at some form of a 
"static Portage" copy (possibly
vendoring/bundling all Python dependencies) to completely decouple the Portage 
ebuilds from the Python
eclasses other than needing a (modern) Python 3 interpreter.

[I've filed a bug for this here: https://bugs.gentoo.org/821511].

[gentoo-dev] Call for testing from toolchain team: glibc-2.34

2021-10-27 Thread Sam James
Hi,

toolchain@ would appreciate if folks could keyword glibc-2.34 locally
on non-production machines to help root out any bugs.

The tracker bug is doing pretty well at this point with most remaining
issues being in last-rited packages or being tracked upstream:
https://bugs.gentoo.org/803482.

There are no plans to keyword glibc-2.34 yet given there's
a few large changes and we've not yet got over the libxcrypt
hump (that's coming very shortly, stabilisation planned for
2021-11-01!). But you can't start planning too early ;)

I would not recommend doing this on a machine
with Chromium or anything Electron based for now
as I haven't had a chance to check if the sandbox stuff
has been updated. I _think_ Firefox is fine now (at least
in the latest versions >= 91?).

If you're able, please therefore keyword glibc-2.34 on
systems where you're up for a bit of experimentation,
and ideally rebuild (-ev @world) and report any bugs.

If you feel generous, run FEATURES=test on some/all
stuff too.

Please drop into #gentoo-toolchain if you have any
questions or issues we need to debug together.

Cheers!

Best,
sam



Re: [gentoo-dev] Help with "unrecognized ELF files" needed

2021-10-26 Thread Sam James


> On 15 Oct 2021, at 16:00, xgqt  wrote:
> 
> Hi!
> 
> I'm having a problem with guile packages and portage QA checks.
> Guile puts the compiled bytecode into the /usr/lib64 directory which produces 
> a portage warning that unrecognized ELF files are being installed.
> 
> Example bug reports:
> https://bugs.gentoo.org/727146
> https://bugs.gentoo.org/817230
> 
> What can I do about this issue?

You can silence the QA warnings _if they're not genuine_ with QA_* variables. 
See https://devmanual.gentoo.org/eclass-reference/ebuild/index.html
or man ebuild.

But looks like you did that already now.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Up for grabs: net-misc/httpie

2021-10-25 Thread Sam James
Hi,

net-misc/httpie is up for grabs as a result
of the proxied maintainer retiring.

Will need a stablereq soon but other than that,
in a great state.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: dev-util/mutrace

2021-10-25 Thread Sam James
# Sam James  (2021-10-25)
# Fails to build with glibc-2.34 and binutils-2.34+. Not many other
# distros packaging this and no upstream commits since 2011. No reverse
# dependencies.
# bug #707848, bug #806476. Removal on 2021-11-25.
dev-util/mutrace


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: sys-libs/libhugetlbfs

2021-10-25 Thread Sam James
# Sam James  (2021-10-25)
# Fails to build with glibc-2.34 and appears to be obsolete. No
# reverse dependencies.
# bug #806079. Removal on 2021-11-25.
sys-libs/libhugetlbfs


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: sys-apps/mtree

2021-10-25 Thread Sam James
# Sam James  (2021-10-25)
# Fails to build with glibc-2.33 and no activity upstream; seems like
# a new fork should be packaged.
# bug #775962. Removal on 2021-11-25.
sys-apps/mtree



signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: dev-perl/MongoDB

2021-10-25 Thread Sam James
# Sam James  (2021-10-25)
# Fails to build with glibc-2.30(!) and was abandoned upstream
# a few years ago. No reverse dependencies in tree.
# bug #728556. Removal on 2021-11-25.
dev-perl/MongoDB



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [RFC] Removing separate "security supported" arch list

2021-10-23 Thread Sam James


> On 21 Oct 2021, at 18:04, Aaron Bauman  wrote:
> 
> On Thu, Oct 21, 2021 at 10:05:20AM +0200, Micha=C5=82 G=C3=B3rny wrote:
>> Hello,
>> =20
>> Splitting from the discussion in [1] (moving more arhitectures to
>> ~arch), I'd like to propose that we remove the "security supported"
>> architecture list from [2] and instead level security support with
>> the general architecture support in Gentoo, e.g. by having all
>> architectures with stable profiles be "security supported".
>> 
> 
> I fully support this approach and have long advocated for it.
> 
> Overall, all stables arches should be security supported and if
> they begin lagging behind they should be dropped to unstable.
> 
> I am sure there are some nuances, but this is a great start.
> 
> Let's do it!

+1 for reasons you said + in the original proposal.

We don't currently adhere to the stated policy verbatim
anyway and I don't think it adds anything right now.

best,
sam



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [RFC] Removing separate "security supported" arch list

2021-10-23 Thread Sam James


> On 23 Oct 2021, at 14:40, Sam James  wrote:
> 
> 
> 
>> On 23 Oct 2021, at 02:55, Thomas Deutschmann  wrote:
>> 
>> On 2021-10-21 17:16, Mike Gilbert wrote:
>>> On Thu, Oct 21, 2021 at 4:05 AM Michał Górny  wrote:
>>>> 4. In the end, Security team isn't really respecting this policy.
>>>> In the end, this leads to absurdities like GLSA being released before
>>>> a package is stable on amd64, and confusing the users [4].
>>> This is certainly an absurd mistake, but I think it is unrelated to
>>> the topic of your message. It looks like Whissi jumped the gun on
>>> releasing a GLSA, which could happen regardless of the policy. Am I
>>> missing some context?
>> 
>> Yeah, #4 is bullshit.
>> 

> Well, it's not bullshit per se, it's just not consistent with the policy. We 
> should
> update the policy to reflect real life.
> 
> What I'd probably like us to do is have at least amd64 stable before
> publishing in future (and if there's a reason amd64 can't be, we probably
> can't/shouldn't stable on other arches anyway).

... additionally, even if we're not going to update the policy page (I don't see
why we shouldn't), what exactly does this leave "security supported" meaning...?

mgorny pointed this out already but there's no real point to having
the designation: it makes no difference wrt cleanups and also
no real difference to when we publish GLSAs either.

> [...]

> best,
> sam



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [RFC] Removing separate "security supported" arch list

2021-10-23 Thread Sam James


> On 23 Oct 2021, at 02:55, Thomas Deutschmann  wrote:
> 
> On 2021-10-21 17:16, Mike Gilbert wrote:
>> On Thu, Oct 21, 2021 at 4:05 AM Michał Górny  wrote:
>>> 4. In the end, Security team isn't really respecting this policy.
>>> In the end, this leads to absurdities like GLSA being released before
>>> a package is stable on amd64, and confusing the users [4].
>> This is certainly an absurd mistake, but I think it is unrelated to
>> the topic of your message. It looks like Whissi jumped the gun on
>> releasing a GLSA, which could happen regardless of the policy. Am I
>> missing some context?
> 
> Yeah, #4 is bullshit.
> 

Well, it's not bullshit per se, it's just not consistent with the policy. We 
should
update the policy to reflect real life.

What I'd probably like us to do is have at least amd64 stable before
publishing in future (and if there's a reason amd64 can't be, we probably
can't/shouldn't stable on other arches anyway).

> The security team was never happy with the situation to hold back GLSAs until 
> last architecture was marked stable.
> 
> Saying that we are not respecting our own own policy is absurd. The team 
> discussed this in 2018 and we agreed that it is fine to already publish a 
> GLSA in case a GLSA is ready and when at least one major architecture (amd64 
> or x86 at that time) was marked stable. That exception doesn't require a 
> formal policy update.
> 

I don't get why this means we shouldn't just update the page..?

> We even wanted to go one step further and release GLSA when no fixed version 
> is available at all to inform users and give them a chance to take actions on 
> their own (to be able to take actions on your own, i.e. you first need to be 
> aware of a problem). However, this would be too complicated and would 
> frustrate many users.

Aye, although this would involve different instructions.

> 
> The lived practice with releasing GLSA already when just one major 
> architecture has set stable keyword (and in most cases we covered amd64 and 
> x86 at release time) received good feedback and is accepted by users and 
> didn't cause any problems (can't remember that we ever got GLSA feedback for 
> other architectures than amd64 or x86).
> 

best,
sam



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-portage-dev] [RFC] LTS branch of Portage

2021-10-20 Thread Sam James


> On 19 Oct 2021, at 00:24, Francesco Riosa  wrote:
> 
> Sorry but portage is too strictly related to the ebuilds in tree, recent 
> removal of EAPI=5 from most eclasses underlined that.
> Or to put id differently if you want a LTS portage you also need a certain 
> number of "protected" eclasses and ebuilds
> It seems a lot of (very appreciated but don't count on me) work

FWIW, I don't think it'd be a big deal to just agree that we wouldn't rip out 
old EAPI support from eclasses
and also slow down with adopting new EAPIs for ebuilds (which there's vague 
consensus about wrt EAPI 8
being done a bit too quickly anyway).

i.e. I don't think this is really a blocker.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [RFC] Moving more architectures to ~arch only

2021-10-17 Thread Sam James


> On 18 Oct 2021, at 01:50, Sam James  wrote:
> 
> 
> 
>> On 14 Oct 2021, at 14:40, Marek Szuba  wrote:
>> 
>> Dear everyone,
>> 
>> Following some private discussions, I feel quite strongly now that it would 
>> both considerably improve certain processes and make our use of limited 
>> manpower more efficient were we to further reduce the number of stable 
>> arches in Gentoo Linux. Specifically, I propose to drop
>> - hppa,
>> - ppc,
>> - sparc,
>> - x86
>> to ~arch-only status.
> 
> [snip]
> 
> My suggested actions:
> 
> [snip]
> 
> - We drop any large suites of packages at least to ~arch where they're 
> problematic. A good place
> to start would probably be scientific stuff which isn't a test dependency (or 
> likely to become one)
> in future of e.g. the Python stack. There's quite a few niche sci 
> applications stable on e.g. x86
> which probably don't have a need to be.
> 

One more while it's in my head:

- Try break the assumption in developers' heads (mine too!) that we should 
stable
x86 while we're stabling amd64 and so on, if it's not already stable there. Try
reduce growing the x86 stable base unless somebody wants it.

>> [snip]
> 
> best,
> sam
> 



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [RFC] Moving more architectures to ~arch only

2021-10-17 Thread Sam James


> On 14 Oct 2021, at 14:40, Marek Szuba  wrote:
> 
> Dear everyone,
> 
> Following some private discussions, I feel quite strongly now that it would 
> both considerably improve certain processes and make our use of limited 
> manpower more efficient were we to further reduce the number of stable arches 
> in Gentoo Linux. Specifically, I propose to drop
> - hppa,
> - ppc,
> - sparc,
> - x86
> to ~arch-only status.

I'm not sure we should go down this route. Dakon's email covers a lot of the 
reasons, but I'll try to add to it my own rationale too:

- Most failures found via arch testing _aren't_ arch-specific, but they serve 
as a useful quality check. That is,
usually, we're not held back by some odd e.g. SIGBUS that nobody knows how to 
fix.

- We're not really helping users by making such a change. Any problems which 
prevent stabilisation still exist. We're
just reducing the quality of the Gentoo experience for users on these arches.

My suggested actions:

- As referenced below, make more developers aware they're welcome to have 
access to our various exotic
hardware!

- Encourage developers to run test suites on their packages. This is a modern 
part of Gentoo development
and isn't optional if a package has a functioning test suite which isn't hell 
to get running - i.e. you should really
_try_.

- Further, encouraging tatt/pkg-testing-tools like Dakon suggested before 
pushing new ebuilds. A wiki page
I've started might prove helpful too: 
https://wiki.gentoo.org/wiki/User:Sam/Useful_scripts#Testing.

- We drop any large suites of packages at least to ~arch where they're 
problematic. A good place
to start would probably be scientific stuff which isn't a test dependency (or 
likely to become one)
in future of e.g. the Python stack. There's quite a few niche sci applications 
stable on e.g. x86
which probably don't have a need to be.

The gist being, I think we can focus our efforts (and try educate + encourage 
others to help)
without completely shutting the door here.

I'm quite happy helping with these arches right now (although hppa is 
problematic
due to the speed of our current hardware, we are wondering if we can get some
other kit) as long as we all continue to chip in. But more help is very welcome 
and desired.

What would probably help more than anything else right now is dropping stable 
keywords
for irrelevant packages (not wasting time on some stablereqs where nobody is 
probably
using that $application on $arch) and having a tool to easily report bugs so I 
don't waste time
copying/pasting logs.

(slyfox actually mentioned his desire for such a tool in his farewell post: 
https://trofi.github.io/posts/226-farewell-gentoo-dev.html).

Now, addressing the rest of the email:

> Note that this does NOT mean we intend to drop support for those arches 
> altogether.
> 
> There are IMHO several good reasons for this:
> - most of the arches from this list are quite dated and either aren't really 
> developed upstream any more or got superseded by newer ones (for the record, 
> it's been 18 years since the first amd64 CPUs came out)

But users of this hardware can only really get by on Gentoo without 
super-super-frequent updates. Not all versions added to ~arch to get stabilised 
and also once stabled are less likely
to have e.g. build failures so less wasted time.

> - we have got very few people actually supporting these arches, and in case 
> of hppa there is also the hardware bottleneck. Subsequently, stabilisation 
> requests often take a long time to resolve

I think a better way of tackling this is to make developers aware they can even 
access a lot of this hardware! I don't think
many developers realise they're welcome to have access to our various arch 
testing machines -- they're not just
for a select few. We want more help!

> [snip]

best,
sam



signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: app-text/htmlc

2021-10-16 Thread Sam James
# Sam James  (2021-10-16)
# Fails to compile with modern OCaml
# bug #704464, bug #780090
# Removal on 2021-11-16.
app-text/htmlc


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: dev-cpp/rttr

2021-10-16 Thread Sam James
# Sam James  (2021-10-17)
# Fails to build with glibc 2.34 and no reverse dependencies.
# bug #806508
# Removal on 2021-11-17.
dev-cpp/rttr


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Last rites: dev-ml/async_ssl and dev-ml/ppx_tools_versioned

2021-10-14 Thread Sam James
# Sam James  (2021-10-15)
# Drop broken packages with no reverse dependencies
# Removal on 2021-11-15. bug #817965, bug #771756, bug #767196
dev-ml/async_ssl
dev-ml/ppx_tools_versioned



signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] [PATCH 3/3] opam.eclass: restrict strip

2021-10-10 Thread Sam James
This breaks at least ocamlopt.

Closes: https://bugs.gentoo.org/803047
Closes: https://bugs.gentoo.org/811315
Signed-off-by: Sam James 
---
 eclass/opam.eclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/eclass/opam.eclass b/eclass/opam.eclass
index 262d726a8a7a2..6b9d43204206d 100644
--- a/eclass/opam.eclass
+++ b/eclass/opam.eclass
@@ -21,6 +21,8 @@ esac
 # Do not complain about CFLAGS etc since ml projects do not use them.
 QA_FLAGS_IGNORED='.*'
 
+RESTRICT="strip"
+
 # @ECLASS-VARIABLE: OPAM_INSTALLER_DEP
 # @PRE_INHERIT
 # @DESCRIPTION:
-- 
2.33.0




[gentoo-dev] [PATCH 2/3] findlib.eclass: restrict strip

2021-10-10 Thread Sam James
This breaks at least ocamlopt.

Closes: https://bugs.gentoo.org/803047
Closes: https://bugs.gentoo.org/811315
Signed-off-by: Sam James 
---
 eclass/findlib.eclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/eclass/findlib.eclass b/eclass/findlib.eclass
index 3b19b30c57c9b..6239beccb7a43 100644
--- a/eclass/findlib.eclass
+++ b/eclass/findlib.eclass
@@ -25,6 +25,8 @@ QA_FLAGS_IGNORED='.*'
 # Required to use the ocamlopt? dep in RDEPEND below
 IUSE="ocamlopt"
 
+RESTRICT="strip"
+
 # From this findlib version, there is proper stublibs support.
 DEPEND=">=dev-ml/findlib-1.0.4-r1"
 [[ ${FINDLIB_USE} ]] && DEPEND="${FINDLIB_USE}? ( ${DEPEND} )"
-- 
2.33.0




[gentoo-dev] [PATCH 1/3] dune.eclass: restrict strip

2021-10-10 Thread Sam James
This breaks at least ocamlopt.

Closes: https://bugs.gentoo.org/803047
Closes: https://bugs.gentoo.org/811315
Signed-off-by: Sam James 
---
 eclass/dune.eclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/eclass/dune.eclass b/eclass/dune.eclass
index 02a8a870ef43e..8a6e01893932a 100644
--- a/eclass/dune.eclass
+++ b/eclass/dune.eclass
@@ -28,6 +28,9 @@ esac
 # Do not complain about CFLAGS etc since ml projects do not use them.
 QA_FLAGS_IGNORED='.*'
 
+# Breaks with ocamlopt
+RESTRICT="strip"
+
 EXPORT_FUNCTIONS src_compile src_test src_install
 
 RDEPEND=">=dev-lang/ocaml-4:=[ocamlopt?] dev-ml/dune:="
-- 
2.33.0




[gentoo-portage-dev] [PATCH] lib/_emerge/actions.py: warn on missing /run

2021-10-06 Thread Sam James
Newer versions of build-docbook-catalog use
/run/lock. This exposed that we weren't
asking users to mount /run in the handbook.

Check if it exists and warn if it doesn't.

This should primarily (exclusively?) be a
problem in chroots given an init system
should be creating this.

Bug: https://bugs.gentoo.org/816303
Signed-off-by: Sam James 
---
 lib/_emerge/actions.py | 34 ++
 1 file changed, 22 insertions(+), 12 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 05a115250..1b40bebd3 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2986,17 +2986,26 @@ def validate_ebuild_environment(trees):
 check_locale()
 
 
-def check_procfs():
-procfs_path = "/proc"
-if platform.system() not in ("Linux",) or os.path.ismount(procfs_path):
-return os.EX_OK
-msg = "It seems that %s is not mounted. You have been warned." % 
procfs_path
-writemsg_level(
-"".join("!!! %s\n" % l for l in textwrap.wrap(msg, 70)),
-level=logging.ERROR,
-noiselevel=-1,
-)
-return 1
+def check_mounted_fs():
+paths = {"/proc": False, "/run": False}
+
+for path in paths.keys():
+if platform.system() not in ("Linux",) or os.path.ismount(path):
+paths[path] = True
+continue
+
+msg = "It seems that %s is not mounted. You have been warned." % path
+writemsg_level(
+"".join("!!! %s\n" % l for l in textwrap.wrap(msg, 70)),
+level=logging.ERROR,
+noiselevel=-1,
+)
+
+# Were any of the mounts we were looking for missing?
+if False in paths.values():
+return 1
+
+return os.EX_OK
 
 
 def config_protect_check(trees):
@@ -3474,7 +3483,8 @@ def run_action(emerge_config):
 repo_name_check(emerge_config.trees)
 repo_name_duplicate_check(emerge_config.trees)
 config_protect_check(emerge_config.trees)
-check_procfs()
+
+check_mounted_fs()
 
 for mytrees in emerge_config.trees.values():
 mydb = mytrees["porttree"].dbapi
-- 
2.33.0




Re: [gentoo-portage-dev] [PATCH 2/2] lib/_emerge/resolver/output_helpers.py: explicitly state 'all satisfied'

2021-10-06 Thread Sam James


> On 6 Oct 2021, at 19:05, Duncan <1i5t5.dun...@cox.net> wrote:
> 
> Sam James posted on Sat,  2 Oct 2021 21:11:56 +0100 as excerpted:
> 
>> This makes things a bit less confusing and tries to avoid users focusing
>> on (soft) blocks which aren't actually the problem they're hitting.
> 
> I was pleasantly surprised by that "all satisfied" earlier today.
> Thanks.  I could get my brain around the old format but this is /so/ much
> nicer! =:^)
> 

Big thanks for giving this positive feedback! Great motivator :)

And glad it helped!

Best,
sam


signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] Package up for grabs: app-text/mupdf

2021-10-05 Thread Sam James
Hi,

Putting app-text/mupdf up for grabs because I've not used it for a while.

It needs a version bump (1.19.0) and some minor patches rebasing.

Please note that anyone taking this over should keep an eye on upstream's
git occasionally to look for security relevant patches.

Best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH] 2021-10-08-openssh-rsa-sha1: add news item

2021-10-05 Thread Sam James


> On 5 Oct 2021, at 18:43, Mike Gilbert  wrote:
> 
> Signed-off-by: Mike Gilbert 
> ---
> .../2021-10-08-openssh-rsa-sha1.en.txt| 26 +++
> 1 file changed, 26 insertions(+)
> create mode 100644 
> 2021-10-08-openssh-rsa-sha1/2021-10-08-openssh-rsa-sha1.en.txt
> 
> diff --git a/2021-10-08-openssh-rsa-sha1/2021-10-08-openssh-rsa-sha1.en.txt 
> b/2021-10-08-openssh-rsa-sha1/2021-10-08-openssh-rsa-sha1.en.txt
> new file mode 100644
> index 000..cfdcc4a
> --- /dev/null
> +++ b/2021-10-08-openssh-rsa-sha1/2021-10-08-openssh-rsa-sha1.en.txt
> @@ -0,0 +1,26 @@
> +Title: OpenSSH RSA SHA-1 signatures
> +Author: Mike Gilbert 
> +Posted: 2021-10-08
> +Revision: 1
> +News-Item-Format: 2.0
> +Display-If-Installed: net-misc/openssh
> +
> +As of version 8.8, OpenSSH disables RSA signatures using the SHA-1
> +hash algorithm by default. This change affects both the client and
> +server components.

lgtm


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [PATCH 0/1] Add 'notes' element to metadata.xml (GLEP 68)

2021-10-05 Thread Sam James


> On 5 Oct 2021, at 21:29, Alec Warner  wrote:
> 
> I thought we were going to go with the github-pages type route
> (markdown, rendered online or locally.)
> 

So, the thinking was, we could allow somewhat shorthand
notes or for the people who want to invest more time, allow
the GitHub-pages type route.

But I'm open to the idea of just recommending people
use that if there's no appetite for mixed types.

> -A
> 
> On Tue, Oct 5, 2021 at 11:28 AM Sam James  wrote:
>> 
>> This is a preliminary version/draft of a proposed change to
>> GLEP 68.
>> 
>> I'd like to introduce a method for developers to signal anything
>> special about a package and how to correctly maintain it.
>> 
>> Sam James (1):
>>  glep-0068: Add notes element for package maintenance instructions
>> 
>> glep-0068.rst | 26 +++---
>> 1 file changed, 23 insertions(+), 3 deletions(-)
>> 
>> --
>> 2.33.0
>> 
>> 
> 



signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] [PATCH 1/1] glep-0068: Add notes element for package maintenance instructions

2021-10-05 Thread Sam James
This adds a new '' element to allow maintainers to describe
package-specific quirks or other instructions on how to correctly
maintain a package. This is intended to encourage developers to document
knowledge and increase the bus-factor of packages which are delicate
but must live beyond a maintainer.

Signed-off-by: Sam James 
---
 glep-0068.rst | 26 +++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/glep-0068.rst b/glep-0068.rst
index 83e54d9..21bdf2a 100644
--- a/glep-0068.rst
+++ b/glep-0068.rst
@@ -4,10 +4,10 @@ Title: Package and category metadata
 Author: Michał Górny 
 Type: Standards Track
 Status: Final
-Version: 1.1
+Version: 1.2
 Created: 2016-03-14
-Last-Modified: 2021-09-11
-Post-History: 2016-03-16, 2018-02-20
+Last-Modified: 2021-10-05
+Post-History: 2016-03-16, 2018-02-20, 2021-10-05
 Content-Type: text/x-rst
 Requires: 67
 Replaces: 34, 46, 56
@@ -160,6 +160,8 @@ element can contain, in any order:
   in different languages (at most one for each language), as detailed
   in `USE flag descriptions`_.
 
+- at most one  element containing notes on maintenance.
+
 - at most one  element providing information on upstream
   of the package, as detailed in `Upstream descriptions`_.
 
@@ -213,6 +215,18 @@ The  element can contain the following elements:
   which the description applies, and contains text, optionally interspersed
   with  and  elements.
 
+Notes element
+~
+
+The  element describes important information on how to maintain
+a package.
+
+The  element has an obligatory ``type=""`` attribute whose value is
+can be either ``text`` or ``url``. If its value is ``text``, then the
+ value is formed of multi-line text. If its value is ``url``, the
+ value should be a link to a page containing information on how
+to correct maintain the package.
+
 Upstream descriptions
 ~
 The  element provides information on the upstream of a package.
@@ -484,6 +498,12 @@ Example metadata.xml file
 Konfiguriert das Paket 
mit Unterstützung für bar (benötigt dev-libs/bar)
 Konfiguriert das 
Paket mit Unterstützung für bar
   
+  
+  Almost every release involves checking a diff of the build system.
+  Be careful of bundled libfoo which sometimes has custom 
modifications.
+  Versioning scheme is unusual.
+  Track record of ABI breaks within minor releases: check!
+  
   
 
   upstr...@example.com
-- 
2.33.0




[gentoo-dev] [PATCH 0/1] Add 'notes' element to metadata.xml (GLEP 68)

2021-10-05 Thread Sam James
This is a preliminary version/draft of a proposed change to
GLEP 68.

I'd like to introduce a method for developers to signal anything
special about a package and how to correctly maintain it.

Sam James (1):
  glep-0068: Add notes element for package maintenance instructions

 glep-0068.rst | 26 +++---
 1 file changed, 23 insertions(+), 3 deletions(-)

-- 
2.33.0




[gentoo-dev] Packages up for grabs: misc packages from chainsaw@

2021-10-04 Thread Sam James
Hi all,

Chainsaw asked retirement@ to drop him to maintainer-needed on his packages, so 
here's the resultant
packages with no maintainers left:

app-admin/ansible-lint
app-arch/rpm
dev-python/progressbar2
dev-python/pytest-flakes
dev-python/requests-credssp
net-libs/iax
net-misc/arpsponge
net-misc/astmanproxy
net-misc/bgpq3
net-misc/openr2
net-misc/sipsak
net-misc/stuntman
sys-firmware/iwl6005-ucode
sys-firmware/iwl6030-ucode
www-apache/mod_auth_radius
www-apache/mod_auth_xradius
x11-plugins/pidgin-mpris

best,
sam


signature.asc
Description: Message signed with OpenPGP


[gentoo-portage-dev] [PATCH 1/1] bin/misc-function.sh: check scanelf return code

2021-10-02 Thread Sam James
This is part of a series of fixes for the linked bug (failure
to preserve libraries in some situations).

We need to check if scanelf failed when calling it in the
installed-files QA check as we later use it to populate the VDB.

Silently continuing results in either blank e.g. PROVIDES,
NEEDED{,.ELF.2} or those files may be missing entirely,
resulting in a corrupt state both on the system and in
any generated binpkgs.

Adds an escape variable (PORTAGE_NO_SCANELF_CHECK) to allow
re-emerging pax-utils if it's broken.

Bug: https://bugs.gentoo.org/811462
Signed-off-by: Sam James 
---
 bin/misc-functions.sh | 64 +--
 1 file changed, 50 insertions(+), 14 deletions(-)

diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index bd1fb7553..e4defa550 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -177,25 +177,61 @@ install_qa_check() {
if type -P scanelf > /dev/null ; then
# Save NEEDED information after removing self-contained 
providers
rm -f "$PORTAGE_BUILDDIR"/build-info/NEEDED{,.ELF.2}
+
# We don't use scanelf -q, since that would omit libraries like
# musl's /usr/lib/libc.so which do not have any DT_NEEDED or
# DT_SONAME settings. Since we don't use scanelf -q, we have to
# handle the special rpath value "  -  " below.
-   scanelf -yRBF '%a;%p;%S;%r;%n' "${D%/}/" | { while IFS= read -r 
l; do
-   arch=${l%%;*}; l=${l#*;}
-   obj="/${l%%;*}"; l=${l#*;}
-   soname=${l%%;*}; l=${l#*;}
-   rpath=${l%%;*}; l=${l#*;}; [ "${rpath}" = "  -  " ] && 
rpath=""
-   needed=${l%%;*}; l=${l#*;}
-
-   # Infer implicit soname from basename (bug 715162).
-   if [[ -z ${soname} && $(file "${D%/}${obj}") == *"SB 
shared object"* ]]; then
-   soname=${obj##*/}
-   fi
+   scanelf_output=$(scanelf -yRBF '%a;%p;%S;%r;%n' "${D%/}/")
+
+   case $? in
+   0)
+   # Proceed
+   ;;
+   159)
+   # Unknown syscall
+   eerror "Failed to run scanelf (unknown syscall)"
+
+   if [[ -z ${PORTAGE_NO_SCANELF_CHECK} ]]; then
+   # Abort only if the special recovery 
variable isn't set
+   eerror "Please upgrade pax-utils with:"
+   eerror " PORTAGE_NO_SCANELF_CHECK=1 
emerge -v1 app-misc/pax-utils"
+   eerror "Aborting to avoid corrupting 
metadata"
+   die "${0##*/}: Failed to run scanelf! 
Update pax-utils?"
+   fi
+   ;;
+   *)
+   # Failed in another way
+   eerror "Failed to run scanelf (returned: $?)!"
+
+   if [[ -z ${PORTAGE_NO_SCANELF_CHECK} ]]; then
+   # Abort only if the special recovery 
variable isn't set
+   eerror "Please report this bug at 
https://bugs.gentoo.org/!;
+   eerror "It may be possible to re-emerge 
pax-utils with:"
+   eerror " PORTAGE_NO_SCANELF_CHECK=1 
emerge -v1 app-misc/pax-utils"
+   eerror "Aborting to avoid corrupting 
metadata"
+   die "${0##*/}: Failed to run scanelf!"
+   fi
+   ;;
+   esac
+
+   if [[ -n ${scanelf_output} ]]; then
+   while IFS= read -r l; do
+   arch=${l%%;*}; l=${l#*;}
+   obj="/${l%%;*}"; l=${l#*;}
+   soname=${l%%;*}; l=${l#*;}
+   rpath=${l%%;*}; l=${l#*;}; [ "${rpath}" = "  -  
" ] && rpath=""
+   needed=${l%%;*}; l=${l#*;}
+
+   # Infer implicit soname from basename (bug 
715162).
+   if [[ -z ${soname} && $(file "${D%/}${obj}") == 
*"SB shared object"* ]]; then
+   soname=${obj##*/}
+   fi
 
-   echo &q

[gentoo-portage-dev] [PATCH 0/1] Check for errors in scanelf

2021-10-02 Thread Sam James
Posted at https://github.com/gentoo/portage/pull/750 originally
and merged in 3.0.24, but posting here for posterity.

Related to the general preserve-libs issues I've been working on.

Sam James (1):
  bin/misc-function.sh: check scanelf return code

 bin/misc-functions.sh | 64 +--
 1 file changed, 50 insertions(+), 14 deletions(-)

-- 
2.33.0




[gentoo-portage-dev] [PATCH 2/2] lib/_emerge/resolver/output_helpers.py: explicitly state 'all satisfied'

2021-10-02 Thread Sam James
This makes things a bit less confusing and tries to avoid
users focusing on (soft) blocks which aren't actually
the problem they're hitting.

Signed-off-by: Sam James 
---
 lib/_emerge/resolver/output_helpers.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/_emerge/resolver/output_helpers.py 
b/lib/_emerge/resolver/output_helpers.py
index f80b79ccf..6ce812189 100644
--- a/lib/_emerge/resolver/output_helpers.py
+++ b/lib/_emerge/resolver/output_helpers.py
@@ -163,6 +163,8 @@ class _PackageCounters:
 myoutput.append(
 bad(" (%s unsatisfied)") % (self.blocks - 
self.blocks_satisfied)
 )
+else:
+myoutput.append(" (all satisfied)")
 return "".join(myoutput)
 
 
-- 
2.33.0




[gentoo-portage-dev] [PATCH 1/2] lib/_emerge/resolver/output.py: say 'soft blocking' explicitly

2021-10-02 Thread Sam James
Before:
```
[blocks b  ] >perl-core/Scalar-List-Utils-1.550.0-r999 
(">perl-core/Scalar-List-Utils-1.550.0-r999" is blocking 
virtual/perl-Scalar-List-Utils-1.550.0)

```

After:
```
[blocks b  ] >perl-core/Scalar-List-Utils-1.550.0-r999 
(">perl-core/Scalar-List-Utils-1.550.0-r999" is soft blocking 
virtual/perl-Scalar-List-Utils-1.550.0)

```

This should make it a little bit clearer when a block matters,
especially given we already explicitly say 'hard blocking'
for the opposite case.

Signed-off-by: Sam James 
---
 lib/_emerge/resolver/output.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/resolver/output.py b/lib/_emerge/resolver/output.py
index e891d84c0..7b5602a78 100644
--- a/lib/_emerge/resolver/output.py
+++ b/lib/_emerge/resolver/output.py
@@ -108,7 +108,7 @@ class Display:
 if blocker.atom.blocker.overlap.forbid:
 blocking_desc = "hard blocking"
 else:
-blocking_desc = "blocking"
+blocking_desc = "soft blocking"
 if self.resolved != blocker.atom:
 addl += colorize(
 self.blocker_style,
-- 
2.33.0




[gentoo-portage-dev] [PATCH 2/2] Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg

2021-10-02 Thread Sam James
This is part of a series of fixes for the linked bug (failure
to preserve libraries in some situations).

When unpacking a binpkg to be installed, we should check
for the existence of PROVIDES if we're installing any
dynamic libraries. If PROVIDES does not exist in that case,
this suggests that e.g. scanelf malfunctioned or some corruption occurred.

Bug: https://bugs.gentoo.org/811462
Signed-off-by: Sam James 
---
 lib/_emerge/Binpkg.py | 28 +++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
index c7dde69bd..9b876f354 100644
--- a/lib/_emerge/Binpkg.py
+++ b/lib/_emerge/Binpkg.py
@@ -2,7 +2,7 @@
 # Distributed under the terms of the GNU General Public License v2
 
 import functools
-
+import glob
 import _emerge.emergelog
 from _emerge.EbuildPhase import EbuildPhase
 from _emerge.BinpkgFetcher import BinpkgFetcher
@@ -13,6 +13,7 @@ from _emerge.EbuildMerge import EbuildMerge
 from _emerge.EbuildBuildDir import EbuildBuildDir
 from _emerge.SpawnProcess import SpawnProcess
 from portage.eapi import eapi_exports_replace_vars
+from portage.output import colorize
 from portage.util import ensure_dirs
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
 import portage
@@ -425,6 +426,31 @@ class Binpkg(CompositeTask):
 self._async_unlock_builddir(returncode=self.returncode)
 return
 
+# Before anything else, let's do an integrity check.
+(provides,) = self._bintree.dbapi.aux_get(self.pkg.cpv, ["PROVIDES"])
+if not provides:
+# Let's check if we've got inconsistent results.
+# If we're installing dynamic libraries (.so files), we should
+# really have a PROVIDES.
+# (This is a complementary check at the point of ingestion for the
+# creation check in doebuild.py)
+# Note: we could check a non-empty PROVIDES against the list of 
.sos,
+# but this doesn't gain us anything. We're interested in failure
+# to properly parse the installed files at all, which should really
+# be a global problem (e.g. bug #811462)
+installed_dynlibs = glob.glob(
+self.settings["D"] + "/**/*.so", recursive=True
+)
+if installed_dynlibs:
+self._writemsg_level(
+colorize(
+"BAD",
+"!!! Error! Installing dynamic libraries (.so) with 
blank PROVIDES!",
+),
+noiselevel=-1,
+level=logging.ERROR,
+)
+
 try:
 with io.open(
 _unicode_encode(
-- 
2.33.0




[gentoo-portage-dev] [PATCH 1/2] doebuild.py: check for inconsistent PROVIDES/image post-src_install

2021-10-02 Thread Sam James
This is part of a series of fixes for the linked bug (failure
to preserve libraries in some situations).

At the point of installation (even if not merging), we need
to detect inconsistent metadata: PROVIDES should be populated
if we're installing any dynamic libraries. This suggests that
e.g. scanelf malfunctioned or some corruption occurred.

Bug: https://bugs.gentoo.org/811462
Signed-off-by: Sam James 
---
 lib/portage/package/ebuild/doebuild.py | 24 +++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/lib/portage/package/ebuild/doebuild.py 
b/lib/portage/package/ebuild/doebuild.py
index 9650a8444..dc3fe3d97 100644
--- a/lib/portage/package/ebuild/doebuild.py
+++ b/lib/portage/package/ebuild/doebuild.py
@@ -3,6 +3,7 @@
 
 __all__ = ["doebuild", "doebuild_environment", "spawn", "spawnebuild"]
 
+import glob
 import grp
 import gzip
 import errno
@@ -3079,7 +3080,7 @@ def _post_src_install_soname_symlinks(mysettings, out):
 ) as f:
 f.write(soname_deps.requires)
 
-if soname_deps.provides is not None:
+if soname_deps.provides:
 with io.open(
 _unicode_encode(
 os.path.join(build_info_dir, "PROVIDES"),
@@ -3091,6 +3092,27 @@ def _post_src_install_soname_symlinks(mysettings, out):
 errors="strict",
 ) as f:
 f.write(soname_deps.provides)
+else:
+# Let's check if we've got inconsistent results.
+# If we're installing dynamic libraries (.so files), we should
+# really have a PROVIDES.
+# (This is a complementary check at the point of creation for the
+# ingestion check in Binpkg.py)
+# Note: we could check a non-empty PROVIDES against the list of .sos,
+# but this doesn't gain us anything. We're interested in failure
+# to properly parse the installed files at all, which should really
+# be a global problem (e.g. bug #811462)
+installed_dynlibs = glob.glob(image_dir + "/**/*.so", recursive=True)
+
+if installed_dynlibs:
+self._writemsg_level(
+colorize(
+"BAD",
+"!!! Error! Installing dynamic libraries (.so) with blank 
PROVIDES!",
+),
+noiselevel=-1,
+level=logging.ERROR,
+)
 
 if unrecognized_elf_files:
 qa_msg = ["QA Notice: Unrecognized ELF file(s):"]
-- 
2.33.0




[gentoo-portage-dev] [PATCH 0/2] Detect broken VDB on merging/binpkg creation

2021-10-02 Thread Sam James
Further fixes for when the VDB is corrupted by e.g. broken scanelf.

Already posted on GH a while ago at https://github.com/gentoo/portage/pull/744.


Sam James (2):
  doebuild.py: check for inconsistent PROVIDES/image post-src_install
  Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg

 lib/_emerge/Binpkg.py  | 28 +-
 lib/portage/package/ebuild/doebuild.py | 24 +-
 2 files changed, 50 insertions(+), 2 deletions(-)

-- 
2.33.0




Re: [gentoo-dev] RFC: dev-libs/openssl USE=bindist removal

2021-09-29 Thread Sam James

> On 27 Sep 2021, at 23:50, Robin H. Johnson  wrote:
> 
> Deadline for responses: 2021/10/14!
> 
> The Foundation would like to propose that RedHat/Fedora "hobble" patch
> presently applied when USE=bindist is true shall be removed from
> dev-libs/openssl.
> 
> RedHat's stated reasons for the patch were originally to avoid any patent
> concerns, but they have also morphed over time to present some "insecure"
> things from being used entirely:
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening
> "All ECC curves < 224 bits (since RHEL 6)"
> "All binary field ECC curves (since RHEL 6)"
> 
> However, the Foundation would also like to be sure that no users feel that
> patchset provides something critical to their usage of Gentoo.
> 
> If nobody speaks up as saying that the "hobble" patch is REQUIRED for their 
> use
> cases, the Foundation proposes that usage of the patchset be dropped from the
> main tree.
> 
> Any users who might be concerned about patent compliance are encouraged to do
> their own due diligence, as OpenSSL was the only Gentoo package that shipped
> this type of patch, and even Fedora's upstream did not completely patch out EC
> in other packages.
> 
> [snip]

Thanks for this. You've ended up addressing the comments & concerns I raised 
the other day
on the (slightly derailed) other thread [0]. There's a PR on this on GitHub too 
[1] to handle the
removal.

As I suspect was already clear, I support this move in the absence of new 
information
(which I suspect will not be forthcoming).

[0] 
https://archives.gentoo.org/gentoo-dev/message/99551035af66db79f60c6bd8ef7138a8
[1] https://github.com/gentoo/gentoo/pull/18894

best,
sam


signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] [RFC] Portage einfo, elog... output format change

2021-09-29 Thread Sam James


> On 29 Sep 2021, at 22:52, A Schenck  wrote:
> 
> [snip]

> Didn't expect to be the only dissenting opinion on something like this
> but. . . Some applications parse portage output looking for these
> 'zings'. At very least app-portage/kuroo does it this way; there must be
> others, right? This is obviously not the ideal way to get information
> out of portage, but it's been stable for the 15 years Kuroo has existed.
> 10-ish years ago dol-sen started some work on building and API for
> portage, but then got sucked into core portage development to the point
> of abandoning their GTK+ portage GUI porthole, which was the original
> impetus for an API, and as far as we know, the API never made it to the
> point it could replace parsing the output.
> 

Valid point, and I wish that we could get more useful information out
of Portage with easy APIs.

> It wouldn't be worth blocking progress for one application that not many
> people use, but assuming there are others that will also break with this
> change. Are we sure there's no way to support ago's (very valuable) work
> without breaking other things?

Note that this isn't just for ago's purposes. One, it makes it easier to 
generally
grep or parse portage output, but also (very importantly), it makes Portage's
output more _accessible_.

Right now, Portage is *way* too reliant on colour, which isn't very helpful
for colourblind or eyesight impaired users/developers.

> -A
> 
>> 
>> The PR doing this is: https://github.com/gentoo/portage/pull/759
>> 
>> Example screenshot:
>> https://user-images.githubusercontent.com/110765/135119090-16e9599d-1b0f-41b8-a965-a55577183ffd.png

best,
sam



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] Guidance on distributed patented software

2021-09-26 Thread Sam James


> On 25 Sep 2021, at 20:44, Joshua Kinard  wrote:
> 
>> [snip]
>> 
>> ECDSA and the NIST curves have been around since > 20 years, so it's
>> simply impossible that there are any valid patents covering those.
>> (There is of course a slight possibility that there may be patents
>> covering specific implementation details of ECDSA/NIST curves that were
>> only described later.)
> 
> Then we are either A) being too paranoid and should just drop bindist from
> the OpenSSL ebuilds, or B) we are not being paranoid enough and packages
> like dropbear/libtomcrypt need bindist added, no?  It seems we're stuck in
> the middle here because we don't have the right information.  If Red Hat or
> IBM are being non-responsive over this, then surely some other distro out
> there has already figured things out?
> 

Agreed.

Furthermore, it's not clear to me that Debian or Ubuntu are "hobbling"
their OpenSSL implementations. I may have missed something, but I
don't see anything in:
- https://salsa.debian.org/debian/openssl/-/tree/debian/unstable/debian/patches 
(or the rules file)
- 
https://git.launchpad.net/ubuntu/+source/openssl/tree/debian/patches?h=applied/ubuntu/impish
 (or the rules file)

The only thing I have found is an old bug report for Ubuntu 
(https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/597984)
referencing disabling some non-EC parts.

> 
>> I'm not entirely sure what you'd like to ask the libtomcrypt authors.
>> "We think there may be patents, but we don't know. Did you consider
>> that?"
> 
> No, actually, I was thinking something more along the lines of "Hey, are you
> aware of these supposed patent claims about ECC/ECDSA implementations that
> Red Hat says exist, and if so, did you do any research on them that you
> could possibly share that led you to feeling confident to release your
> implementation into the public domain".

I'd like to make some points about our continued use of the "hobble" patch
for at least < OpenSSL 3 too:

- RH won't publicly say what they're worried about wrt EC in OpenSSL. This 
could be to
avoid patent trolls, but this isn't really consistent with the patch being 
"enough"
to protect them - or us - anyway?

- We don't know what patents the Fedora patch is allegedly preventing us
from infringing.

- If Fedora's patch is based on legal advice, there's no reason to believe
that it necessarily applies to us.

- We have no way of verifying the correctness (or completeness) of the Fedora
patch we use because it is unclear what it is protecting against.

- Even the latest version of Fedora's hobble patch _script_ only references
patents expiring, at the latest, in 2020: 
https://src.fedoraproject.org/rpms/openssl/blob/rawhide/f/hobble-openssl.

And, as you observed, this doesn't appear to be applied consistently anyway. 
Dropbear
in Fedora appears to allow EC.

Right now, I'm far more concerned about the possible security impact of
applying patches whose correctness is not vouched for, nor do we truly
understand their purpose.

In addition, some of the changes in our current OpenSSL 1.1.x patches are 
fragile
and easily mis-applied or mis-rebased. Here's an example of such a possibly
problematic hunk:

@@ -2026,9 +1945,9 @@ int speed_main(int argc, char **argv)
 #  endif

 #  ifndef OPENSSL_NO_EC
-ecdsa_c[R_EC_P160][0] = count / 1000;
-ecdsa_c[R_EC_P160][1] = count / 1000 / 2;
-for (i = R_EC_P192; i <= R_EC_P521; i++) {
+ecdsa_c[R_EC_P224][0] = count / 1000;
+ecdsa_c[R_EC_P224][1] = count / 1000 / 2;
+for (i = R_EC_P256; i <= R_EC_P521; i++) {
 ecdsa_c[i][0] = ecdsa_c[i - 1][0] / 2;
 ecdsa_c[i][1] = ecdsa_c[i - 1][1] / 2;
 if (ecdsa_doit[i] <= 1 && ecdsa_c[i][0] == 0)
@@ -2040,7 +1959,7 @@ int speed_main(int argc, char **argv)
 }
 }
 }
-#   ifndef OPENSSL_NO_EC2M
+#   if 0

By not using easy macros where possible, we're making it far easier to have 
compile-time
or even runtime errors.

> 
> But I am open to better language.  I just don't wanna sit here not knowing.
> Someone out there has to have the right information to settle this.
> 

Yep. We consult legal advice or stop using half-measures.

Best,
sam



signature.asc
Description: Message signed with OpenPGP


[gentoo-dev] [PATCH] 2021-09-24-possible-failure-to-preserve-libraries: add item

2021-09-20 Thread Sam James
Bug: https://bugs.gentoo.org/811462
Signed-off-by: Hank Leininger 
Signed-off-by: Sam James 
---
 ...sible-failure-to-preserve-libraries.en.txt | 103 ++
 1 file changed, 103 insertions(+)
 create mode 100644 
2021-09-24-possible-failure-to-preserve-libraries/2021-09-24-possible-failure-to-preserve-libraries.en.txt

diff --git 
a/2021-09-24-possible-failure-to-preserve-libraries/2021-09-24-possible-failure-to-preserve-libraries.en.txt
 
b/2021-09-24-possible-failure-to-preserve-libraries/2021-09-24-possible-failure-to-preserve-libraries.en.txt
new file mode 100644
index 000..4b4758b
--- /dev/null
+++ 
b/2021-09-24-possible-failure-to-preserve-libraries/2021-09-24-possible-failure-to-preserve-libraries.en.txt
@@ -0,0 +1,103 @@
+Title: Possible failure to preserve libraries
+Author: Sam James 
+Author: Hank Leininger 
+Posted: 2021-09-24
+Revision: 1
+News-Item-Format: 2.0
+Display-If-Installed: sys-apps/portage
+
+Possible inconsistency of Portage's internal database (VDB) has been
+observed in some cases, usually when glibc has been upgraded to a new
+major version, but pax-utils has not yet been upgraded to a version
+compatible with it.
+
+While this largely only affects users of Portage's FEATURES="preserve-libs",
+it also affects those who may choose to use it in future given the database
+is corrupt even if certain files are not being used for preserving libraries
+at present.
+
+The full technical details and investigation can be found on a Wiki page
+[0] and on Bugzilla [1]. Work is underway to prevent this happening
+again both within Portage [2] (possibly more to come) and within the
+glibc and pax-utils ebuilds [3][4].
+
+To detect whether a system is affected, emerge the
+app-portage/recover-broken-vdb package:
+```
+$ emerge --ask --verbose --oneshot app-portage/recover-broken-vdb
+```
+which provides two tools: recover-broken-vdb-find-broken.sh and
+recover-broken-vdb.
+
+Then run recover-broken-vdb-find-broken.sh:
+```
+$ recover-broken-vdb-find-broken.sh | tee broken_vdb_packages
+```
+
+It is recommended that this check be run on all Gentoo systems.
+
+If you have any output, read on.
+
+Fixing a broken system is not always straightforward. It is strongly
+recommended to take a backup of your full system before proceeding,
+as well as a copy of /var/db/pkg (the VDB):
+
+1. A tool has been developed [5] to attempt to fix the consistency
+  of the Portage database. Using this tool to modify the VDB is NOT
+  mandatory (read the full news item before proceeding) - you can skip
+  to Step 2 if you wish, but fixing the integrity of the VDB
+  makes it as safe as reasonably possible to proceed with
+  rebuilding packages.
+
+  Run:
+  ```
+  # Take a backup of /var/db/pkg before proceeding, such as by doing:
+  $ cp /var/db/pkg /var/db/pkg.orig
+
+  # And then:
+  $ emerge --ask --verbose --oneshot --noreplace \
+   app-portage/recover-broken-vdb
+
+  $ recover-broken-vdb
+
+  # The tool will output to a random temporary directory.
+  # Inspect the results, and then update the real /var/db/pkg/
+  # by doing either:
+
+  $ recover-broken-vdb --output /var/db/pkg
+
+  # Or, manually copying the new files from the temporary directory tree
+  # into your real /var/db/pkg/ directory tree.
+  ```
+
+  If you choose not to use this recovery tool, please re-emerge
+  the affected packages (next step) and then -e @world, but
+  there is more risk attached if not all packages can
+  be rebuilt at their current versions.
+
+2. Attempt to rebuild the affected packages, first upgrading
+  app-portage/pax-utils to the latest version:
+  ```
+  $ emerge --ask --verbose --oneshot ">=app-misc/pax-utils-1.3.3"
+  $ emerge --ask --verbose --oneshot $(cat broken_vdb_packages)
+  ```
+
+Given that there are possible other side-effects of the corruption/bug,
+it is strongly recommended that if any corruption is detected, all
+packages on the system should be rebuilt, after following the above
+steps:
+```
+$ emerge --ask --emptytree @world
+```
+
+Please see the wiki [0] for a full description of the background
+of this problem and handling corner cases such as e.g. already
+being affected by system breakage [6] as a result of the bug.
+
+[0] https://wiki.gentoo.org/wiki/Project:Toolchain/Corrupt_VDB_ELF_files
+[1] https://bugs.gentoo.org/811462
+[2] https://github.com/gentoo/portage/pull/744
+[3] https://bugs.gentoo.org/811462#c6
+[4] https://bugs.gentoo.org/811462#c7
+[5] https://github.com/thesamesam/recover-broken-vdb
+[6] https://wiki.gentoo.org/wiki/Fix_my_Gentoo
-- 
2.33.0




[gentoo-dev] [PATCH] 2021-09-24-possible-failure-to-preserve-libraries: add item

2021-09-20 Thread Sam James
Bug: https://bugs.gentoo.org/811462
Signed-off-by: Hank Leininger 
Signed-off-by: Sam James 
---
 ...sible-failure-to-preserve-libraries.en.txt | 98 +++
 1 file changed, 98 insertions(+)
 create mode 100644 
2021-09-24-possible-failure-to-preserve-libraries/2021-09-24-possible-failure-to-preserve-libraries.en.txt

diff --git 
a/2021-09-24-possible-failure-to-preserve-libraries/2021-09-24-possible-failure-to-preserve-libraries.en.txt
 
b/2021-09-24-possible-failure-to-preserve-libraries/2021-09-24-possible-failure-to-preserve-libraries.en.txt
new file mode 100644
index 000..00a209d
--- /dev/null
+++ 
b/2021-09-24-possible-failure-to-preserve-libraries/2021-09-24-possible-failure-to-preserve-libraries.en.txt
@@ -0,0 +1,98 @@
+Title: Possible failure to preserve libraries
+Author: Sam James 
+Author: Hank Leininger 
+Posted: 2021-09-24
+Revision: 1
+News-Item-Format: 2.0
+Display-If-Installed: sys-apps/portage
+
+Possible inconsistency of Portage's internal database (VDB) has been
+observed in some cases, usually when glibc has been upgraded to a new
+major version, but pax-utils has not yet been upgraded to a version
+compatible with it.
+
+The full technical details and investigation can be found on a Wiki page
+[0] and on Bugzilla [1]. Work is underway to prevent this happening
+again both within Portage [2] (possibly more to come) and within the
+glibc and pax-utils ebuilds [3][4].
+
+To detect whether a system is affected, emerge the
+app-portage/recover-broken-vdb package:
+```
+$ emerge --ask --verbose --oneshot app-portage/recover-broken-vdb
+```
+which provides two tools: recover-broken-vdb-find-broken.sh and
+recover-broken-vdb.
+
+Then run recover-broken-vdb-find-broken.sh:
+```
+$ recover-broken-vdb-find-broken.sh | tee broken_vdb_packages
+```
+
+It is recommended that this check be run on all Gentoo systems.
+
+If you have any output, read on.
+
+Fixing a broken system is not always straightforward. It is strongly
+recommended to take a backup of your full system before proceeding,
+as well as a copy of /var/db/pkg (the VDB):
+
+1. A tool has been developed [5] to attempt to fix the consistency
+  of the Portage database. Using this tool is NOT mandatory
+  (read the full news item before proceeding) - you can skip
+  to Step 2 if you wish, but fixing the integrity of the VDB
+  makes it as safe as reasonably possible to proceed with
+  rebuilding packages.
+
+  Run:
+  ```
+  # Take a backup of /var/db/pkg before proceeding, such as by doing:
+  $ cp /var/db/pkg /var/db/pkg.orig
+
+  # And then:
+  $ emerge --ask --verbose --oneshot --noreplace \
+   app-portage/recover-broken-vdb
+
+  $ recover-broken-vdb
+
+  # The tool will output to a random temporary directory.
+  # Inspect the results, and then update the real /var/db/pkg/
+  # by doing either:
+
+  $ recover-broken-vdb --output /var/db/pkg
+
+  # Or, manually copying the new files from the temporary directory tree
+  # into your real /var/db/pkg/ directory tree.
+  ```
+
+  If you choose not to use this recovery tool, please re-emerge
+  the affected packages (next step) and then -e @world, but
+  there is more risk attached if not all packages can
+  be rebuilt at their current versions.
+
+2. Attempt to rebuild the affected packages, first upgrading
+  app-portage/pax-utils to the latest version:
+  ```
+  $ emerge --ask --verbose --oneshot ">=app-misc/pax-utils-1.3.3"
+  $ emerge --ask --verbose --oneshot $(cat broken_vdb_packages)
+  ```
+
+Given that there are possible other side-effects of the corruption/bug,
+it is strongly recommended that if any corruption is detected, all
+packages on the system should be rebuilt, after following the above
+steps:
+```
+$ emerge --ask --emptytree @world
+```
+
+Please see the wiki [0] for a full description of the background
+of this problem and handling corner cases such as e.g. already
+being affected by system breakage [6] as a result of the bug.
+
+[0] https://wiki.gentoo.org/wiki/Project:Toolchain/Corrupt_VDB_ELF_files
+[1] https://bugs.gentoo.org/811462
+[2] https://github.com/gentoo/portage/pull/744
+[3] https://bugs.gentoo.org/811462#c6
+[4] https://bugs.gentoo.org/811462#c7
+[5] https://github.com/thesamesam/recover-broken-vdb
+[6] https://wiki.gentoo.org/wiki/Fix_my_Gentoo
-- 
2.33.0




Re: [gentoo-dev] [PATCH 1/5] profiles/default/linux: remove busybox from @system

2021-09-09 Thread Sam James

> On 9 Sep 2021, at 20:47, Mike Gilbert  wrote:
> 
> busybox[static] was added to @system as a system recovery tool.
> 
> If the system is in such a state that a static shell is needed for
> recovery, it is likely that remote access is also broken, and the
> sysadmin will need to log into a console. At that point, they could boot
> from recovery media anyway.
> 

Indeed, @system is for essential tools, and busybox isn't one of them.

It's a choice. Maybe folks prefer toybox.

> Also, stage3 tarballs are often used to build containers, where having
> a recovery tool installed is completely pointless.
> 

Please tag Bug: https://bugs.gentoo.org/750920 in the commits (ideally all
so no context is lost for future references).

> Signed-off-by: Mike Gilbert 
> ---
> profiles/default/linux/packages | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)

best,
sam


signature.asc
Description: Message signed with OpenPGP


  1   2   3   >