[gentoo-dev] Last rites: sys-apps/baselayout-prefix

2022-08-06 Thread Fabian Groffen
# Fabian Groffen  (2022-08-06)
# superseeded by sys-apps/baselayout, removal in 30 days.  bug #836114
sys-apps/baselayout-prefix


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting

2022-07-28 Thread Fabian Groffen
On 28-07-2022 20:11:24 +0200, Fabian Groffen wrote:
> On 28-07-2022 19:57:50 +0200, Arfrever Frehtes Taifersar Arahesis wrote:
> > But this code starts with 'if [[ -z ${USERLAND} ]]', so it should not
> > be run when USERLAND="GNU".
> > 
> > USERLAND="GNU" is set in make.defaults, which is parsed by Portage
> > early (at Python side) and these values are passed by Portage to
> > ebuild.sh subprocess.
> 
> Hmmm, that would be nice, features/prefix/make.defaults includes setting
> that.
> 
> Makes me still wonder why we do this, is it a fallback for when the
> profiles aren't available (no tree yet) or something?
> 
> It exports XARGS, which only used in the ___parallel_xargs function.
> That function in turn is used in 2 places.  So all this stuff is done to
> add -r to xargs if xargs isn't BSD xargs.  Iow --no-run-if-empty, which
> is cool, but apparently not strictly necessary (fugly warning I guess at
> worst).  We're already checking xargs to support --max-procs=, so we
> could in the same go check for support of --no-run-if-empty, and drop
> the whole XARGS thing and declutter the code quite a lot :)

Sorry, I grepped wrong.  XARGS is used throughout the code, so it needs
to be initialised in isolated-functions.sh.  It could be initialised by
simply probing instead of relying on uname though.

As for USERLAND, emerge-webrsync and delta-webrsync appear to use it,
but they retrieve it from portageq, so I guess USERLAND is not really
used in Portage itself.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting

2022-07-28 Thread Fabian Groffen
On 28-07-2022 19:57:50 +0200, Arfrever Frehtes Taifersar Arahesis wrote:
> 2022-07-28 17:47 UTCに、Fabian Groffen は書いた:
> > bin/isolated-functions.sh does the following bit:
> >
> > if [[ -z ${USERLAND} ]] ; then
> >case $(uname -s) in
> >*BSD|DragonFly)
> >export USERLAND="BSD"
> >;;
> >*)
> >export USERLAND="GNU"
> >;;
> >esac
> > fi
> >
> > (after which it uses USERLAND for a single purpose, to export XARGS to
> > either GNU '[g]xargs -r', or BSD 'xargs')
> >
> > This bit is problematic for Prefix, because Prefix may run on *BSD, but
> > use USERLAND=GNU.
> 
> But this code starts with 'if [[ -z ${USERLAND} ]]', so it should not
> be run when USERLAND="GNU".
> 
> USERLAND="GNU" is set in make.defaults, which is parsed by Portage
> early (at Python side) and these values are passed by Portage to
> ebuild.sh subprocess.

Hmmm, that would be nice, features/prefix/make.defaults includes setting
that.

Makes me still wonder why we do this, is it a fallback for when the
profiles aren't available (no tree yet) or something?

It exports XARGS, which only used in the ___parallel_xargs function.
That function in turn is used in 2 places.  So all this stuff is done to
add -r to xargs if xargs isn't BSD xargs.  Iow --no-run-if-empty, which
is cool, but apparently not strictly necessary (fugly warning I guess at
worst).  We're already checking xargs to support --max-procs=, so we
could in the same go check for support of --no-run-if-empty, and drop
the whole XARGS thing and declutter the code quite a lot :)

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting

2022-07-28 Thread Fabian Groffen
Hi,

bin/isolated-functions.sh does the following bit:

if [[ -z ${USERLAND} ]] ; then
   case $(uname -s) in
   *BSD|DragonFly)
   export USERLAND="BSD"
   ;;
   *)
   export USERLAND="GNU"
   ;;
   esac
fi

(after which it uses USERLAND for a single purpose, to export XARGS to
either GNU '[g]xargs -r', or BSD 'xargs')

This bit is problematic for Prefix, because Prefix may run on *BSD, but
use USERLAND=GNU.  The problem is theoretical at this point,
occasionally people come in and want to use Prefix on *BSD again.

Now I think Gentoo/BSD is theoretical too.  It officially is no longer
maintained[1].

So, question here is, do we want to retain this bit of historical
compatibility work (also check things like gsed wrapper), or can it go,
basically removing a potential problem for Gentoo Prefix?



[1] https://wiki.gentoo.org/wiki/Gentoo_FreeBSD



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Rust is here to eat your Pythonz

2022-07-27 Thread Fabian Groffen
On 27-07-2022 13:30:32 +0200, Michał Górny wrote:
> On Wed, 2022-07-27 at 12:52 +0200, Fabian Groffen wrote:
> > On 26-07-2022 08:42:57 +0200, Michał Górny wrote:
> > > Hi, everyone.
> > > 
> > > Just a quick FYI: since Rust is going to be marked stable on the last
> > > architecture (sparc) that it's going to support in Gentoo, we're going
> > > to start cleaning up old dev-python/cryptography soon.  If you don't
> > > want Rust, removing cryptography entirely will be your only option.
> > 
> > I believe this isn't always a "don't want", there's systems on which it
> > simply is impossible to get Rust (ppc64-linux-musl for example).  How
> > much does "removing cryptography" impact an Gentoo system wrt
> > availability of packages?  I see pip in the list, so I expect the Python
> > env will be seriously crippled?  An estimation of the impact would be
> > appreciated here.
> > 
> 
> Check the wd40 profile, all relevant masks are included there.

That helps, thanks!
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Rust is here to eat your Pythonz

2022-07-27 Thread Fabian Groffen
On 26-07-2022 08:42:57 +0200, Michał Górny wrote:
> Hi, everyone.
> 
> Just a quick FYI: since Rust is going to be marked stable on the last
> architecture (sparc) that it's going to support in Gentoo, we're going
> to start cleaning up old dev-python/cryptography soon.  If you don't
> want Rust, removing cryptography entirely will be your only option.

I believe this isn't always a "don't want", there's systems on which it
simply is impossible to get Rust (ppc64-linux-musl for example).  How
much does "removing cryptography" impact an Gentoo system wrt
availability of packages?  I see pip in the list, so I expect the Python
env will be seriously crippled?  An estimation of the impact would be
appreciated here.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] phase-functions: make ED, EROOT read-only

2022-07-26 Thread Fabian Groffen
On 25-07-2022 19:43:21 -0400, Mike Gilbert wrote:
> On Mon, Jul 25, 2022 at 1:03 PM Fabian Groffen  wrote:
> >
> > bin/phase-functions.sh: make ED and EROOT read-only too
> >
> > Like D, make ED and EROOT read-only vars.
> 
> Makes sense.

https://github.com/gentoo/portage/pull/870

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] misc-functions: Prefix fixes

2022-07-26 Thread Fabian Groffen
On 25-07-2022 19:42:51 -0400, Mike Gilbert wrote:
> On Mon, Jul 25, 2022 at 12:47 PM Fabian Groffen  wrote:
> >
> > bin/misc-functions.sh: some Prefix fixes
> >
> > - ED needs not to exist, whereas D does, so ensure we check for that,
> >   and create ED if absent, necessary for further checks to succeed
> > - use EPREFIX in INSTALL_MASK
> 
> Seems good to me.

https://github.com/gentoo/portage/pull/870 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] world-writable: tune for Prefix

2022-07-26 Thread Fabian Groffen
On 25-07-2022 19:39:38 -0400, Mike Gilbert wrote:
> On Mon, Jul 25, 2022 at 12:41 PM Fabian Groffen  wrote:
> >
> > bin/install-qa-check.d/90world-writable: include EPREFIX in reports
> >
> > It is much less confusing and consistent to report full paths including
> > the leading EPREFIX.
> 
> Makes sense to me.

https://github.com/gentoo/portage/pull/869

thanks

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] multilib-strict: fix for Prefix

2022-07-26 Thread Fabian Groffen
On 25-07-2022 19:38:57 -0400, Mike Gilbert wrote:
> On Mon, Jul 25, 2022 at 12:26 PM Fabian Groffen  wrote:
> >
> > bin/install-qa-check.d/80multilib-strict: use file/find from Prefix
> >
> > diff --git a/bin/install-qa-check.d/80multilib-strict 
> > b/bin/install-qa-check.d/80multilib-strict
> > index afd223250..3db4ecce3 100644
> > --- a/bin/install-qa-check.d/80multilib-strict
> > +++ b/bin/install-qa-check.d/80multilib-strict
> > @@ -1,7 +1,7 @@
> >  # Strict multilib directory checks
> >  multilib_strict_check() {
> > if has multilib-strict ${FEATURES} && \
> > -  [[ -x /usr/bin/file && -x /usr/bin/find ]] && \
> > +  [[ -x ${EPREFIX}/usr/bin/file && -x ${EPREFIX}/usr/bin/find ]] 
> > && \
> >[[ -n ${MULTILIB_STRICT_DIRS} && -n ${MULTILIB_STRICT_DENY} ]]
> >     then
> > rm -f "${T}/multilib-strict.log"
> 
> Seems good to me.

https://github.com/gentoo/portage/pull/868

Thanks

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets

2022-07-26 Thread Fabian Groffen
On 26-07-2022 04:01:37 +0100, Sam James wrote:
> 
> 
> > On 26 Jul 2022, at 00:33, Mike Gilbert  wrote:
> > 
> > On Mon, Jul 25, 2022 at 11:38 AM Fabian Groffen  wrote:
> >> 
> >> bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects
> >> 
> >> Check for dylib on Darwin, so on everything else.
> >> 
> >> Signed-off-by: Fabian Groffen 
> >> 
> >> diff --git a/bin/install-qa-check.d/80libraries 
> >> b/bin/install-qa-check.d/80libraries
> >> index 8dc35bb87..a477ec9cb 100644
> >> --- a/bin/install-qa-check.d/80libraries
> >> +++ b/bin/install-qa-check.d/80libraries
> >> @@ -140,7 +140,9 @@ lib_check() {
> >>local abort="no"
> >>local a s
> >>for a in "${ED%/}"/usr/lib*/*.a ; do
> >> -   s=${a%.a}.so
> >> +   [[ ${CHOST} == *-darwin* ]] \
> >> +   && s=${a%.a}.dylib \
> >> +       || s=${a%.a}.so
> > 
> > I would find this much easier to read if you converted it to an
> > if/else statement instead of chaining && and ||.
> 
> Yes, please.

https://github.com/gentoo/portage/pull/867

Thanks


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check

2022-07-26 Thread Fabian Groffen
On 25-07-2022 19:29:35 -0400, Mike Gilbert wrote:
> Could you please create a PR at https://github.com/gentoo/portage so
> that the CI system can test the changes for this patch series?

https://github.com/gentoo/portage/pull/866

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check

2022-07-26 Thread Fabian Groffen
On 25-07-2022 19:50:41 +0200, Ulrich Mueller wrote:
> >>>>> On Mon, 25 Jul 2022, Fabian Groffen wrote:
> 
> > @@ -50,6 +51,16 @@ if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" 
> > ]]; then
> > __helpers_die "${helper} used with \${D} or \${ED}"
> > exit 1
> >  fi
> > +if [[ -n ${EPREFIX} && \
> > +   ${_E_INSDESTTREE_#${EPREFIX}} != "${_E_INSDESTTREE_}" ]];
> 
> The semicolon is redundant.

removed, thanks


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL

2022-07-26 Thread Fabian Groffen
On 26-07-2022 09:03:18 +0200, Florian Schmaus wrote:
> On 26.07.22 05:00, Sam James wrote:
> >> On 25 Jul 2022, at 16:28, Fabian Groffen  wrote:
> >>
> >> bin/ebuild-helpers/emake: force SHELL to be set
> >>
[snip]
> >>
> >> diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
> >> index 60718a2e4..21da85845 100755
> >> --- a/bin/ebuild-helpers/emake
> >> +++ b/bin/ebuild-helpers/emake
> >> @@ -12,7 +12,7 @@
> >> source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
> >>
> >> cmd=(
> >> -  ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
> >> +  ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
> >> )
> >>
> >> if [[ ${PORTAGE_QUIET} != 1 ]] ; then
> >>
> > 
> > I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh to 
> > avoid using
> > an ancient host sh?
> 
> I was about to write the same (also using EPREFIX, but EBROOT seems what 
> you want, as you figured).
> 
> But then I wondered if "make SHELL=$BROOT/bin/sh" wouldn't override 
> explicitly set SHELL values in Makefiles. Assume a package has
> 
> SHELL = /bin/zsh
> 
> in one of its Makefiles. Then emake would reset this to 'sh'. Which 
> appears like it could cause build issues.
> 
> If this is the case, then I am not sure what we can do about it. It 
> appears fragile, if not impossible, to ask 'make' which value for SHELL 
> it would assume, so that emake could adjust the path. Another option 
> could be that affected packages define a variable in their ebuild, e.g. 
> EMAKE_SHELL="zsh", which emake could extend with BROOT before passing 
> the resulting value as SHELL to make.

So, I can also envision we drop this patch, and I see if I can patch
make(1) to use $EPREFIX/bin/sh instead of /bin/sh by default.  Not sure,
but this would retain the behaviour Portage is doing now for non-Prefix,
and would get the behaviour we want in Prefix.

On an alternative note, there is CONFIG_SHELL (used for setting which shell
to use with configure), which I think in many cases bleeds through to
make, but should there be a MAKE_SHELL perhaps as well?  Then the
default would be pretty much ok.

(We never ran into any problems forcing SHELL to bash in Prefix, but
perhaps that's not a representative test for the whole of Gentoo.)

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH] phase-functions: make ED, EROOT read-only

2022-07-25 Thread Fabian Groffen
bin/phase-functions.sh: make ED and EROOT read-only too

Like D, make ED and EROOT read-only vars.

Signed-off-by: Fabian Groffen 

diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index ccf7eeea7..212b19fc1 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -12,7 +12,7 @@ PORTAGE_READONLY_METADATA="BDEPEND DEFINED_PHASES DEPEND 
DESCRIPTION
PDEPEND RDEPEND REPOSITORY RESTRICT SLOT SRC_URI"
 
 PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE EBUILD_PHASE_FUNC \
-   EBUILD_SH_ARGS EMERGE_FROM FILESDIR MERGE_TYPE \
+   EBUILD_SH_ARGS ED EMERGE_FROM EROOT FILESDIR MERGE_TYPE \
PM_EBUILD_HOOK_DIR \
PORTAGE_ACTUAL_DISTDIR PORTAGE_ARCHLIST PORTAGE_BASHRC  \
PORTAGE_BINPKG_FILE PORTAGE_BINPKG_TAR_OPTS PORTAGE_BINPKG_TMPFILE \

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH] misc-functions: Prefix fixes

2022-07-25 Thread Fabian Groffen
bin/misc-functions.sh: some Prefix fixes

- ED needs not to exist, whereas D does, so ensure we check for that,
  and create ED if absent, necessary for further checks to succeed
- use EPREFIX in INSTALL_MASK

Signed-off-by: Fabian Groffen 

diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index c8bac08e7..8fcc23588 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -19,6 +19,8 @@ source "${PORTAGE_BIN_PATH}/ebuild.sh" || exit 1
 install_symlink_html_docs() {
if ! ___eapi_has_prefix_variables; then
local ED=${D}
+   else
+   [[ ! -d ${ED} && -d ${D} ]] && dodir /
fi
cd "${ED}" || die "cd failed"
# Symlink the html documentation (if DOC_SYMLINKS_DIR is set in 
make.conf)
@@ -83,7 +87,7 @@ install_qa_check() {
local EPREFIX= ED=${D}
fi
 
-   cd "${ED}" || die "cd failed"
+   cd "${D}" || die "cd failed"
 
# Collect the paths for QA checks, highest prio first.
paths=(
@@ -367,7 +718,7 @@ preinst_mask() {
local f x
for f in man info doc; do
if has no${f} ${FEATURES}; then
-   INSTALL_MASK+=" /usr/share/${f}"
+       INSTALL_MASK+=" ${EPREFIX}/usr/share/${f}"
fi
done
 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH] world-writable: tune for Prefix

2022-07-25 Thread Fabian Groffen
bin/install-qa-check.d/90world-writable: include EPREFIX in reports

It is much less confusing and consistent to report full paths including
the leading EPREFIX.

Signed-off-by: Fabian Groffen 

diff --git a/bin/install-qa-check.d/90world-writable 
b/bin/install-qa-check.d/90world-writable
index 820683bd6..90b961a86 100644
--- a/bin/install-qa-check.d/90world-writable
+++ b/bin/install-qa-check.d/90world-writable
@@ -2,7 +2,7 @@
 
 world_writable_check() {
# Now we look for all world writable files.
-   local unsafe_files=$(find "${ED}" -type f -perm -2 | sed -e 
"s:^${ED}:/:")
+   local unsafe_files=$(find "${ED}" -type f -perm -2 | sed -e 
"s:^${D}:/:")
local OLDIFS x prev_shopts=$-
 
OLDIFS=$IFS
@@ -19,7 +21,7 @@ world_writable_check() {
eqawarn
fi
 
-   local unsafe_files=$(find "${ED}" -type f '(' -perm -2002 -o -perm 
-4002 ')' | sed -e "s:^${ED}:/:")
+   local unsafe_files=$(find "${ED}" -type f '(' -perm -2002 -o -perm 
-4002 ')' | sed -e "s:^${D}:/:")
if [[ -n ${unsafe_files} ]] ; then
    eqawarn "QA Notice: Unsafe files detected (set*id and world 
writable)"
 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH] multilib-strict: fix for Prefix

2022-07-25 Thread Fabian Groffen
bin/install-qa-check.d/80multilib-strict: use file/find from Prefix

diff --git a/bin/install-qa-check.d/80multilib-strict 
b/bin/install-qa-check.d/80multilib-strict
index afd223250..3db4ecce3 100644
--- a/bin/install-qa-check.d/80multilib-strict
+++ b/bin/install-qa-check.d/80multilib-strict
@@ -1,7 +1,7 @@
 # Strict multilib directory checks
 multilib_strict_check() {
if has multilib-strict ${FEATURES} && \
-  [[ -x /usr/bin/file && -x /usr/bin/find ]] && \
+  [[ -x ${EPREFIX}/usr/bin/file && -x ${EPREFIX}/usr/bin/find ]] && \
   [[ -n ${MULTILIB_STRICT_DIRS} && -n ${MULTILIB_STRICT_DENY} ]]
then
        rm -f "${T}/multilib-strict.log"

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets

2022-07-25 Thread Fabian Groffen
bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects

Check for dylib on Darwin, so on everything else.

Signed-off-by: Fabian Groffen 

diff --git a/bin/install-qa-check.d/80libraries 
b/bin/install-qa-check.d/80libraries
index 8dc35bb87..a477ec9cb 100644
--- a/bin/install-qa-check.d/80libraries
+++ b/bin/install-qa-check.d/80libraries
@@ -140,7 +140,9 @@ lib_check() {
local abort="no"
local a s
for a in "${ED%/}"/usr/lib*/*.a ; do
-   s=${a%.a}.so
+   [[ ${CHOST} == *-darwin* ]] \
+   && s=${a%.a}.dylib \
+   || s=${a%.a}.so
if [[ ! -e ${s} ]] ; then
s=${s%usr/*}${s##*/usr/}
if [[ -e ${s} ]] ; then

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH] emake: explicitly set SHELL

2022-07-25 Thread Fabian Groffen
bin/ebuild-helpers/emake: force SHELL to be set

On Prefix systems /bin/sh can be anything, including very ancient.  So
ensure we're running with bash, since that's what Gentoo Linux is
expecting /bin/sh to be (by default, at least).

Provide a fallback for the (near impossible) case that we use a bash
that doesn't set BASH, or when we don't use bash at all.  This is not
expected, though, as we explicitly require bash throughout all Portage,
so we don't really care about using a non-Prefixed one, for this really
shouldn't happen.

Signed-off-by: Fabian Groffen 

diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
index 60718a2e4..21da85845 100755
--- a/bin/ebuild-helpers/emake
+++ b/bin/ebuild-helpers/emake
@@ -12,7 +12,7 @@
 source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
 
 cmd=(
-   ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
+   ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
 )
 
 if [[ ${PORTAGE_QUIET} != 1 ]] ; then

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] usage of /bin/bash in shebangs

2022-07-25 Thread Fabian Groffen
On 25-07-2022 08:09:02 -0700, Zac Medico wrote:
> On 7/24/22 23:17, Fabian Groffen wrote:
> > On 24-07-2022 13:58:31 -0700, Zac Medico wrote:
> >> On 7/24/22 12:29, Fabian Groffen wrote:
> >>> Hi,
> >>>
> >>> Quick question, I noticed that portage uses /bin/bash hardcoded in
> >>> shebang of scripts, while it uses /usr/bin/env python for python
> >>> executable files.
> >>>
> >>> Is there anything against using /usr/bin/env bash for shell scripts?
> >>> Changing this would help for Prefix.
> >>
> >>
> >> I can't think of any reason not to do this.
> > 
> > Do you want to see a patch, or is it OK to push this myself?
> 
> It's OK to just push this yourself. Thanks!

Pushed, thanks, b716e3591


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check

2022-07-25 Thread Fabian Groffen
bin/ebuild-helpers/doins: fix D check, add EPREFIX check

ED = D/EPREFIX, so checking for ED includes EPREFIX, which when this is
absent fails to check for D.  Simply check for D instead, which catches
both the case for D and ED.

Add check for usage of EPREFIX, like for using D with helpers.

Signed-off-by: Fabian Groffen 

diff --git a/bin/ebuild-helpers/doins b/bin/ebuild-helpers/doins
index 24fe48121..4315a038f 100755
--- a/bin/ebuild-helpers/doins
+++ b/bin/ebuild-helpers/doins
@@ -42,7 +42,7 @@ if ! ___eapi_has_prefix_variables; then
export ED="${D}"
 fi
 
-if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" ]]; then
+if [[ ${_E_INSDESTTREE_#${D}} != "${_E_INSDESTTREE_}" ]]; then
__vecho "---" 1>&2
__vecho "You should not use \${D} or \${ED} with helpers." 1>&2
__vecho "  --> ${_E_INSDESTTREE_}" 1>&2
@@ -50,6 +51,16 @@ if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" ]]; 
then
__helpers_die "${helper} used with \${D} or \${ED}"
exit 1
 fi
+if [[ -n ${EPREFIX} && \
+   ${_E_INSDESTTREE_#${EPREFIX}} != "${_E_INSDESTTREE_}" ]];
+then
+   __vecho "---" 1>&2
+   __vecho "You should not use \${EPREFIX} with helpers." 1>&2
+   __vecho "  --> ${_E_INSDESTTREE_}" 1>&2
+   __vecho "---" 1>&2
+   __helpers_die "${helper} used with \${EPREFIX}"
+   exit 1
+fi
 
 if ___eapi_doins_and_newins_preserve_symlinks; then
DOINS_ARGS+=( --preserve_symlinks )

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] usage of /bin/bash in shebangs

2022-07-25 Thread Fabian Groffen
On 24-07-2022 13:58:31 -0700, Zac Medico wrote:
> On 7/24/22 12:29, Fabian Groffen wrote:
> > Hi,
> > 
> > Quick question, I noticed that portage uses /bin/bash hardcoded in
> > shebang of scripts, while it uses /usr/bin/env python for python
> > executable files.
> > 
> > Is there anything against using /usr/bin/env bash for shell scripts?
> > Changing this would help for Prefix.
> 
> 
> I can't think of any reason not to do this.

Do you want to see a patch, or is it OK to push this myself?

Thanks,
Fabian



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] usage of /bin/bash in shebangs

2022-07-24 Thread Fabian Groffen
Hi,

Quick question, I noticed that portage uses /bin/bash hardcoded in
shebang of scripts, while it uses /usr/bin/env python for python
executable files.

Is there anything against using /usr/bin/env bash for shell scripts?
Changing this would help for Prefix.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] proposal

2022-07-06 Thread Fabian Groffen
On 06-07-2022 15:50:30 +0200, Michał Górny wrote:
> On Mon, 2022-07-04 at 16:19 +0200, Florian Schmaus wrote:
> > I'd like to propose a new metadata XML element for packages:
> > 
> >  
> > 
> > Maintainers can signal to other developers (and of course contributors 
> > in general) that they are happy with others to make changes to the 
> > ebuilds without prior consultation of the maintainer.
> 
> I don't think adding such an element is a good idea.  In my opinion,
> this will strengthen the assumption that "unless otherwise noted, you
> don't dare touch anything" (even though that's not your goal).  "Common
> sense" should really be good enough for almost everything.

Right, "common sense".  The problem with that one, is that "common" is
not as "common" as you think it is.  Ask a bunch of people, and you'll
find that what they consider "common" isn't the same.

So, if you do this, then please define clearly what you think is OK.
For example for me:

- feel free to add patches necessary for operation
- feel free to fix constructs (like an if or case that should apply for
  something else/different/mode)
- feel free to fix typos
- please do not needlessly change style: if you do not "maintain" the
  ebuild, respect the style of the maintainer, so only add the changes
  you need, keep it minimal, respect the original even though you don't
  like it (and don't use QA as an excuse to change style)
- when you make a change, make sure you check for bugs in the following
  days, so you can cleanup yourself should there be fallout

> I mean, I do realize that 10 years ago, in the golden years of Gentoo,
> it was considered normal for developers to be like "my package, my
> fortress, don't you dare add systemd unit or I will retire" but today I
> think we're aiming for a more welcoming developer base, and I think
> we're actually going in that direction.  What I'm afraid is that some
> people will use this as an excuse to push back once again.

Not sure I have the same memories of how it used to be 10 years ago.  I
actually think it is pretty much the same as it is now, as it was then.
Different and fewer people, but still different
preferences/opinions/common sense.

> Can you really think of a case when common sense really, really wouldn't
> work?  I mean, sure, we all make mistakes but we should be able to learn
> from them and do better next time.  This also implies package
> maintainers learning that they're not the only people who will ever
> touch the package in question and starting to document the pitfalls.

Honestly, I've never been a fan of "maintainership".  It basically is
some sort of "sign" that says "beware for the dog, stay away".  However,
it's true that sometimes people really delve into a package, and thereby
know very much how it works, and what you should/should not do.
Something like LLVM is a good example, maybe.  Anyway, in such
situation, I think extreme care should be taken by non-maintainers.
Dunno how to best indicate that, and/or if that's feasible -- like you
said, it quickly ends up being an excuse for declaring a package to be
off-limits.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] last rites: dev-python/thrift

2022-04-01 Thread Fabian Groffen
Dropped this mask again, I missed a dep in the tree.

Fabian

On 01-04-2022 13:53:24 +0200, Fabian Groffen wrote:
> # Fabian Groffen  (2022-04-01)
> # unused package, not enabling tests, bug #796830
> # Removal in 30 days
> dev-python/thrift


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] last rites: dev-python/thrift

2022-04-01 Thread Fabian Groffen
On 01-04-2022 16:59:40 +0500, Anna Vyalkova wrote:
> On 2022-04-01 13:53, Fabian Groffen wrote:
> > # Fabian Groffen  (2022-04-01)
> > # unused package, not enabling tests, bug #796830
> > # Removal in 30 days
> > dev-python/thrift
> 
> It has a reverse dependency in GURU
> dev-python/jaeger-client
> 
> It will be needed if OpenStack packages are restored some day.

Can we move/copy the package to GURU?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] last rites: dev-python/thrift

2022-04-01 Thread Fabian Groffen
# Fabian Groffen  (2022-04-01)
# unused package, not enabling tests, bug #796830
# Removal in 30 days
dev-python/thrift

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Policy on conditional patching

2022-03-28 Thread Fabian Groffen
Hi,

On 28-03-2022 13:05:03 +0200, Thomas Bracht Laumann Jespersen wrote:
> Hi!
> 
> I've been working on a new section in the devmanual regarding conditional
> patching. In a PR [0] Sam suggested adding a section to clarify that 
> conditional
> patching should be avoided, because it can quickly become a maintenance and
> testing burden.
> 
> The devmanual PR is here: [1]
> 
> Ulrich points out that conditional patching is actually suggested elsewhere, 
> and
> that actively discouraging conditional patching is a policy change, which 
> should
> be brought up on this list. So this is what I'm doing now. He also pointed 
> out a
> comment in eutils.eclass re [2] (the comment now lives in epatch.eclass).
> 
> The overall policy (proposal, I guess) is something like:
> 
>  * Patches should be written such that they affect behaviour correctly based 
> on
>e.g. build time definitions or config options, rather than USE flags
>directly.
> 
>  * They should be applied unconditionally, so that, for version bumps and 
> other
>development happening with a package, any failure to apply the patch will 
> be
>caught by the developer.
> 
>  * Patching is ideally only done to make the package in question build 
> properly,
>and should not be done to modify the runtime behaviour of the package. 
> (This
>is what USE flags and configuration options of the package are for.)
> 
> Sam made a specific point re musl: "for e.g. musl patches, we want a portable
> fix, not a hack which is only applied for musl"
> 
> Feedback on this very welcome. I'm grappling a bit with the exact wording to 
> go
> for, so input on that is also appreciated.
> 
> I think my question to this list is: Should it be policy that conditional
> patching is to be avoided?

We really don't want conditional patches, but I from my experience
reality is that sometimes you have to.  Therefore this should remain
possible.  I have no problems with highly discouraging conditional
patching.

About your other points, I think they are kind of debatable.  Gentoo
wants to be close to upstream, but with your set of rules, one can't
e.g. add hpn patches to ssh, which would be really silly, as the point
of Gentoo is that since you build from source, you can actually apply
such patches, conditionally if they don't have a means to be disabled.

In other words, I think the gist of your points is to be in an ideal
world, but unfortunately reality is far from it.  That said, repeating
myself, nothing wrong with discouraging quick 'n' dirty, for as long as
it remains a big fat warning and advice.

My €0.02
Fabian

> [0]: https://github.com/gentoo/gentoo/pull/24709#discussion_r832361402
> [1]: https://github.com/gentoo/devmanual/pull/281
> [2]: 
> https://gitweb.gentoo.org/archive/repo/gentoo-2.git/tree/eclass/eutils.eclass?id=50e8beda904760c773e5c67fdfe8242255e13495#n175
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] Last rites: www-apps/agendav

2022-03-16 Thread Fabian Groffen
# Fabian Groffen  (2022-03-16)
# little activity upstream, doesn't work with PHP 8.0
# Removal in 30 days
www-apps/agendav

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Changing the VDB format

2022-03-14 Thread Fabian Groffen
Hi,

I've recently been thinking about this too.

On 13-03-2022 18:06:21 -0700, Matt Turner wrote:
> The VDB uses a one-file-per-variable format. This has some
> inefficiencies, with many file systems. For example the 'EAPI' file
> that contains a single character will consume a 4K block on disk.
> I recommend json and think it is the best choice because:

[snip]

> - json provides the smallest on-disk footprint
> - json is part of Python's standard library (so is yaml, and toml will
> be in Python 3.11)
> - Every programming language has multiple json parsers
> -- lots of effort has been spent making them extremely fast.

I would like to suggest to use "tar".  The reason behind this is a bit
convoluted, but I try to be as clear and sound as I can:
- "new style" bin-packages use tar too
- tar-file allows to keep all individual files/members, e.g. for legacy
  tools to unpack and look at the VDB that way
- tar-file allows streaming, so single file read, for efficient
  retrieval
- single tar-file for entire VDB, allows to make it "atomic", one can
  modify tar archives lateron to add new vdb entries, or perform
  updates -- again, without inplace (e.g. memory backing) this could be
  done atomic)
- tar-file could be used for (rsync) tree metadata (md5-cache) in the
  same way, e.g. re-use streaming approach, or unpack for legacy tools
- tar-file could be used for Packages file, instead of flat file with
  keys, basically just write VDB entries with some additional keys, very
  similar in practise.
- tar-files are slightly easier to manage from command line, tools to do
  so exist for a long time and are installed.  (jq isn't pulled in by
  @system these days, I think)
- tar-files can easily (optionally) be compressed retaining streaming
  abilities (this is for these usages very likely to pay off), a much
  higher dictionary benefit for a single tar vs many files.
- single tar-file is much more efficient to GPG-sign (which would allow
  some securing of the VDB, not sure if useful though)
- going back to the first point, vdb entry from binary package could
  simply be dropped into the vdb tar, and vice-versa
- going back to metadata, dep-resolving could simply load the entire
  system available/installed packages with two reads in memory (if it
  has enough of that -- pretty common these days), which should allow
  for vast speedups, especially on cold(ish) filesystems.

> I think we would have a significant time period for the transition. I
> think I would include support for the new format in Portage, and ship
> a tool with portage to switch back and forth between old and new
> formats on-disk. Maybe after a year, drop the code from Portage to
> support the old format?

Here I believe that with tar-format, initially code could be written to
instead of accessing a file directly, it could open up the tar-file,
locate the member it needs, and then retrieve that instead.  This is a
bit naive, but probably sort of managable, and allows to having a switch
that specifies which format to write.  It's easy to detect which form
you have automatically.  E.g. nothing has to change for users unless
they actively make a change for it.

Like you, I think the main reason for doing this should be performance,
basically allowing faster operations.

I feel though that we should aim to use a single solution to maintain a
number of "trees" that we have: metadata, vdb, Packages/binpkgs, for
they all seem to exhibit a similar (IO) behaviour when being employed.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] Re: Deprecating repoman

2022-03-12 Thread Fabian Groffen
Just wondering, is there a "migration guide" or something?  I've never
used pkg* since joining in 2005.  I can derive some things from the
first look at the below commit, but an "expert opinion" to just map the
standard things from repoman to appropriate commands would be nice.

Are those pkg* keyworded *everywhere*, by the way?

Thanks,
Fabian


On 11-03-2022 13:19:34 -0800, Matt Turner wrote:
> I've filed a PR against devmanual.git to remove references to repoman
> and replace them with references to pkgdev where appropriate. Most
> everywhere already had pkgdev/pkgcheck text in place so there wasn't
> much to do. See: https://github.com/gentoo/devmanual/pull/274
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] Last rites: sys-apps/tapi

2022-02-27 Thread Fabian Groffen
# Fabian Groffen  (2022-02-27)
# Masked for removal, needs updates, significant amount of work, no
# Clang toolchain available to test with
# Removal on 2022-03-29.  Bug #834306
sys-apps/tapi

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] last-rite: dev-perl/Mac-Pasteboard

2022-01-29 Thread Fabian Groffen
# Fabian Groffen  (2022-01-29)
# Fails to compile with GCC on macOS.  No revdeps.
# Removal on 2022-02-28.  Bug #832309.
dev-perl/Mac-Pasteboard

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] making rust-bin ordered first in virtual/rust

2022-01-21 Thread Fabian Groffen
On 21-01-2022 11:29:54 +0100, Peter Böhm wrote:
> Dear Developers,
> 
> I like your goal to make Gentoo more user-friendly but Gentoo is a source
> based distribution and I dont like binary versions as a default. My question
> is:
> 
> Who has problems with "big" packages like rust or firefox ?
> 
> Only User which doesnt know there is a binary version. So, in every case we
> need to describe it in our AMD64 handbook.
> 
> Am Freitag, 21. Januar 2022, 10:22:14 CET schrieb Mart Raudsepp:
> > Anyhow, my vote is to default to rust-bin - people can easily be told
> > to move to dev-lang/rust at their convenience and then explicitly
> > depclean rust-bin.
> 
> I am dreaming about another solution where this is not needed:
> 
> In our /etc/portage/make.conf we can have a new:
> 
> MAKEBIN="rust firefox"
> 
> ... resulting in an automatic switch to the binary version of all included
> packages ... of course this is also as recommendation in our AMD64 handbook
> (with a clue to delete it if not desired).

or ... if we could have Portage check the requirements for building a
package, and if it cannot be met, that it tries to resolve the || case,
which would be the -bin variant in this case.

Not sure if the information is available to Portage at dependency
resolution time though.

Fabian

> 
> Kind reagards,
> Peter
> 
> 
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal

2021-12-04 Thread Fabian Groffen
On 04-12-2021 10:24:23 +0100, Michał Górny wrote:
> On Sat, 2021-12-04 at 10:15 +0100, Fabian Groffen wrote:
> > On 04-12-2021 10:13:09 +0100, Michał Górny wrote:
> > > On Sat, 2021-12-04 at 09:56 +0100, Fabian Groffen wrote:
> > > > Why don't you change your color.map instead of changing the default for
> > > > everyone?
> > > 
> > > Why should we keep a stupid default?  Should we optimize Gentoo for
> > > people who don't want to be able to read Portage's output?
> > 
> > You're assuming everyone uses the terminal in the way you do.  I simply
> > don't think that's how the world looks like.
> 
> On the other hand, you're assuming that everyone uses the terminal
> in the way you do.

It eludes me how you came to that conclusion.

> > No need for calling things stupid, IMO.
> 
> Using dark blue on black background is stupid.

... then don't use black background or dark blue text?

Now, if you would make a supported claim that all terminals we install
use a black background by default, your change becomes more valid.

However, we then still don't know if people leave that default or use
something else, but we could make some educated guess about the amount
of people not changing the default.

My point, because I think this wasn't clear to you, is and always was,
how many people is this change going to be disruptive to.  And should
we make a hint to users when they install this version of Portage that
they can revert/change this by altering color.map (and how)?

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal

2021-12-04 Thread Fabian Groffen
On 04-12-2021 10:13:09 +0100, Michał Górny wrote:
> On Sat, 2021-12-04 at 09:56 +0100, Fabian Groffen wrote:
> > Why don't you change your color.map instead of changing the default for
> > everyone?
> 
> Why should we keep a stupid default?  Should we optimize Gentoo for
> people who don't want to be able to read Portage's output?

You're assuming everyone uses the terminal in the way you do.  I simply
don't think that's how the world looks like.

No need for calling things stupid, IMO.

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal

2021-12-04 Thread Fabian Groffen
Why don't you change your color.map instead of changing the default for
everyone?

Thanks,
Fabian

On 04-12-2021 05:48:49 +0100, Michał Górny wrote:
> The "darkblue" color is often barely visible on dark terminals which
> makes reading emerge output really hard (I basically have to copy-paste
> it a lot in order to be able to read it at all).  Replace it with teal
> that does not seem to have any significant use in the output.
> 
> Signed-off-by: Michał Górny 
> ---
>  lib/portage/output.py | 6 +++---
>  man/color.map.5   | 6 +++---
>  2 files changed, 6 insertions(+), 6 deletions(-)
> 
> 
> The before-after screenshot can be found on the PR:
> https://github.com/gentoo/portage/pull/776
> 
> 
> diff --git a/lib/portage/output.py b/lib/portage/output.py
> index 42f487f8a..77375b012 100644
> --- a/lib/portage/output.py
> +++ b/lib/portage/output.py
> @@ -157,7 +157,7 @@ _styles["UNMERGE_WARN"] = ("red",)
>  _styles["SECURITY_WARN"] = ("red",)
>  _styles["MERGE_LIST_PROGRESS"] = ("yellow",)
>  _styles["PKG_BLOCKER"] = ("red",)
> -_styles["PKG_BLOCKER_SATISFIED"] = ("darkblue",)
> +_styles["PKG_BLOCKER_SATISFIED"] = ("teal",)
>  _styles["PKG_MERGE"] = ("darkgreen",)
>  _styles["PKG_MERGE_SYSTEM"] = ("darkgreen",)
>  _styles["PKG_MERGE_WORLD"] = ("green",)
> @@ -165,8 +165,8 @@ _styles["PKG_BINARY_MERGE"] = ("purple",)
>  _styles["PKG_BINARY_MERGE_SYSTEM"] = ("purple",)
>  _styles["PKG_BINARY_MERGE_WORLD"] = ("fuchsia",)
>  _styles["PKG_UNINSTALL"] = ("red",)
> -_styles["PKG_NOMERGE"] = ("darkblue",)
> -_styles["PKG_NOMERGE_SYSTEM"] = ("darkblue",)
> +_styles["PKG_NOMERGE"] = ("teal",)
> +_styles["PKG_NOMERGE_SYSTEM"] = ("teal",)
>  _styles["PKG_NOMERGE_WORLD"] = ("blue",)
>  _styles["PROMPT_CHOICE_DEFAULT"] = ("green",)
>  _styles["PROMPT_CHOICE_OTHER"] = ("red",)
> diff --git a/man/color.map.5 b/man/color.map.5
> index 288bf7fbd..92a1baa91 100644
> --- a/man/color.map.5
> +++ b/man/color.map.5
> @@ -40,7 +40,7 @@ Defines color used for numbers indicating merge progress.
>  \fBPKG_BLOCKER\fR = \fI"red"\fR
>  Defines color used for unsatisfied blockers.
>  .TP
> -\fBPKG_BLOCKER_SATISFIED\fR = \fI"darkblue"\fR
> +\fBPKG_BLOCKER_SATISFIED\fR = \fI"teal"\fR
>  Defines color used for satisfied blockers.
>  .TP
>  \fBPKG_MERGE\fR = \fI"darkgreen"\fR
> @@ -63,10 +63,10 @@ package.
>  Defines color used for world packages planned to be merged using a binary
>  package.
>  .TP
> -\fBPKG_NOMERGE\fR = \fI"darkblue"\fR
> +\fBPKG_NOMERGE\fR = \fI"teal"\fR
>  Defines color used for packages not planned to be merged.
>  .TP
> -\fBPKG_NOMERGE_SYSTEM\fR = \fI"darkblue"\fR
> +\fBPKG_NOMERGE_SYSTEM\fR = \fI"teal"\fR
>  Defines color used for system packages not planned to be merged.
>  .TP
>  \fBPKG_NOMERGE_WORLD\fR = \fI"blue"\fR
> -- 
> 2.34.1
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] LTS branch of Portage

2021-10-05 Thread Fabian Groffen
In all fairness, there haven't been that much incidents with Portage in
the past under Zac's supervision, isn't it a bit overkill to
bureaucratise the release model just after one incident?  It appears to
me that changes to Portage need to be considered very carefully, always,
since it affects everyone.

Thanks,
Fabian

On 05-10-2021 10:31:40 +0200, Michał Górny wrote:
> Hi, everyone.
> 
> I've been thinking about this for some time already, and the recent
> FILESDIR mess seems to confirm it: I'd like to start a more stable LTS
> branch of Portage.
> 
> Roughly, the idea is that:
> 
> - master becomes 3.1.x, and primary development happens there
> 
> - 3.0.x becomes the LTS branch and only major bugfixes are backported
> there
> 
> As things settle down in the future, master would become 3.2.x, 3.1.x
> would become LTS, 3.0.x will be discontinued and so on.
> 
> WDYT?
> 
> -- 
> Best regards,
> Michał Górny
> 
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Portage einfo, elog... output format change

2021-10-05 Thread Fabian Groffen
On 05-10-2021 09:35:32 +0200, Michał Górny wrote:
> On Thu, 2021-09-30 at 09:18 +0200, Fabian Groffen wrote:
> > > > Final question, am I understanding correctly that normal lines are not
> > > > prefixed with something?  Would it be, for consistency, alignment, and
> > > > certainty of selecting rows something to use a prefix for those lines
> > > > too (assuming they aren't at this point)?
> > > 
> > > I don't know, we've never done that.  I suppose it would be possible but
> > > it is even more controversial and unlike the proposed changes, it would
> > > actually require mangling the process output.
> > 
> > If I remember correctly, Portage already does.  In which case, doing
> > this (even if it were adding leading spaces) would not be that much
> > work?
> > 
> 
> I'm afraid this is not that simple.  You need to account for all escape
> sequences that can affect editing already output data including clean
> handling of line wrapping.  In the end, we'd have to have a complete
> detachtty-class terminal emulator inside Portage.

Fair enough, thanks for looking into it.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Portage einfo, elog... output format change

2021-10-03 Thread Fabian Groffen
On 02-10-2021 23:03:56 -0400, Ionen Wolkens wrote:
> On Sat, Oct 02, 2021 at 10:53:37PM -0400, Ionen Wolkens wrote:
> > Guess there's a lot of other options that could be considered as well.
> > 
> > --- files
> > >>> text
> >  * current, it wasn't aligned now that I look at it again
> > (relying only on color to convey type feels clearly wrong to me)
> > 
> > --- files
> > >>>> text
> > [QA] new based on current PR
> > 
> > >>> text
> > [QA] aligned 1 character further, maybe skipping changing >>> is fine?
> > (then again that it's further is what threw me off at first)
> > 
> > >>> text
> > QA* similar to before, but aligned using only 3 chars
> > 
> > >>> text
> > [Q] kinda more obscure but it can work
> > 
> 
> Guess should also add these:
> >>> text
> Q* Notice:
> E* Some error happened
> (closest to before by making use of the former leading space, thus
>  no alignment changes)

FWIW, I like this one.  Perhaps even with lowercase

make[4]: leaving directory src
q* soname lacks version
e* failed to die

Fabian

> 
> >>> text
> QA Notice:
> EE Some error happened
> (at least clearer than Q* Notice, but unsure about no separator.. guess
> it could work?)
> 
> > >>>> text
> > QA* probably closest to how it was before alignment-wise, but meh at 4 >
> > 
> > >>> message
> > QA> not convinced about this one, but throwing it here anyway
> > (other characters could be considered as well)
> > 
> > Maybe a poll of some kind may help, personally undecided on what I
> > like better beside agreeing that a change is needed.
> 
> 
> -- 
> ionen



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Portage einfo, elog... output format change

2021-09-30 Thread Fabian Groffen
On 30-09-2021 08:44:33 +0200, Michał Górny wrote:
> On Thu, 2021-09-30 at 08:40 +0200, Fabian Groffen wrote:
> > Hi,
> > 
> > Would it be possible to have some switch (e.g. --style=legacy) that
> > controls this new vs. the old behaviour?  Would perhaps allow
> > applications that parse the output to work via setting this in the
> > global opts.
> 
> Patches welcome.  It shouldn't be hard, my commit shows which files need
> to be edited to alter the prefixes and how to pass them into ebd.

I see.

> > In addition, much like the colour map, how do you see this change in
> > light of eclasses, init-scripts, etc. that also use the same scheme as
> > Portage at the moment?  Would you expect to change those too at some
> > point?
> 
> Eclasses are supposed to use standard einfo, elog... functions, so they
> should just work™.  If someone's reinventing the wheel, it's not my
> problem.
> 
> Init scripts aren't supposed to be used inside the PM, so that's out of
> scope.

I was just referring to the overall "feel" of Gentoo, which your work
changes.  It is ok that you don't plan on doing anything there.

> > Final question, am I understanding correctly that normal lines are not
> > prefixed with something?  Would it be, for consistency, alignment, and
> > certainty of selecting rows something to use a prefix for those lines
> > too (assuming they aren't at this point)?
> 
> I don't know, we've never done that.  I suppose it would be possible but
> it is even more controversial and unlike the proposed changes, it would
> actually require mangling the process output.

If I remember correctly, Portage already does.  In which case, doing
this (even if it were adding leading spaces) would not be that much
work?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Portage einfo, elog... output format change

2021-09-30 Thread Fabian Groffen
Hi,

Would it be possible to have some switch (e.g. --style=legacy) that
controls this new vs. the old behaviour?  Would perhaps allow
applications that parse the output to work via setting this in the
global opts.

In addition, much like the colour map, how do you see this change in
light of eclasses, init-scripts, etc. that also use the same scheme as
Portage at the moment?  Would you expect to change those too at some
point?

Final question, am I understanding correctly that normal lines are not
prefixed with something?  Would it be, for consistency, alignment, and
certainty of selecting rows something to use a prefix for those lines
too (assuming they aren't at this point)?

Thanks,
Fabian

On 28-09-2021 17:36:25 +0200, Michał Górny wrote:
> Hi, everyone.
> 
> I know I'm going to regret asking this... but I've prepared a change to
> the Portage output format and I think it asks for a wider discussion
> than internally in Portage team.
> 
> The primary problem with the current output format is that different
> kinds of messages differ only in color.  This makes them
> indistinguishable without colors and hard to grep.  ago's been asking
> for a better way to grep for QA warnings and this is pretty much what
> motivated me to do this.
> 
> The proposed new format distinguished message types both using colors
> and strings.  This is roughly inspired by Xorg logs.  For example,
> instead of:
> 
>  * some message
>  * other message
>  * hell if i know what this is
> 
> You get:
> 
> [WW] some message
> [EE] other message
> [QA] hell if i know what this is
> 
> I've also added more colors to explicitly distinguish einfo from elog,
> and ewarn from eqawarn.  Then, I've replaced most of '>>>' and '!!!'
> used by Portage with four-character versions to keep messages aligned. 
> The 'zings' used for merging files remain three-character, so now it's
> easily possible to distinguish messages from installed file list.
> 
> The PR doing this is: https://github.com/gentoo/portage/pull/759
> 
> Example screenshot:
> https://user-images.githubusercontent.com/110765/135119090-16e9599d-1b0f-41b8-a965-a55577183ffd.png
> 
> -- 
> Best regards,
> Michał Górny
> 
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] In what phase are file "merged"?

2021-06-23 Thread Fabian Groffen
On 23-06-2021 08:47:58 +0200, Ulrich Mueller wrote:
> >>>>> On Wed, 23 Jun 2021, Joakim Tjernlund wrote:
> 
> > In PMS 9.2 is says:
> > The call order for upgrading, downgrading or reinstalling a package is:
> 
> > pkg_pretend (only for EAPIs listed in table 9.2), which is called 
> > outside of the normal call order process.
> > pkg_setup
> > src_unpack
> > src_prepare (only for EAPIs listed in table 9.3)
> > src_configure (only for EAPIs listed in table 9.4)
> > src_compile
> > src_test (except if RESTRICT=test)
> > src_install
> > pkg_preinst
> > pkg_prerm for the package being replaced
> > pkg_postrm for the package being replaced
> > pkg_postinst
> 
> > It does not say where in this list new files merged and old files
> > unmerged, can anyone enlighten me?
> 
> It's somewhat hidden, but it's there:
> https://projects.gentoo.org/pms/8/pms.html#x1-950009.1.10
> 
>9.1.10 pkg_preinst
>... immediately before merging the package to the live filesystem. ...
> 
>9.1.11 pkg_postinst
>... immediately after merging the package to the live filesystem. ...

Aha, so does this mean pkg_prerm and pkg_postrm are run with replacing
package in place, e.g. if they refer to scripts installed by the
replaced package they may no longer exist or be the same?

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] KeyError: 'ED' ?

2021-06-10 Thread Fabian Groffen
On 09-06-2021 14:19:38 -0400, Mike Gilbert wrote:
> On Thu, May 6, 2021 at 6:54 AM Joakim Tjernlund
>  wrote:
> >
> > On Wed, 2021-05-05 at 16:47 +, Joakim Tjernlund wrote:
> > > I am upgrading an old portage(2.3.76) to 3.0.18 using qmerge(binary pkg) 
> > > and I get this:
> > >  qmerge -OK sys-apps/portage
> > > [R] sys-apps/portage-3.0.18
> > >  * Checking for suitable kernel configuration options...
> > >  [ ok ]
> > >  * Using python3.8 in global scope
> > > FEATURES variable contains unknown value(s): disabled
> > > Traceback (most recent call last):
> > >   File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
> > > return _run_code(code, main_globals, None,
> > >   File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
> > > exec(code, run_globals)
> > >   File 
> > > "/var/tmp/qmerge/sys-apps/portage-3.0.18/image/usr/lib/python3.8/site-packages/portage/_compat_upgrade/default_locations.py",
> > >  line 93, in 
> > > main()
> > >   File 
> > > "/var/tmp/qmerge/sys-apps/portage-3.0.18/image/usr/lib/python3.8/site-packages/portage/_compat_upgrade/default_locations.py",
> > >  line 63, in main
> > > config_path = os.path.join(os.environ['ED'], 
> > > GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals')
> > >   File "/usr/lib/python3.8/os.py", line 675, in __getitem__
> > > raise KeyError(key) from None
> > > KeyError: 'ED'
> > >
> > > so portage fails to install properly, I am guessin this is becuse I have 
> > > an old portage to begin with?
> > > Can it be fixed?
> >
> > The error goes away when I do: ED=/ qmerge -OK sys-apps/portage
> > Does that mean that it is qmerge(aka portage-utils) that needs to set ED 
> > during merge? which PHASES ?
> 
> According to PMS, a package manager must define ED in src_install,
> pkg_preinst, and pkg_postinst.

Perhaps qmerge should use export for these vars.  In any case this seems
tracked in bug 792273.

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] last-rite: app-admin/diamond

2021-05-24 Thread Fabian Groffen
# Fabian Groffen  (2021-05-24)
# Hack on hack no longer sustainable
# - seemingly dead upstream
# - no (official) Python 3 support
# removal in 30 days, bug: https://bugs.gentoo.org/791613

migrate to something like app-metrics/collectd using the the
write_graphite plugin.

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [News item review] Exim >=4.94 transports: tainted not permitted

2021-05-06 Thread Fabian Groffen
On 06-05-2021 15:01:33 +0200, Andreas K. Huettel wrote:
> > Unfortunately there is not much documentation on "tainted" data for
> > Exim[1], and to resolve this, non-official sources need to be used,
> > such as [2] and [3].
> 
> This is a safety mechanism that is part of Perl (essentially a way of 
> tracking data that is derived from "insecure" sources).
> 
> So it probably would make sense to at least point towards that concept 
> in Perl.

I think the concept is clear to most from the descriptions one can find.
The big problem however is the solution, how to fix one's configuration.

Luckily it seems people find their way to Exim's bugtracker to get help
there.

Thanks for the suggestion though,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] [News item review v2] Exim >=4.94 transports: tainted not permitted

2021-05-02 Thread Fabian Groffen
Title: Exim>=4.94 transports: tainted not permitted
Author: Fabian Groffen 
Posted: 2021-05-??
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: mail-mta/exim

The Message Transfer Agent Exim disallows tainted variables in transport
configurations since version 4.94.  Existing exim.conf configurations
in /etc/exim need to be reviewed for breakage prior to upgrading to
 >=mail-mta/exim-4.94 to avoid error conditions at runtime.

Since the release of Exim-4.94, transports refuse to use tainted data in
constructing a delivery location.  If you use this in your transports,
your configuration will break, causing errors and possible downtime.

Particularly, the use of $local_part in any transport, should likely be
updated with $local_part_data.  Check your local_delivery transport,
which historically used $local_part.

Unfortunately there is not much documentation on "tainted" data for
Exim[1], and to resolve this, non-official sources need to be used, such
as [2] and [3].



[1] https://lists.exim.org/lurker/message/20201109.222746.24ea3904.en.html
[2] https://mox.sh/sysadmin/tainted-filename-errors-in-exim-4.94/
[3] 
https://jimbobmcgee.wordpress.com/2020/07/29/de-tainting-exim-configuration-variables/

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [News item review] Exim >=4.94 transports: tainted not permitted

2021-05-02 Thread Fabian Groffen
On 02-05-2021 12:23:30 +0200, Ulrich Mueller wrote:
> >>>>> On Sun, 02 May 2021, Fabian Groffen wrote:
> 
> > Title: Exim >=4.94 disallows tainted variables in transport configurations
> 
> Title is too long (GLEP 42 allows 50 chars max).

ah, missed that

> I have no idea what this news item is trying to tell me. But I don't use
> Exim, so probably that's the reason. :) Maybe mention at least that Exim
> is a mailer?

Fair point.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] [News item review] Exim >=4.94 transports: tainted not permitted

2021-05-02 Thread Fabian Groffen
Title: Exim >=4.94 disallows tainted variables in transport configurations
Author: Fabian Groffen 
Posted: 2021-05-??
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: mail-mta/exim

Since the release of Exim-4.94, transports refuse to use tainted data in
constructing a delivery location.  If you use this in your transports,
your configuration will break, causing errors and possible downtime.

Particularly, the use of $local_part in any transport, should likely be
updated with $local_part_data.  Check your local_delivery transport,
which historically used $local_part.

Unfortunately there is not much documentation on "tainted" data for
Exim[1], and to resolve this, non-official sources need to be used, such
as [2] and [3].



[1] https://lists.exim.org/lurker/message/20201109.222746.24ea3904.en.html
[2] https://mox.sh/sysadmin/tainted-filename-errors-in-exim-4.94/
[3] 
https://jimbobmcgee.wordpress.com/2020/07/29/de-tainting-exim-configuration-variables/

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Packages up for grabs: x11-misc/xbindkeys

2021-02-21 Thread Fabian Groffen
Jeroen was sadly removed from the project.

You go ahead and take it.

Fabian

On 22-02-2021 07:49:55 +0200, Jaco Kroon wrote:
> Hi,
> 
> Looks like Jeroen was the person mostly committing on this, Jeroen - are
> you intending to keep maintaining this?
> 
> The PR is legit but incomplete (it merely adds the patch, not use it).
> 
> The patch has been merged upstream in 2014 already, instead we should
> stable 1.8.7.
> 
> I'll adjust the referenced bug referenced by the PR accordingly.
> 
> Kind Regards,
> Jaco
> 
> On 2021/02/21 05:08, Jonas Stein wrote:
> > Dear all
> >
> > the following packages are up for grabs after dropping
> > desktop-misc:
> >
> > x11-misc/xbindkeys
> > https://packages.gentoo.org/packages/x11-misc/xbindkeys
> >
> > There is an open PR
> > https://github.com/gentoo/gentoo/pull/14706
> >
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] [PATCH] eclass/db.eclass: use get_libname for Darwin support

2021-01-12 Thread Fabian Groffen
Signed-off-by: Fabian Groffen 
---
 eclass/db.eclass | 24 
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/eclass/db.eclass b/eclass/db.eclass
index 9a246d18979..52afe0b765f 100644
--- a/eclass/db.eclass
+++ b/eclass/db.eclass
@@ -23,13 +23,13 @@ db_fix_so() {
cd "${LIB}" || die
 
# first clean up old symlinks
-   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*so' -delete || die
-   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*so.[23]' -delete 
|| die
+   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*'"$(get_libname)" 
-delete || die
+   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*'"$(get_libname 
"[23]")" -delete || die
find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*a' -delete || die
 
# now rebuild all the correct ones
local ext
-   for ext in so a; do
+   for ext in so dylib a; do
for name in libdb libdb_{cxx,tcl,java,sql,stl}; do
target="$(find . -maxdepth 1 -type f -name 
"${name}-*.${ext}" |sort -V |tail -n 1)"
[[ -n "${target}" ]] && ln -sf ${target//.\//} 
${name}.${ext}
@@ -37,17 +37,17 @@ db_fix_so() {
done;
 
# db[23] gets some extra-special stuff
-   if [[ -f libdb1.so.2 ]]; then
-   ln -sf libdb1.so.2 libdb.so.2
-   ln -sf libdb1.so.2 libdb1.so
-   ln -sf libdb1.so.2 libdb-1.so
+   if [[ -f libdb1$(get_libname 2) ]]; then
+   ln -sf libdb1$(get_libname 2) libdb$(get_libname 2)
+   ln -sf libdb1$(get_libname 2) libdb1$(get_libname)
+   ln -sf libdb1$(get_libname 2) libdb-1$(get_libname)
fi
# what do we do if we ever get 3.3 ?
local i
for i in libdb libdb_{cxx,tcl,java,sql,stl}; do
-   if [[ -f ${i}-3.2.so ]]; then
-   ln -sf ${i}-3.2.so ${i}-3.so
-   ln -sf ${i}-3.2.so ${i}.so.3
+   if [[ -f $i-3.2$(get_libname) ]]; then
+   ln -sf $i-3.2$(get_libname) $i-3$(get_libname)
+   ln -sf $i-3.2$(get_libname) $i$(get_libname 3)
fi
done
 
@@ -143,8 +143,8 @@ db_src_install_usrlibcleanup() {
mv "${LIB}/libdb_cxx.a" "${LIB}/libdb_cxx-${SLOT}.a" || die
fi
 
-   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*so' -delete || die
-   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*so.[23]' -delete 
|| die
+   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*'"$(get_libname)" 
-delete || die
+   find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*'"$(get_libname 
"[23]")" -delete || die
einfo "removing unversioned static archives"
find "${LIB}" -maxdepth 1 -type l -name 'libdb[1._-]*a' -delete || die
 
-- 
2.26.2




Re: [gentoo-dev] [RFC] Dealing with global /usr/bin/libtool use vs CC/CXX etc.

2021-01-10 Thread Fabian Groffen
On 10-01-2021 14:34:13 +0100, Michał Górny wrote:
> The vast majority of libtool-based programs use configure script to
> generate libtool.  However, a few non-autoconf packages also use libtool
> by calling system-installed /usr/bin/libtool.  The problem is that this
> libtool hardcodes the values of CC/CXX at its' build time, so unless it
> is rebuilt frequently, packages end up using the stale values.
> The problem is known since 2005 [1] and hasn't been resolved yet.
> 
> I can think of two ways of solving it:
> 
> 1. We could patch system-installed libtool to respect environment
> variables such as CC, CXX, etc.  This will probably require carrying
> a (possibly non-trivial) patch forever.  On the bright side, libtool is
> not exactly a package seeing frequent releases.  I mean, the current
> version is from 2015.
> 
> 2. We could regenerate libtool and force local instance of libtool
> in the packages needing it.  The main advantage of this is that it's
> a no-brainer.  I could make a quick eclass that does configure a local
> instance and prepends it into PATH.
> 
> WDYT?

I would prefer option 2, also because on some systems usr/bin/libtool is
some entirely different tool than GNU libtool.

I remember this being much more of a problem ~15 years ago, so I wonder
do we have an easy way of crafting a list of affected packages, such
that we can see how big the problem actually is nowadays?  I'm thinking
perhaps tinderbox logs or something can reveal /usr/bin/libtool usage
somehow.

Thanks,
Fabian

> [1] https://bugs.gentoo.org/88596


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [PATCH] acct-user.eclass: don't modify existing user by default

2021-01-08 Thread Fabian Groffen
On 04-01-2021 17:14:47 +0100, Michał Górny wrote:
> Yes, I don't mind an option, as long as it spews a big fat ewarn that
> the user loses the right to support.  However, that's still not
> the right solution to the immediate problem, and I'm currently working
> on a better patch, so I'd prefer if you waited with that to avoid merge
> conflicts.

Could you please share your intended approach?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] _get_lock_fn: support multiprocessing spawn start method (bug 758230)

2020-12-05 Thread Fabian Groffen
Thanks Zac!

On 04-12-2020 14:58:22 -0800, Zac Medico wrote:
> Ensure that _get_lock_fn arguments to multiprocessing.Process will
> successfully pickle, as required by the spawn start method, which
> is the default for macOS since Python 3.8.
> 
> Since file descriptors are not inherited unless the fork start
> method is used, the subprocess should only try to close an
> inherited file descriptor for the fork start method.
> 
> Bug: https://bugs.gentoo.org/758230
> Signed-off-by: Zac Medico 
> ---
>  lib/portage/locks.py | 36 +++-
>  1 file changed, 23 insertions(+), 13 deletions(-)
> 
> diff --git a/lib/portage/locks.py b/lib/portage/locks.py
> index 1073343be..193045c03 100644
> --- a/lib/portage/locks.py
> +++ b/lib/portage/locks.py
> @@ -44,18 +44,7 @@ def _get_lock_fn():
>   if _lock_fn is not None:
>   return _lock_fn
>  
> - def _test_lock(fd, lock_path):
> - os.close(fd)
> - try:
> - with open(lock_path, 'a') as f:
> - fcntl.lockf(f.fileno(), 
> fcntl.LOCK_EX|fcntl.LOCK_NB)
> - except EnvironmentError as e:
> - if e.errno == errno.EAGAIN:
> - # Parent process holds lock, as expected.
> - sys.exit(0)
>  
> - # Something went wrong.
> - sys.exit(1)
>  
>   fd, lock_path = tempfile.mkstemp()
>   try:
> @@ -64,8 +53,16 @@ def _get_lock_fn():
>   except EnvironmentError:
>   pass
>   else:
> - proc = multiprocessing.Process(target=_test_lock,
> - args=(fd, lock_path))
> + proc = multiprocessing.Process(
> + target=_subprocess_test_lock,
> + args=(
> + # Since file descriptors are not 
> inherited unless the fork start
> + # method is used, the subprocess should 
> only try to close an
> + # inherited file descriptor for the 
> fork start method.
> + fd if 
> multiprocessing.get_start_method() == "fork" else None,
> + lock_path,
> + ),
> + )
>   proc.start()
>   proc.join()
>   if proc.exitcode == os.EX_OK:
> @@ -80,6 +77,19 @@ def _get_lock_fn():
>   _lock_fn = fcntl.flock
>   return _lock_fn
>  
> +def _subprocess_test_lock(fd, lock_path):
> + if fd is not None:
> + os.close(fd)
> + try:
> + with open(lock_path, 'a') as f:
> + fcntl.lockf(f.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB)
> + except EnvironmentError as e:
> + if e.errno == errno.EAGAIN:
> +     # Parent process holds lock, as expected.
> + sys.exit(0)
> +
> + # Something went wrong.
> + sys.exit(1)
>  
>  _open_fds = {}
>  _open_inodes = {}
> -- 
> 2.26.2
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] Last-rites: sys-kernel/xnu-headers, sys-libs/darwin-libc-headers, dev-libs/libmissing

2020-11-23 Thread Fabian Groffen
# Fabian Groffen  (2020-11-23)
# No longer used, not really functional either, noone should be using
# this, removal in 30 days.
sys-kernel/xnu-headers
sys-libs/darwin-libc-headers
dev-libs/libmissing

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecating AMD64 17.0 profiles?

2020-11-10 Thread Fabian Groffen
On 10-11-2020 09:21:53 +, Alexey Sokolov wrote:
> > What instructed you to perform the migration?  Was it the news-item?  I
> > don't think it should apply for Prefix profiles, and perhaps we should
> > be happy the tool won't work.
> 
> It was the big scary warning about the deprecation whenever I run
> emerge. It contains list of steps.

Ok.  I don't know if we can do anything about that.

> > Did it do anything to your system like creating a lib64 directory?  Does
> > anything work (because I have doubts on whether your system can still
> > find the libs in there now).
> 
> Yes. Attaching logs.

The logs seem to indicate that it thinks all libs on your systems do not
belong to any package.  This suggests the tool cannot locate the VDB or
something, as most of the things in the list are obviously owned by
packages.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecating AMD64 17.0 profiles?

2020-11-10 Thread Fabian Groffen
On 10-11-2020 09:49:27 +0100, Michał Górny wrote:
> > > So what you're saying is that you've had the wrong value of SYMLINK_LIB
> > > for ages, and now you've created a meaningless 17.1 profile that chnages
> > > it but isn't actually supposed to change anything, correct?
> > 
> > I guess, because the amd64 17.0 profile is deprecated with force, and I
> > had to do something ...
> 
> Now that's a lie.  Only the regular amd64 profiles are deprecated.
> There are no deprecation notices e.g. in the x32 profile or prefix
> profiles.

Our profiles either directly depend on the amd64/17.0 profile, or we use
a sub-profile from amd64/17.0 profile, so if it's going to get removed,
we are having a problem, don't we?


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecating AMD64 17.0 profiles?

2020-11-10 Thread Fabian Groffen
On 10-11-2020 09:34:52 +0100, Michał Górny wrote:
> On Tue, 2020-11-10 at 08:55 +0100, Fabian Groffen wrote:
> > On 09-11-2020 19:38:28 +, Alexey Sokolov wrote:
> > > Hi Fabian
> > > I tried to migrate my prefix to 17.1, and there are issues.
> > > 
> > > 1) unsymlink-lib requires "--root ~/gentoo" and otherwise produces an
> > > error "/usr/lib is a real directory! was the migration done already?"
> > 
> > I think unsymlink-lib doesn't have Prefix support, but in addition,
> > what unsymlink-lib is trying to achieve, is not a thing perhaps on
> > Prefix.
> > 
> > A prefix system (at least all of mine) doesn't have libXX or lib/XX
> > (a.k.a.  multilib) directories.  The /usr-split was long ago removed,
> > and thus what we have is:
> > 
> >   lib -> usr/lib
> > 
> > Now, SYMLINK_LIB=no seems to split into lib and lib64, but lib64 does
> > not exist on Prefix systems.
> > 
> 
> So what you're saying is that you've had the wrong value of SYMLINK_LIB
> for ages, and now you've created a meaningless 17.1 profile that chnages
> it but isn't actually supposed to change anything, correct?

I guess, because the amd64 17.0 profile is deprecated with force, and I
had to do something ...


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecating AMD64 17.0 profiles?

2020-11-09 Thread Fabian Groffen
On 09-11-2020 19:38:28 +, Alexey Sokolov wrote:
> Hi Fabian
> I tried to migrate my prefix to 17.1, and there are issues.
> 
> 1) unsymlink-lib requires "--root ~/gentoo" and otherwise produces an
> error "/usr/lib is a real directory! was the migration done already?"

I think unsymlink-lib doesn't have Prefix support, but in addition,
what unsymlink-lib is trying to achieve, is not a thing perhaps on
Prefix.

A prefix system (at least all of mine) doesn't have libXX or lib/XX
(a.k.a.  multilib) directories.  The /usr-split was long ago removed,
and thus what we have is:

  lib -> usr/lib

Now, SYMLINK_LIB=no seems to split into lib and lib64, but lib64 does
not exist on Prefix systems.

Since Prefix is non-multilib by design*, I wonder if unsymlink-lib is
necessary in the best case, but going to break the Prefix system in the
worst case.

What instructed you to perform the migration?  Was it the news-item?  I
don't think it should apply for Prefix profiles, and perhaps we should
be happy the tool won't work.

* non-multilib is a decision dating back a decade or so, which means
  effectively any Prefix install you encounter should be non-multilib


> 2) $ unsymlink-lib --root ~/gentoo --migrate --pretend
> usage: unsymlink-lib [-h] [-p] [--root ROOT] [--analyze] [--migrate]
>  [--rollback] [--finish] [--force-rollback]
>  [--resume-finish] [-P PREFIX] [--hardlink]
> unsymlink-lib: error: Requested action requires root privileges
> 
> Well, I worked it around by adding "is_root = True" to unsymlink-lib

Did it do anything to your system like creating a lib64 directory?  Does
anything work (because I have doubts on whether your system can still
find the libs in there now).

> 
> 3) Step 9 (Rebuild gcc) fails:
> configure:4372: checking whether the C compiler works
> 
> 
> 
> configure:4394: x86_64-pc-linux-gnu-gccconftest.c  >&5
> 
> 
> 
> /home/user/gentoo/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../x86_64-pc-linux-gnu/bin/as:
> error while loading shared libraries:
> libopcodes-2.34.0.gentoo-sys-devel-binutils-st.so: cannot open shared

Something like this I was suspecting.  Can you still rollback?  If you
can, I'd try that and hope it restores your system in working order.

For any Prefix user, DO NOT run unsymlink-lib, I believe you should NOT
NEED IT!

Thanks,
Fabian

> object file: No such file or directory
> configure:4398: $? = 1
> 
> 
> 
> configure:4436: result: no
> 
> The file exists:
> $ find ~ -name libopcodes-2.34.0.gentoo-sys-devel-binutils-st.so
> /home/user/gentoo/usr/lib/binutils/x86_64-pc-linux-gnu/2.34/libopcodes-2.34.0.gentoo-sys-devel-binutils-st.so
> 
> -- 
> Best regards,
> Alexey "DarthGandalf" Sokolov
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecating AMD64 17.0 profiles?

2020-11-07 Thread Fabian Groffen
On 07-11-2020 11:42:44 +, Alexey Sokolov wrote:
> 22.10.2020 20:16, Andreas K. Hüttel пишет:
> > Am Donnerstag, 22. Oktober 2020, 18:44:48 EEST schrieb Michał Górny:
> >> On Thu, 2020-10-22 at 11:17 -0400, Brian Evans wrote:
> >>> Users frequently are choosing the wrong profile versions in new installs
> >>> and accidentally downgrading to 17.0 with some saying they see it first.
> >>>
> >>> A simple reordering could help new installs.
> > 
> > Independent of this useful change, it's probably time to deprecate the 
> > amd64 
> > 17.0 profiles!
> > 
> 
> Prefix bootstrap script still makes new installations to use it

This should be solved with

b0445c0a8dd6d2f792c5bb088b154aca53868353
a9c478dc881ee18fefc7342da994b00e60eaad8e

on gentoo.git and

0d7f6b6eb00d0f51f35019846b8f79048b30be93

on prefix.git.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] net-misc/asterisk fails to compile: clang/LLVM: bug 731280

2020-08-28 Thread Fabian Groffen
On 28-08-2020 08:52:09 +0100, Sergei Trofimovich wrote:
> On Fri, 28 Aug 2020 07:37:54 +0100
> Sergei Trofimovich  wrote:
> 
> > On Fri, 28 Aug 2020 08:15:47 +0200
> > Jaco Kroon  wrote:
> > 
> > > Hi All,
> > > 
> > > https://bugs.gentoo.org/731280
> > > 
> > > Summary:
> > > 
> > > This machine uses a clang/LLVM toolchain.
> > > Asterisk fails to compile, ./configure fails with:
> > > 
> > > checking for RAII support... checking for clang -fblocks...
> > > configure: error: BlocksRuntime is required for clang, please install
> > > libblocksruntime
> > > 
> > > I suspect this explains the issue:
> > > 
> > > https://github.com/mackyle/blocksruntime
> > > 
> > > I have no idea how to actually solve this.  
> > 
> > honggfuzz also needs it as it happens to use -fblocks on clang:
> > https://bugs.gentoo.org/729256
> > 
> > We need to package the library. I can give it a try.
> 
> Packaged library as sys-libs/blocksruntime:
> 
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ae376c73ef197d6c7aa619e821c436ccab0cd77e
> 
> Usage example for app-forensics/honggfuzz:
> 
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=fd841336dfdefbc14907e2d9b1eb1a1a3f5f8b8e

This is cool, but shouldn't it be something like openmp?  (e.g. blocks)

There is no reason blocks have to be used if not on macOS (where system
headers use blocks features).

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] euses(1) Reimplementation

2020-07-09 Thread Fabian Groffen
706#c4
> [4] https://dl.acm.org/doi/abs/10.1145/116825.116845
> [5] 
> https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strcasestr.c;h=d2964c5548b9ea7a68fc5b18b25ddfe7ddd6835c;hb=HEAD#l45
> [6] http://git.suugaku.co.uk/ash-euses/tree/
> [7] http://git.suugaku.co.uk/ash-euses/snapshot/ash-euses-0.3.tar.gz
> 
> -- 
> 
> Ashley Dixon
> suugaku.co.uk
> 
> 2A9A 4117
> DA96 D18A
> 8A7B B0D2
> A30E BF25
> F290 A8AA
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Speeding up Tree Verification

2020-07-01 Thread Fabian Groffen
On 30-06-2020 13:13:29 -0500, Sid Spry wrote:
> On Tue, Jun 30, 2020, at 1:20 AM, Fabian Groffen wrote:
> > Hi,
> > 
> > On 29-06-2020 21:13:43 -0500, Sid Spry wrote:
> > > Hello,
> > > 
> > > I have some runnable pseudocode outlining a faster tree verification 
> > > algorithm.
> > > Before I create patches I'd like to see if there is any guidance on 
> > > making the
> > > changes as unobtrusive as possible. If the radical change in algorithm is
> > > acceptable I can work on adding the changes.
> > > 
> > > Instead of composing any kind of structured data out of the portage tree 
> > > my
> > > algorithm just lists all files and then optionally batches them out to 
> > > threads.
> > > There is a noticeable speedup by eliding the tree traversal operations 
> > > which
> > > can be seen when running the algorithm with a single thread and comparing 
> > > it to
> > > the current algorithm in gemato (which should still be discussed here?).
> > 
> > I remember something that gemato used to use multiple threads, but
> > because it totally saturated disk-IO, it was brought back to a single
> > thread.  People were complaining about unusable systems.
> > 
> 
> I think this is an argument for cgroups limits support on the portage process 
> or
> account as opposed to an argument against picking a better algorithm. That is
> something I have been working towards, but I am only one man.

But this requires a) cgroups support, and b) the privileges to use it.
Shouldn't be a problem in the normal case, but just saying.

> > In any case, can you share your performance results?  What speedup did
> > you see, on warm and hot FS caches?  Which type of disk do you use?
> > 
> 
> I ran all tests multiple times to make them warm off of a Samsung SSD, but
> nothing very precise yet.
> 
> % gemato verify --openpgp-key signkey.asc /var/db/repos/gentoo
> [...]
> INFO:root:Verifying /var/db/repos/gentoo...
> INFO:root:/var/db/repos/gentoo verified in 16.45 seconds
> 
> sometimes going higher, closer to 18s, vs.
> 
> % ./veriftree.py
> 4.763171965983929
> 
> So roughly an order of magnitude speedup without batching to threads.

That is kind of a change.  Makes one wonder if you really did the same
work.

> > You could compare against qmanifest, which uses OpenMP-based
> > paralllelism while verifying the tree.  On SSDs this does help.
> > 
> 
> I lost my notes -- how do I specify to either gemato or qmanifest the GnuPG
> directory? My code is partially structured as it is because I had problems 
> doing
> this. I rediscovered -K/--openpgp-key in gemato but am unsure for qmanifest.

qmanifest doesn't do much magic out of the standard gnupg practices.
(It is using gpgme.)  If you want it to use a different gnupg dir, you
may change HOME, or GNUPGHOME.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Speeding up Tree Verification

2020-06-30 Thread Fabian Groffen
: str) -> bool:
> response = dns.resolver.query(domain, dns.rdatatype.NS)
> nsname = response.rrset[0]
> response = dns.resolver.query(str(nsname), dns.rdatatype.A)
> nsaddr = response.rrset[0].to_text()
> 
> # DNSKEY
> request = dns.message.make_query(domain,
> dns.rdatatype.DNSKEY, want_dnssec=True)
> response = dns.query.udp(request, nsaddr)
> if response.rcode() != 0:
> raise Exception('Unable to request dnskey.')
> 
> answer = response.answer
> if len(answer) != 2:
> raise Exception('Malformed answer to dnskey query.')
> 
> name = dns.name.from_text(domain)
> try:
> dns.dnssec.validate(answer[0], answer[1], {name: answer[0]})
> except dns.dnssec.ValidationFailure as exc:
> # Validation failed. The raise causes python to abort with status 1.
> #raise exc
> return False
> except AttributeError as exc:
> # Validation may have failed; DNSKEY missing signer attribute. dig 
> may report
> # domain as valid.
> #
> # TODO: Additional state where subdomain of valid domain may fail 
> with 3 nested
> # KeyErrors. Avoid temptation to wildcard catch. Safer to put in 
> process?
> #raise exc
> return False
> else:
> return True
> 
> def hash_localpart(incoming: bytes) -> str:
> '''Z-base32 the localpart of an e-mail address
> 
> 
> https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-08#section-3.1
> describes why this is needed.
> 
> See https://tools.ietf.org/html/rfc6189#section-5.1.6 for a
> description of the z-base32 scheme.
> '''
> zb32 = "ybndrfg8ejkmcpqxot1uwisza345h769"
> 
> b = hashlib.sha1(incoming).digest()
> ret = ""
> assert(len(b) * 8 == 160)
> for i in range(0, 160, 5):
> byte = i // 8
> offset = i - byte * 8
> # offset | bits remaining in k+1 | right-shift k+1
> # 3 | 0 | x
> # 4 | 1 | 7
> # 5 | 2 | 6
> # 6 | 3 | 5
> # 7 | 4 | 4
> if offset < 4:
> n = (b[byte] >> (3 - offset))
> else:
> n = (b[byte] << (offset - 3)) + (b[byte + 1] >> (11 - offset))
> 
> ret += zb32[n & 0b1]
> return ret
> 
> def build_web_key_uri(address: str) -> str:
> local, remote = address.split('@')
> local = hash_localpart(local.encode('utf-8'))
> return 'https://' + remote + '/.well-known/openpgpkey/hu/' + \
> local
> 
> def stream_to_file(uri: str, fname: str) -> None:
> with requests.get(uri, verify=True, stream=True) as r:
> from pprint import pprint
> pprint(r.headers)
> with open(fname, 'wb') as f:
> shutil.copyfileobj(r.raw, f)
> ```
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Fabian Groffen
Hi Chun-Yu,

On 26-06-2020 23:34:12 -0700, Chun-Yu Shei wrote:
> Hi,
> 
> I was recently interested in whether portage could be speed up, since
> dependency resolution can sometimes take a while on slower machines.
> After generating some flame graphs with cProfile and vmprof, I found 3
> functions which seem to be called extremely frequently with the same
> arguments: catpkgsplit, use_reduce, and match_from_list.  In the first
> two cases, it was simple to cache the results in dicts, while
> match_from_list was a bit trickier, since it seems to be a requirement
> that it return actual entries from the input "candidate_list".  I also
> ran into some test failures if I did the caching after the
> mydep.unevaluated_atom.use and mydep.repo checks towards the end of the
> function, so the caching is only done up to just before that point.
> 
> The catpkgsplit change seems to definitely be safe, and I'm pretty sure
> the use_reduce one is too, since anything that could possibly change the
> result is hashed.  I'm a bit less certain about the match_from_list one,
> although all tests are passing.
> 
> With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world"
> speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup.  "emerge
> -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 sec
> (2.5% improvement).  Since the upgrade case is far more common, this
> would really help in daily use, and it shaves about 30 seconds off
> the time you have to wait to get to the [Yes/No] prompt (from ~90s to
> 60s) on my old Sandy Bridge laptop when performing normal upgrades.
> 
> Hopefully, at least some of these patches can be incorporated, and please
> let me know if any changes are necessary.

This sounds like a good job to me!  Do you have any idea what the added
memory pressure for these changes are?

Thanks,
Fabian
> 
> Thanks,
> Chun-Yu
> 
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] Use env to find python

2020-06-17 Thread Fabian Groffen
On 16-06-2020 23:10:28 -0500, Sid Spry wrote:
> On Tue, Jun 16, 2020, at 3:57 PM, Michał Górny wrote:
> > '/usr/bin/env python' (with no extra options) is the portable shebang.
> > 
> 
> I added `-S` to preserve the options passed via the shebang line. It seems 
> they can be left off, does anyone know otherwise?

-S is not supported on some platforms.

In any case, I think this patch is wrong.  In the Prefix case, the
shebangs are set during install, in the non-Prefix case I think there is
machinery in place to replace the shebangs during install.

It seems the problem is already solved, why do you need these shebangs
changed?

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New USE=-native-symlinks for gcc-config and binutils-config

2020-05-24 Thread Fabian Groffen
On 24-05-2020 10:48:47 +0100, Sergei Trofimovich wrote:
> On Sat, 23 May 2020 22:41:02 -0400
> Mike Gilbert  wrote:
> > I don't think we
> > want to give people the impression that this is a well-supported
> > configuration. I would only expect people to disable this if they want
> > to break their system intentionally.
> 
> Yeah, today it's certainly a way to get your system in a miserable state.
> My hope is that USE=-native-symlinks can get you a working Gentoo in a
> near future by only fixing real package problems and limitations of their
> build systems.

Portage adds PREROOTPATH, ROOTPATH and a standard set of
/usr/{local/,}{s,}bin to PATH in _doebuild_path.  That means if you add
something like /usr/bin/native-toolchain to PATH only, users will get
gcc, ld, etc., while root and Portage will lack these.  One can wonder
if root should have direct access to compilers, but it's easy enough to
add the compilers to PATH if really necessary.

I guess what I'm trying to say is: you can hide effect of the setup for
users if you'd like.  That is, after we had buildbots point out the bulk
of packages that are wrong of course.

Thanks,
Fabian
 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Anti-spam for goose

2020-05-23 Thread Fabian Groffen
On 22-05-2020 21:58:54 +0200, Michał Górny wrote:
> Let's put it like this.  This thing starts working.  Package X is
> broken, and we see that almost nobody is using it.  We remove that
> package.  Now somebody is angry.  He submits a lot of fake data to
> render the service useless so that we don't make any future decisions
> based on it.

I'm affraid that has a heroic flair to me.  The service should never be
used for decisions like that, because it's a biased sample at most.
Doing stuff like this simply destroys the soul of the distribution.

I hope this isn't one of your genuine objectives with the service.  If
it is, I can see why you fear spam so much.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] Anti-spam for goose

2020-05-21 Thread Fabian Groffen
 alternative of using a proof-of-work algorithm was suggested to me
> yesterday.  The idea is that every submission has to be accompanied with
> the result of some cumbersome calculation that can't be trivially run
> in parallel or optimized out to dedicated hardware.
> 
> On the plus side, it would rely more on actual physical hardware than IP
> addresses provided by ISPs.  While it would be a waste of CPU time
> and memory, doing it just once a week wouldn't be that much harm.
> 
> On the minus side, it would penalize people with weak hardware.
> 
> For example, 'time hashcash -m -b 28 -r test' gives:
> 
> - 34 s (-s estimated 38 s) on Ryzen 5 3600
> 
> - 3 minutes (estimated 92 s) on some old 32-bit Celeron M
> 
> At the same time, it would still permit a lot of fake submissions.  For
> example, randomx [1] claims to require 2G of memory in fast mode.  This
> would still allow me to use 7 threads.  If we adjusted the algorithm to
> take ~30 seconds, that means 7 submissions every 30 s, i.e. 20k
> submissions a day.
> 
> So in the end, while this is interesting, it doesn't seem like
> a workable anti-spam measure.
> 
> 
> Option 3: explicit CAPTCHA
> ==
> A traditional way of dealing with spam -- require every new system
> identifier to be confirmed by solving a CAPTCHA (or a few identifiers
> for one CAPTCHA).
> 
> The advantage of this method is that it requires a real human work to be
> performed, effectively limiting the ability to submit spam.
> The disadvantage is that it is cumbersome to users, so many of them will
> just resign from participating.
> 
> 
> Other ideas
> ===
> Do you have any other ideas on how we could resolve this?
> 
> 
> [1] https://github.com/tevador/RandomX
> 
> 
> -- 
> Best regards,
> Michał Górny
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] RFC: Gentoo Identity Provider

2020-05-19 Thread Fabian Groffen
On 18-05-2020 18:42:24 -0700, Alec Warner wrote:
> TL;DR: What if we launched id.gentoo.org[1], an identity provider that 
> provides
> authentication for Gentoo properties? Basically, 1 username / password for 
> wiki,
> bugs, email, forums, and any other http service[0][1].

I'd be in favour of SSO for all http-, imap- and smtp-based Gentoo services.

Thanks,
Fabian

> 
> Today Gentoo has numerous systems that mostly work in a segmented way.
> 
>  - To connect to hosts, we use ssh keys.
>  - Git is authenticated via ssh keys.
>  - Email uses LDAP passwords.
>  - Bugzilla has its own identities, with their own passwords.
>  - Wiki is separate, with its own passwords.
>  - Forums are separate.
>  - Infra has an additional 4 systems that use separate credentials.
> 
> Some applications support 2FA (such as wiki.)
> Some applications do not support 2FA.
> Applications that require 2FA have a configuration for each app, so you have N
> configurations.
> 
> If we configured id.gentoo.org[2] you would have 1 identity across all gentoo
> properties.
> 
> Is this a thing people are interested in?
>  
> [0] It's unlikely operations for git via ssh would change in this rollout.
> [1] Its unclear if the scope is "gentoo developers" or "any community member."
> The former have LDAP accounts and @gentoo.org[3] email addresses and so we can
> manage them easily; managing 1000s of other accounts in the IDP remains to be
> seem.
> 
> 
> References
>1. http://id.gentoo.org
>2. http://id.gentoo.org
>3. http://gentoo.org

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Last standing Python 2.7 dependency

2020-05-03 Thread Fabian Groffen
On 02-05-2020 23:24:42 -0700, Brian Dolbec wrote:
> On Sun, 3 May 2020 07:28:50 +0200
> Viktar Patotski  wrote:
> 
> > Hi all,
> > 
> > I'd also like to clean my system and have it Python 2.7 free. Are
> > there any guidelines to check which packages are still using pyton2_7
> > in my system?
> > 
> > Thanks,
> > Viktar
> > 
> 
> There are both equery and enalyze commands in gentoolkit that can give
> you reports about what pkgs are installed.
> 
> equery hasuse
> enalyze analyze [use|pkguse]
> 
> for help on them:
> equery -h
> equery hasuse -h
> enalyze -h
> enalyze a -h

In addition to these great tools, portage-utils' quse might also be
useful:

% quse python2_7
...


Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz

2020-04-26 Thread Fabian Groffen
On 26-04-2020 21:29:42 +0200, Michał Górny wrote:
> On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote:
> > Bug: https://bugs.gentoo.org/715108
> > Signed-off-by: Matt Turner 
> > ---
> > Strawman patch. Bikeshed away.
> > 
> 
> xz is generally slow and doesn't do parallel good.  If we want to change
> this, we should go for something cool like zstd that scales better.

I'd go for zstd too.  It seems to be the best of both worlds, good
compression at a good speed.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [PATCH] fcaps.eclass: disable fcaps() on Prefix.

2020-03-07 Thread Fabian Groffen
On 08-03-2020 15:26:50 +0800, hero...@gentoo.org wrote:
> From: Benda Xu 
> 
> Gentoo Prefix runs with a normal user and cannot grant extra
> capabilities.  Exit gracefully with a message.
> 
> Signed-off-by: Benda Xu 
> ---
>  eclass/fcaps.eclass | 5 +
>  1 file changed, 5 insertions(+)
> 
> diff --git a/eclass/fcaps.eclass b/eclass/fcaps.eclass
> index 467f955f5e9a..563d177c92d5 100644
> --- a/eclass/fcaps.eclass
> +++ b/eclass/fcaps.eclass
> @@ -78,6 +78,11 @@ DEPEND="filecaps? ( sys-libs/libcap )"
>  fcaps() {
>   debug-print-function ${FUNCNAME} "$@"
>  
> + if [[ ${EUID} != 0 ]] ; then
> + einfo "Insufficient privileges to execute ${FUNCNAME[0]}"
> + return 0

Perhaps add "ignoring" or something in the message here to make it clear
this isn't treated as an error?

Thanks,
Fabian

> + fi
> +
>   # Process the user options first.
>   local owner='root'
>   local group='0'
> -- 
> 2.25.0
> 
> 

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Keywordreqs and slacking arch teams

2019-12-28 Thread Fabian Groffen
On 28-12-2019 22:27:02 +1300, Kent Fredric wrote:
> On Sat, 28 Dec 2019 08:09:33 +0100
> Michał Górny  wrote:
> 
> > How can we improve this?
> 
> Every time this kind of issue comes up, I can't help feeling we need
> some sort of tool more advanced than what we currently have.
> 
> Something that maintains persistence of keyword demands similar to how
> the current bot does, but more ... 
> 
> Its the sort of thing I get tempted to implement myself, but get too
> horrified about the prospect of working with portage internals to do it.

Hmmm, interested to hear what kind of things you're thinking about here.

I feel it would help if we would have the ability to at least
compile/test ebuilds automatically.  Not sure how helpful qemu could be
there, but I know things like compiling for things like arm (RPi) works
reasonably well.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.

2019-12-13 Thread Fabian Groffen
On 13-12-2019 14:24:33 -0500, Michael Orlitzky wrote:
> On 12/13/19 9:28 AM, Fabian Groffen wrote:
> > 
> > We are providing those patches, maybe.  In reality very often the
> > patches originate from somewhere else though.  And you don't want to
> > have to respin all of those just because.  At least that's what I feel.
> > 
> 
> Just because... the context changed? A new "!" in a line of context can
> be the difference between letting someone log in with the right password
> and letting them log in with the wrong password. You should at least
> have to stop and verify that the patch does what you think it does when
> it "gains" fuzz. And at that point, git diff will give you a clean
> version of it.

Counter argument is that we've been doing this for decades, and always
relied on maintainers to check the contents of their patches, without
problems.  We didn't introduce a predictable random number generator or
something.
Your very specific example just illustrates the niche this proposal is
targetting.

As with many of the proposals lately, they just seem to aim at more work
for individual maintainers, with a very low gain ratio.
Better, allow this to be a FEATURE, or whatever, that devs should enable,
but don't spit this in the user's face by default.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.

2019-12-13 Thread Fabian Groffen
On 13-12-2019 15:24:40 +0100, Michał Górny wrote:
> > > ...and why do we consider it correct to apply patches when the context
> > > doesn't match?  If our only goal is to make things 'easier' for
> > > 'everyone', then we could just pass -F and ignore all the context.
> > > 
> > > Though I don't understand why include any context in the first place if
> > > you don't care about it matching.  Sounds like a waste of space to me!
> > 
> > The patch command defaults to -F2. If that makes no sense, why is it
> > the upstream default?
> > 
> 
> You should ask upstream, not me.  But if I were to guess, the answer
> would be because patch(1) is used by random people trying to apply
> random patches they've found somewhere.  We on the other hand are
> applying patches that *we* are supposed to provide.

We are providing those patches, maybe.  In reality very often the
patches originate from somewhere else though.  And you don't want to
have to respin all of those just because.  At least that's what I feel.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Fabian Groffen
On 29-10-2019 15:45:34 +0100, Michał Górny wrote:
> On Tue, 2019-10-29 at 15:33 +0100, Fabian Groffen wrote:
> > In addition, there are currently files there that aren't referenced from
> > ebuilds.  Prefix uses these files during bootstrap, local mirrors are
> > often much faster than dev.g.o.
> > 
> > If the files don't get mirrored anymore, I guess I can create a dummy
> > ebuild that has the files in SRC_URI.
> 
> Ok, this is something I wasn't aware of.  I agree that dummy ebuild
> should not be necessary here.  However, I'm also not sure if distfiles-
> local is really the proper way either, especially that I don't see such
> files on woodpecker right now.

There should be /space/distfiles-local and
/space/distfiles-whitelist/prefix with a list of files to retain on the
mirror.

Thanks,
Fabian

> I don't think the matter is urgent right now, so let's ponder on it
> a bit.  In particular, I think we should have a clear indication of who
> added which files, when, what for and where they came from.  Those are
> precisely the things that the current distfiles-local approach misses.
> 
> > If the files get mirrored, but put in a subdir based on the filename
> > hash, the original query endpoint on distfiles.g.o changes, much like
> > the SRC_URI approach.
> > 
> > Now I can use distfiles.prefix.b.n which redirects to the distfiles.g.o
> > URL with subdir for most part I think, but it's sub-optimal from my
> > point of view.  Calculating the hash is not always feasible due to the
> > lack of b2sum or other means.  Hence my earlier request to have such
> > official translation service on Gentoo hardware.
> > 
> > (I just wrote a small wsgi script that calculates the hash and generates
> > the redirect from Python, served via uwsgi/nginx, but there should be
> > many ways to achieve the same goals, if and only if a blake2b
> > implementation were available for it.)
> 
> This is also something that needs thinking.  I personally don't mind
> having one but it would be nice if it was able to account for geodns
> and such.
> 
> -- 
> Best regards,
> Michał Górny
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Fabian Groffen
On 29-10-2019 15:17:38 +0100, Ulrich Mueller wrote:
> >>>>> On Tue, 29 Oct 2019, Michał Górny wrote:
> 
> > On Tue, 2019-10-29 at 14:09 +0100, Ulrich Mueller wrote:
> >> > What if the file is hosted at a non-standard tcp port upstream
> >> > (like http://example.org:8080/)? The devmanual says that it _must_
> >> > be manually uploaded to /space/distfiles-local/ in such cases.
> 
> >> Or another example, app-emacs/vhdl-mode-3.38.1, where (incompetent,
> >> or nasty?) upstream blocks wget for some reason, but other methods
> >> (e.g., curl, firefox) work? How would I get the file onto the mirrors
> >> there?
> 
> > If I were you, I would've explicitly mirrored the file anyway.
> > If upstream blocks wget, then users who do not use GENTOO_MIRRORS will
> > also suffer due to it.
> 
> All what I'm saying is that there can be unusual circumstances where
> manual uploading of a file is useful. So please don't take that
> possibility away.

In addition, there are currently files there that aren't referenced from
ebuilds.  Prefix uses these files during bootstrap, local mirrors are
often much faster than dev.g.o.

If the files don't get mirrored anymore, I guess I can create a dummy
ebuild that has the files in SRC_URI.

If the files get mirrored, but put in a subdir based on the filename
hash, the original query endpoint on distfiles.g.o changes, much like
the SRC_URI approach.

Now I can use distfiles.prefix.b.n which redirects to the distfiles.g.o
URL with subdir for most part I think, but it's sub-optimal from my
point of view.  Calculating the hash is not always feasible due to the
lack of b2sum or other means.  Hence my earlier request to have such
official translation service on Gentoo hardware.

(I just wrote a small wsgi script that calculates the hash and generates
the redirect from Python, served via uwsgi/nginx, but there should be
many ways to achieve the same goals, if and only if a blake2b
implementation were available for it.)

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Fabian Groffen
On 29-10-2019 05:27:37 +0100, Michał Górny wrote:
> On Tue, 2019-10-29 at 00:24 +0100, Chí-Thanh Christopher Nguyễn wrote:
> > Hi!
> > 
> > > Today you get chastised for using /space/distfiles-local and not
> > > following policy changes.  The devmanual states that it's deprecated
> > > since at least 2011, and talks of using d.g.o [1].
> >  > [1] 
> > https://devmanual.gentoo.org/general-concepts/mirrors/index.html#suitable-download-hosts
> > 
> > Sorry I'm late to the party, but I would like to enquire about what happens 
> > if a file with existing filename but different b2sum gets uploaded to 
> > /space/distfiles-local now?
> 
> The same as before.  It gets put in top-level disfiles directory. 
> Hashes are calculated from filenames, so this wouldn't affect it.  That
> is, if it put those files in subdirectories in the first place because
> it doesn't.

/space/distfiles-local is no longer copied to the mirrors? or just not
copied in the subdir-hierarchy?

> > Doing so and updating the Manifest used to be another (not necessarily 
> > preferred) method to address upstream remaking release packages.
> > 
> 
> It's no longer valid.

Just wondering.  Do you mean it isn't valid that some upstreams do this
(yes horror)?  We surely need a way to work around that ...

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Fabian Groffen
Hi,

On 18-10-2019 15:41:32 +0200, Michał Górny wrote:
> 3. Directly fetching files from distfiles.gentoo.org will become
> a little harder.  To fetch a distfile named 'foo-1.tar.gz', you'd have
> to use something like:
> 
> $ printf '%s' foo-1.tar.gz | b2sum | cut -c1-2
> 1b
> $ wget http://distfiles.gentoo.org/distfiles/1b/foo-1.tar.gz
> ...
> 
> 
> Alternatively, you can:
> 
> $ wget http://distfiles.gentoo.org/distfiles/INDEX
> 
> and grep for the right path there.  This INDEX is also a more
> lightweight alternative to HTML indexes generated by the servers.

Would it be possible to run a service that sends a 302 for the
distfiles/foo-1.tar.gz to the appropriate bucket such that manual
fetching doesn't require to calculate the hash?

I prototyped this myself for distfiles.prefix, and seems like a nice
guesture for at least the transition period?

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Underscores in USE flags

2019-09-21 Thread Fabian Groffen
On 21-09-2019 09:06:01 +0200, Michał Górny wrote:
> On Sat, 2019-09-21 at 08:43 +0200, Fabian Groffen wrote:
> > Why not teach our tools (equery, quse, etc.) to print these USE-flags
> > like Portage does?  (looking them up to be valid expands)
> > Then users have nothing to be confused about (no distinction between
> > foo_bar and FOO="bar"), and new USE_EXPANDS cannot be
> > silently/accidentially introduced.
> 
> I don't see how that solves the problem.  More tools having distinct
> output don't change the fact that anyone with a bit of ebuild knowledge
> will say 'this looks like USE_EXPAND' while looking at it, independently
> of what some tools would say.

Well... someone with a bit of ebuild knowledge would see odd USE-flags.
USE_EXPAND is a (bad) hack, of having some USE-flags mean something
different, or resolve through something different, while in reality they
really don't do anything odd.  In fact, sometimes users have to use
FOO="bar" (make.conf), while other times foo_bar needs to be used
(e.g. use.mask, or IUSE=).

Consistency would be nice, the real question is, what does USE_EXPAND
actually try to achieve, and can we fix it properly in the next EAPI,
such that repoman can also do the proper complaints about USE-flag
(and USE_EXPAND-flag) naming by then.

Back to the thread, the point is, these flags exist today, and renaming
flags is not something to be considered harmless.  As much as the recent
renaming of lm_sensors to lm-sensors caused breakage (and still does,
apparently some tools keep caches, etc.) also renaming USE-flags goes by
problems, in particular for managed systems (Chef, Puppet).  It's not a
matter of just fixing the name for a USE-flag.  This is saying nothing
about whether or not we'd want to change the flag.  It's about the
impact of the change, and whether that is worth it for the noble aim of
consistency or correctness.  I believe this was the OPs point in this
thread.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Underscores in USE flags

2019-09-21 Thread Fabian Groffen
On 20-09-2019 22:53:53 +0200, Michał Górny wrote:
> On Fri, 2019-09-20 at 13:46 -0700, Zac Medico wrote:
> > 
> > If we take this underscore rule to its logical extreme, then we should
> > rename python_targets_python3_7 to python_targets_python3-7, yes?
> 
> Believe me, I would have done that already if not the fact that with all
> the dependency logic around here it would be totally destructive to all
> Gentoo systems.

Honestly, with this reasoning, why force other packages to go through
USE-flag renaming in that case?  A major consumer of USE_EXPAND isn't
sticking to the rule, which makes any benefit of it moot.  Tools cannot
assume the last underscore separates the USE_EXPAND var from its value,
users cannot see what is the value either, without knowledge.

Why not teach our tools (equery, quse, etc.) to print these USE-flags
like Portage does?  (looking them up to be valid expands)
Then users have nothing to be confused about (no distinction between
foo_bar and FOO="bar"), and new USE_EXPANDS cannot be
silently/accidentially introduced.

> But hey, expect hyphen on 3.8.

I honestly feel for consistency and not confusing users, we should
either do them all or stick to the current scheme.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] Last rites: app-portage/hashgen

2019-09-15 Thread Fabian Groffen
# Fabian Groffen  (2019-09-15)
# Incorporated in >=app-portage/portage-utils-0.80 as qmanifest
# Removal in 30 days.  Bug #694428
app-portage/hashgen


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [PATCH] use.desc: Introduce global USE=magic

2019-08-20 Thread Fabian Groffen
Hi Michał,

It looks like you missed two from your original set?

app-editors/nano[magic] Add magic file support (sys-apps/file) to
automatically detect appropriate syntax highlighting
www-servers/pshs[magic] Enable automatic detection of Content-Type using
libmagic (sys-apps/file)

Thanks,
Fabian

On 19-08-2019 22:04:12 +0200, Michał Górny wrote:
> On Sun, 2019-08-11 at 13:21 +0200, Michał Górny wrote:
> > USE=magic is currently used consistently by 12 packages:
> > 
> 
> Merged.
> 
> -- 
> Best regards,
> Michał Górny
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [RFC] Adding extra vars to md5-cache, for QA purposes

2019-07-26 Thread Fabian Groffen
Hi,

On 25-07-2019 14:20:50 +0200, Michał Górny wrote:
> Hi,
> 
> TL;DR: I'd like to make it possible for ebuilds to define additional
> variables that will be stored in md5-cache.  This would be useful for CI
> and other tooling that right now has to parse ebuilds for other data.

Only downside I can think of, is a diskspace increase for the md5-cache.
Not sure if this is going to be substantial, but given things like
PYTHON_COMPAT, perhaps a quick calculation of extra "cost" can be made.
Should diskspace become a problem, one could consider to use a separate
file/dir, that users could rsync-exclude, since Portage won't need it to
operate properly.

Thanks,
Fabian

> 
> 
> The idea is to add a new incremental ebuild/eclass variable (technical
> name: QA_EXTRA_CACHE_VARS) that would define additional data to be
> stored in cache.  For example, python*-r1 eclasses would define
> 'PYTHON_COMPAT', acct-user would define 'ACCT_USER_ID', etc.
> 
> When regenerating cache, the PM would read this variable, and store
> the values of all defined variables into md5-cache.  As a result,
> programs needing those variables can get them straight from cache
> without having to attempt to run or parse ebuilds (which is both slow
> and prone to bugs).
> 
> This would benefit e.g. gpyutils that right now need to attempt to parse
> PYTHON_COMPAT from ebuilds.  It would also benefit writing future
> pkgcheck checks for user/group ID collisions.
> 
> 
> Notes:
> 
> - since md5-cache uses key-value format and allows for future
> extensions, the new values can be added without breaking anything;
> 
> - md5-cache is not specified in the PMS, and the whole thing can be
> implemented without need for EAPI bump,
> 
> - I would like to have this implemented consistently both in Portage
> and pkgcore,
> 
> - we will need to clearly define how to dump arrays.
> 
> 
> What do you think?
> 
> -- 
> Best regards,
> Michał Górny
> 



-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [RFC] New QA policy: Packages must not disable installing manpages via USE flags

2019-07-21 Thread Fabian Groffen
On 21-07-2019 23:47:02 +1200, Kent Fredric wrote:
> Yes, yes, I'm suggesting something perverted like a build server or
> system for aggregating built man-pages on gentoo-servers automatically
> as part of CI, that end users can just trivially fetch. But that's just
> one approach, surely, there are others.

Right, or we go for some (official) form of binpkgs distribution.

Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-dev] Re: RFC: isodate for packages.mask starting on 2019-07-01

2019-06-29 Thread Fabian Groffen
I think ISO 8601 date format is an improvement.

However, as you're suggesting to use a date at UTC time, this begs for
scripting, such as the old echangelog did.  Then, when scripted, the
actual date format is no longer an issue.  Somehow the current format
looks easier to read (for humans) to me, so no real need to change if
there was a script to just add/remove/update entries.  (And commit
without forgetting to add --signoff.)

Fabian

On 29-06-2019 13:23:10 +0200, Jonas Stein wrote:
> Dear all,
> 
> Situation:
> We have different date formats in packages.mask.
> 
> 
> Change:
> I suggest that we start using the date format -mm-dd
> for all dates in packages.mask
> starting with 2019-07-01
> 
> The following changes in packages.mask will introduce the date format,
> specify the timezone, and use Larry as example user.
> 
> 3,6c3,6
> < # When you add an entry to the top of this file, add your name, the
> date, and
> < # an explanation of why something is getting masked. Please be extremely
> < # careful not to commit atoms that are not valid, as it can cause
> large-scale
> < # breakage, especially if it ends up in the daily snapshot.
> ---
> > # When you add an entry to the top of this file, add your name, the date
> > # in the UTC timezone, and an explanation of why something is getting
> masked.
> > # Please be extremely careful not to commit atoms that are not valid,
> as it can
> > # cause large-scale breakage, especially if it ends up in the daily
> snapshot.
> 10c10
> < ## # Dev E. Loper  (28 Jun 2012)
> ---
> > ## # Larry the cow  (2019-07-01)
> 24,26c24,26
> < ## # Dev E. Loper  (23 May 2015)
> < ## # Masked for removal in 30 days.  Doesn't work
> < ## # with new libfoo. Upstream dead, gtk-1, smells
> ---
> > ## # Larry the cow  (2019-07-01)
> > ## # Masked for removal after 2019-08-01.
> > ## # Doesn't work with new libfoo. Upstream dead, gtk-1, smells
> 
> 
> Reason:
> * Larry is the Gentoo Example
> * 2019-01-01 + 30 days is unclear, if we do not use UTC time
> * The new date format is easy to read and write and easy to parse
> internationally.
> 
> Do you have any objections?
> 
> 
> By the way, you can get a formatted string of now in UTC with:
> date -u +"%Y-%m-%d"
> 
> -- 
> Best,
> Jonas
> 




-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] Deprecation and removal of 13.0 profiles is imminent

2019-02-20 Thread Fabian Groffen
On 19-02-2019 22:52:10 +, Sergei Trofimovich wrote:
> A few months passed and we almost finished with a long tail.
> 
> To-be-deprecated profiles yet are:
>   https://bugs.gentoo.org/673278
> prefix/linux-standalone/ppc64
> 
> +prefix@, what is our plan here? Can we drop at least empty
> 'deprecated' file to notify users?

I cleaned it up, it isn't in use any more.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval

2019-01-11 Thread Fabian Groffen
Pushed, thanks

On 10-01-2019 20:37:16 -0800, Zac Medico wrote:
> On 1/10/19 7:30 AM, Fabian Groffen wrote:
> > The reporting of files remaining can look somewhat odd since the report
> > interval is hardcoded to be per 1000 objects.  Adjust this interval to
> > be time based.  This means that modern (fast) machines likely will never
> > see the countdown messages at all.  On slow setups the message will be
> > informative that there is progress, albeit rather slowly.  While at it,
> > report percentage done.
> > 
> > Output before this patch:
> > 
> >  * checking 6158 files for package collisions
> > 5158 files remaining ...
> > 4158 files remaining ...
> > 3158 files remaining ...
> > 2158 files remaining ...
> > 1158 files remaining ...
> > 158 files remaining ...
> > 
> > Possible output after this patch on a slower machine:
> > 
> >  * checking 6158 files for package collisions
> >  48% done,  3145 files remaining ...
> >  96% done,   192 files remaining ...
> > 100% done
> > 
> > Signed-off-by: Fabian Groffen 
> > ---
> >  lib/portage/dbapi/vartree.py | 15 +--
> >  1 file changed, 13 insertions(+), 2 deletions(-)
> > 
> > diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> > index 4b91caea8..909c0e473 100644
> > --- a/lib/portage/dbapi/vartree.py
> > +++ b/lib/portage/dbapi/vartree.py
> > @@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
> > 'portage.util.install_mask:install_mask_dir,InstallMask',
> > 'portage.util.listdir:dircache,listdir',
> > 'portage.util.movefile:movefile',
> > +   'portage.util.monotonic:monotonic',
> > 'portage.util.path:first_existing,iter_parents',
> > 'portage.util.writeable_check:get_ro_checker',
> > 'portage.util._xattr:xattr',
> > @@ -3475,13 +3476,21 @@ class dblink(object):
> > symlink_collisions = []
> > destroot = self.settings['ROOT']
> > totfiles = len(file_list) + len(symlink_list)
> > +   previous = monotonic()
> > +   progress_shown = False
> > +   report_interval = 1.7  # seconds
> > +   falign = len("%d" % totfiles)
> > showMessage(_(" %s checking %d files for package 
> > collisions\n") % \
> > (colorize("GOOD", "*"), totfiles))
> > for i, (f, f_type) in enumerate(chain(
> > ((f, "reg") for f in file_list),
> > ((f, "sym") for f in symlink_list))):
> > -   if i % 1000 == 0 and i != 0:
> > -   showMessage(_("%d files remaining 
> > ...\n") % (totfiles - i))
> > +   current = monotonic()
> > +   if current - previous > report_interval:
> > +   showMessage(_("%3d%% done,  %*d files 
> > remaining ...\n") %
> > +   (i * 100 / totfiles, 
> > falign, totfiles - i))
> > +   previous = current
> > +   progress_shown = True
> >  
> > dest_path = normalize_path(
> > os.path.join(destroot, 
> > f.lstrip(os.path.sep)))
> > @@ -3570,6 +3579,8 @@ class dblink(object):
> > break
> > if stopmerge:
> > collisions.append(f)
> > +   if progress_shown:
> > +   showMessage(_("100% done\n"))
> > return collisions, dirs_ro, symlink_collisions, 
> > plib_collisions
> >  
> > def _lstat_inode_map(self, path_iter):
> > 
> 
> Looks good!
> -- 
> Thanks,
> Zac
> 




-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval

2019-01-10 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done,  3145 files remaining ...
 96% done,   192 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 15 +--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..909c0e473 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
+   'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3475,13 +3476,21 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   previous = monotonic()
+   progress_shown = False
+   report_interval = 1.7  # seconds
+   falign = len("%d" % totfiles)
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   current = monotonic()
+   if current - previous > report_interval:
+   showMessage(_("%3d%% done,  %*d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
falign, totfiles - i))
+   previous = current
+   progress_shown = True
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3579,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if progress_shown:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




Re: [gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval

2019-01-10 Thread Fabian Groffen
On 09-01-2019 21:22:26 -0800, Zac Medico wrote:
> > +   if tnow + tinterv < ninterv:
> > +   showMessage(_("100% done\n"))
> 
> It took me a moment to understand the calculation here. How about if we
> do something like this instead:
> 
> 
> previous = monotonic()
> progress_shown = False

You're right.  I should've gone for readability in the first place.
There is no point in trying to save anything here.

Thanks,
Fabian

> for f in files:
>   current = monotonic()
>   if current - previous > tinterv:
>   showMessage(...)
>   previous = current
>   progress_shown = True
> 
> if progress_shown:
>   showMessage(_("100% done\n"))
> 
> 
> 
> > return collisions, dirs_ro, symlink_collisions, 
> > plib_collisions
> >  
> > def _lstat_inode_map(self, path_iter):
> > 
> 
> 
> -- 
> Thanks,
> Zac
> 




-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval

2019-01-09 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done,  3145 files remaining ...
 96% done,   192 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 13 +++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..df192e6fc 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
+   'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3475,13 +3476,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   tnow = monotonic()
+   tinterv = 2  # seconds
+   ninterv = tnow + tinterv
+   falign = len("%d" % totfiles)
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if monotonic() > ninterv:
+   showMessage(_("%3d%% done,  %*d files 
remaining ...\n") %
+   (falign, i * 100 / 
totfiles, totfiles - i))
+   ninterv = monotonic() + tinterv
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3577,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if tnow + tinterv < ninterv:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




Re: [gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval

2019-01-09 Thread Fabian Groffen
On 08-01-2019 20:59:34 +, M. J. Everitt wrote:
> On 08/01/19 19:15, Zac Medico wrote:
> > On 1/8/19 5:42 AM, Fabian Groffen wrote:
> >> The reporting of files remaining can look somewhat odd since the report
> >> interval is hardcoded to be per 1000 objects.  Adjust this interval to
> >> be time based.  This means that modern (fast) machines likely will never
> >> see the countdown messages at all.  On slow setups the message will be
> >> informative that there is progress, albeit rather slowly.  While at it,
> >> report percentage done.
> >>
> >> Output before this patch:
> >>
> >>  * checking 6158 files for package collisions
> >> 5158 files remaining ...
> >> 4158 files remaining ...
> >> 3158 files remaining ...
> >> 2158 files remaining ...
> >> 1158 files remaining ...
> >> 158 files remaining ...
> >>
> >> Possible output after this patch on a slower machine:
> >>
> >>  * checking 6158 files for package collisions
> >>  48% done, 3145 files remaining ...
> >>  96% done, 192 files remaining ...
> >> 100% done
> >>
> >> Signed-off-by: Fabian Groffen 
> >> ---
> >>  lib/portage/dbapi/vartree.py | 11 +--
> >>  1 file changed, 9 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> >> index 4b91caea8..244195fad 100644
> >> --- a/lib/portage/dbapi/vartree.py
> >> +++ b/lib/portage/dbapi/vartree.py
> >> @@ -3475,13 +3475,18 @@ class dblink(object):
> >>symlink_collisions = []
> >>destroot = self.settings['ROOT']
> >>totfiles = len(file_list) + len(symlink_list)
> >> +  tnow = time.time()
> >> +  tinterv = 2  # seconds
> >> +  ninterv = tnow + tinterv
> >>showMessage(_(" %s checking %d files for package 
> >> collisions\n") % \
> >>(colorize("GOOD", "*"), totfiles))
> >>for i, (f, f_type) in enumerate(chain(
> >>((f, "reg") for f in file_list),
> >>((f, "sym") for f in symlink_list))):
> >> -  if i % 1000 == 0 and i != 0:
> >> -  showMessage(_("%d files remaining 
> >> ...\n") % (totfiles - i))
> >> +  if time.time() > ninterv:
> >> +  showMessage(_("%3d%% done, %d files 
> >> remaining ...\n") %
> >> +  (i * 100 / totfiles, 
> >> totfiles - i))
> >> +  ninterv = time.time() + tinterv
> >>  
> >>dest_path = normalize_path(
> >>os.path.join(destroot, 
> >> f.lstrip(os.path.sep)))
> >> @@ -3570,6 +3575,8 @@ class dblink(object):
> >>break
> >>if stopmerge:
> >>collisions.append(f)
> >> +  if tnow + tinterv < ninterv:
> >> +  showMessage(_("100% done\n"))
> >>return collisions, dirs_ro, symlink_collisions, 
> >> plib_collisions
> >>  
> >>def _lstat_inode_map(self, path_iter):
> >>
> > Please replace time.time() with portage.util.monotonic.monotonic().

Will do.

> It's a bit of a nit-pick, granted, but can we ensure that the count-down's
> remain padded/justified such that the numbers line up for easy at-a-glance
> inspection ? The optimal standard looks somewhat like the pre-merge Sizes:
> 
>  * Final size of build directory: 2696 KiB (2.6 MiB)
>  * Final size of installed tree:  5372 KiB (5.2 MiB)
> 
> Otherwise, I think this will be quite helpful. Thanks.

I think I understand what you mean, let me see if I can do something
without too much complexity.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval

2019-01-08 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be time based.  This means that modern (fast) machines likely will never
see the countdown messages at all.  On slow setups the message will be
informative that there is progress, albeit rather slowly.  While at it,
report percentage done.

Output before this patch:

 * checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...

Possible output after this patch on a slower machine:

 * checking 6158 files for package collisions
 48% done, 3145 files remaining ...
 96% done, 192 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..244195fad 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,18 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   tnow = time.time()
+   tinterv = 2  # seconds
+   ninterv = tnow + tinterv
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if time.time() > ninterv:
+   showMessage(_("%3d%% done, %d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
totfiles - i))
+   ninterv = time.time() + tinterv
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3575,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if tnow + tinterv < ninterv:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




Re: [gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval

2019-01-08 Thread Fabian Groffen
On 08-01-2019 09:17:04 +0100, Ulrich Mueller wrote:
> >>>>> On Tue, 08 Jan 2019, Fabian Groffen wrote:
> 
> > Output before this patch:
> 
> >  * checking 6111 files for package collisions
> >  [...]
> 
> > After:
> 
> >  * checking 6158 files for package collisions
> >  [...]
> 
> Why does the total number of files change?

First was python2, second python3.  The result was indicative only, the
number doesn't change of course.

Sorry for the confusion.

Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


[gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval

2019-01-08 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be regular towards the end.  While at it, report percentage done.

Output before this patch:

 * checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
 files remaining ...
111 files remaining ...

After:

 * checking 6158 files for package collisions
 16% done, 5131 files remaining ...
 33% done, 4104 files remaining ...
 50% done, 3077 files remaining ...
 66% done, 2050 files remaining ...
 83% done, 1023 files remaining ...
100% done

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 14 +++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..78f2b37f2 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   bucksize = 1000
+   buckcnt = int(totfiles / bucksize)
+   if buckcnt == 0 or totfiles % bucksize > int(bucksize / 
2):
+   buckcnt = buckcnt + 1
+   bucksize = int(totfiles / buckcnt) + 1
showMessage(_(" %s checking %d files for package 
collisions\n") % \
-   (colorize("GOOD", "*"), totfiles))
+   (colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if i % bucksize == 0 and i != 0:
+   showMessage(_("%3d%% done, %d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
totfiles - i))
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
@@ -3570,6 +3576,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+   if bucksize < totfiles:
+   showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions, 
plib_collisions
 
def _lstat_inode_map(self, path_iter):
-- 
2.20.1




[gentoo-portage-dev] [PATCH] lib/portage/dbapi/vartree: use dynamic report interval in _collision_protect

2019-01-07 Thread Fabian Groffen
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects.  Adjust this interval to
be regular towards the end.  While at it, report percentage done.

Output before this patch:

 * checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
 files remaining ...
111 files remaining ...

After:

 * checking 6158 files for package collisions
16% done, 5131 files remaining ...
33% done, 4104 files remaining ...
50% done, 3077 files remaining ...
66% done, 2050 files remaining ...
83% done, 1023 files remaining ...

Signed-off-by: Fabian Groffen 
---
 lib/portage/dbapi/vartree.py | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..4d0bf2789 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+   bucksize = 1000
+   buckcnt = int(totfiles / bucksize)
+   if buckcnt == 0 or totfiles % bucksize > int(bucksize / 
2):
+   buckcnt = buckcnt + 1
+   bucksize = int(totfiles / buckcnt) + 1
showMessage(_(" %s checking %d files for package 
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
-   if i % 1000 == 0 and i != 0:
-   showMessage(_("%d files remaining 
...\n") % (totfiles - i))
+   if i % bucksize == 0 and i != 0:
+   showMessage(_("%2d%% done, %d files 
remaining ...\n") %
+   (i * 100 / totfiles, 
totfiles - i))
 
dest_path = normalize_path(
os.path.join(destroot, 
f.lstrip(os.path.sep)))
-- 
2.20.1




Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format

2018-11-27 Thread Fabian Groffen
On 26-11-2018 21:13:53 +, Andrey Utkin wrote:
> On Wed, Nov 21, 2018 at 11:45:54AM +0100, Fabian Groffen wrote:
> > We agree it is hackish, and we agree we can do without.  You simply
> > exaggerate the problem, IMO, which mostly isn't there, because it works
> > fine today.  It can also be solved today using shell tools.
> 
> I am sad that you don't see it as a productivity impediment that the
> user is required to know the custom tooling to do even such a trivial
> non-standard action as manual extraction.

Huh?  tar -jxf doesn't do the trick for you?

> Maybe I will make myself look bad by admitting this, but I'm not meeting
> your expectations. I use Gentoo for ~11 years, and for about one year I
> am using my private binpkgs distributed to all my machines (i.e. I have
> read binary package guide fair number of times, but I stopped rereading
> it when I satisfied my needs). When in need, I still reached to trusty
> tar, and I did not even know what are the names of special tools (a
> toolchain?) qtbz2 and qxpak.
> 
> Just few days ago I messed with binpkgs for investigation purpose. I
> just wanted to extract few to somewhere (definitely not into system
> root), and read a core dump with GDB asking it to use those extracted
> files for debug symbols.
> 
> Of course I used `tar xaf`, because what I know is that it's honest tbz2
> just with metadata appended.
> 
>  # tar xaf boost-1.65.0.tbz2
> 
> bzip2: (stdin): trailing garbage after EOF ignored
> 
> Exit code is 0.
> But the notice is annoying (on subconscious level), because Silence Is
> Golden - "when a program has nothing interesting or surprising to say,
> it should shut up".

You seem to contradict yourself.  You didn't know the tools, yet you say
you needed to, to unpack the files.  But you show here you just unpacked
the files without said knowledge.

> > % head -c `grep -abo 'XPAKPACK' 
> > $EPREFIX/usr/portage/packages/sys-apps/sed-4.5.tbz2 | sed 's/:.*$//'` 
> > $EPREFIX/usr/portage/packages/sys-apps/sed-4.5.tbz2 | tar -jxf -
> > 
> > results in no warnings/errors from bzip about trailing garbage, possible
> > thanks to the spec being smart enough about this.
> 
> Thanks, this is a very concise **custom tool** to handle current binpkg
> format.

As is tar followed by tar.  The obvious advantage of the latter is that
you don't get a warning which could trigger you into thinking something
is wrong.  So, in my opinion, that is a better way of doing it compared
to the current way.

> > Not having to do this, when under stress and pressure to restore a
> > system to get it back into production, is a plus.  Though, in that
> > scenario the trailing garbage warning wouldn't have been that bad
> > either.
> 
> When understress and pressure, the irrelevant warning is not bad?
> I am sure it is really bad for operator's attention.

I've been using Gentoo binpkgs for a long while, I think something like
~14 years ago when I used them extensively.  Perhaps I'm an exception,
but back then I knew already there was an extra bit attached to the
tars, as were all my collegues around me back then.  The fact it comes
up now (as a surprise?) maybe means the knowledge has gone.  So good
thing we're replacing it with something easier to infer from inspecting
it.

Fabian


-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


Re: [gentoo-dev] [pre-GLEP r2] Gentoo binary package container format

2018-11-21 Thread Fabian Groffen
On 20-11-2018 21:33:17 +0100, Michał Górny wrote:
> The volume label
> 
> 
> The volume label provides an easy way for users to identify the binary
> package without dedicated tooling or specific format knowledge.
> 
> The implementations should include a volume label consisting of fixed
> string ``gpkg:``, followed by a single space, followed by full package
> identifier.  However, the implementations must not rely on the volume
> label being present or attempt to parse its value when it is.
> 
> Furthermore, since the volume label is included in the .tar archive
> as the first member, it provides a magic string at a fixed location
> that can be used by tools such as file(1) to easily distinguish Gentoo
> binary packages from regular .tar archives.

Just for clarity on this point.
Are you proposing that we patch file(1) to print the Volume Header here?
file-5.35 seems to not say much but "tar archive" or "POSIX tar archive"
for tar-files containing a Volume Header as shown by tar -tv.

> Container and archive formats
> -
> 
> During the debate, the actual archive formats to use were considered.
> The .tar format seemed an obvious choice for the image archive since
> it is the only widely deployed archive format that stores all kinds
> of file metadata on POSIX systems.  However, multiple options for
> the outer format has been debated.

You mention POSIX, which triggered me.  I think it would be good to
specify which tar format to use.

POSIX.1-2001/pax format doesn't have a 100/256 char filename length
restriction, which is good but it is not (yet) used by default by GNU
tar.  busybox tar can read pax tars, it seems.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature


  1   2   3   4   5   6   7   >