Re: [gentoo-portage-dev] [PATCH] Remove obsolete FEATURES=force-prefix
Sounds fine to me too. On 22-01-2023 09:16:53 +, James Le Cuirot wrote: > On Sun, 2023-01-22 at 09:18 +0100, Ulrich Müller wrote: > > Signed-off-by: Ulrich Müller > > --- > > bin/dohtml.py| 6 ++ > > bin/eapi.sh | 4 ++-- > > lib/portage/const.py | 3 +-- > > lib/portage/package/ebuild/config.py | 11 +++ > > man/make.conf.5 | 8 +--- > > 5 files changed, 9 insertions(+), 23 deletions(-) > > +1 -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
On 28-07-2022 20:11:24 +0200, Fabian Groffen wrote:
> On 28-07-2022 19:57:50 +0200, Arfrever Frehtes Taifersar Arahesis wrote:
> > But this code starts with 'if [[ -z ${USERLAND} ]]', so it should not
> > be run when USERLAND="GNU".
> >
> > USERLAND="GNU" is set in make.defaults, which is parsed by Portage
> > early (at Python side) and these values are passed by Portage to
> > ebuild.sh subprocess.
>
> Hmmm, that would be nice, features/prefix/make.defaults includes setting
> that.
>
> Makes me still wonder why we do this, is it a fallback for when the
> profiles aren't available (no tree yet) or something?
>
> It exports XARGS, which only used in the ___parallel_xargs function.
> That function in turn is used in 2 places. So all this stuff is done to
> add -r to xargs if xargs isn't BSD xargs. Iow --no-run-if-empty, which
> is cool, but apparently not strictly necessary (fugly warning I guess at
> worst). We're already checking xargs to support --max-procs=, so we
> could in the same go check for support of --no-run-if-empty, and drop
> the whole XARGS thing and declutter the code quite a lot :)
Sorry, I grepped wrong. XARGS is used throughout the code, so it needs
to be initialised in isolated-functions.sh. It could be initialised by
simply probing instead of relying on uname though.
As for USERLAND, emerge-webrsync and delta-webrsync appear to use it,
but they retrieve it from portageq, so I guess USERLAND is not really
used in Portage itself.
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
On 28-07-2022 19:57:50 +0200, Arfrever Frehtes Taifersar Arahesis wrote:
> 2022-07-28 17:47 UTCに、Fabian Groffen は書いた:
> > bin/isolated-functions.sh does the following bit:
> >
> > if [[ -z ${USERLAND} ]] ; then
> >case $(uname -s) in
> >*BSD|DragonFly)
> >export USERLAND="BSD"
> >;;
> >*)
> >export USERLAND="GNU"
> >;;
> >esac
> > fi
> >
> > (after which it uses USERLAND for a single purpose, to export XARGS to
> > either GNU '[g]xargs -r', or BSD 'xargs')
> >
> > This bit is problematic for Prefix, because Prefix may run on *BSD, but
> > use USERLAND=GNU.
>
> But this code starts with 'if [[ -z ${USERLAND} ]]', so it should not
> be run when USERLAND="GNU".
>
> USERLAND="GNU" is set in make.defaults, which is parsed by Portage
> early (at Python side) and these values are passed by Portage to
> ebuild.sh subprocess.
Hmmm, that would be nice, features/prefix/make.defaults includes setting
that.
Makes me still wonder why we do this, is it a fallback for when the
profiles aren't available (no tree yet) or something?
It exports XARGS, which only used in the ___parallel_xargs function.
That function in turn is used in 2 places. So all this stuff is done to
add -r to xargs if xargs isn't BSD xargs. Iow --no-run-if-empty, which
is cool, but apparently not strictly necessary (fugly warning I guess at
worst). We're already checking xargs to support --max-procs=, so we
could in the same go check for support of --no-run-if-empty, and drop
the whole XARGS thing and declutter the code quite a lot :)
Thanks,
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
Hi,
bin/isolated-functions.sh does the following bit:
if [[ -z ${USERLAND} ]] ; then
case $(uname -s) in
*BSD|DragonFly)
export USERLAND="BSD"
;;
*)
export USERLAND="GNU"
;;
esac
fi
(after which it uses USERLAND for a single purpose, to export XARGS to
either GNU '[g]xargs -r', or BSD 'xargs')
This bit is problematic for Prefix, because Prefix may run on *BSD, but
use USERLAND=GNU. The problem is theoretical at this point,
occasionally people come in and want to use Prefix on *BSD again.
Now I think Gentoo/BSD is theoretical too. It officially is no longer
maintained[1].
So, question here is, do we want to retain this bit of historical
compatibility work (also check things like gsed wrapper), or can it go,
basically removing a potential problem for Gentoo Prefix?
[1] https://wiki.gentoo.org/wiki/Gentoo_FreeBSD
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
On 26-07-2022 09:20:58 +0200, Fabian Groffen wrote:
> On 26-07-2022 09:03:18 +0200, Florian Schmaus wrote:
> > On 26.07.22 05:00, Sam James wrote:
> > >> On 25 Jul 2022, at 16:28, Fabian Groffen wrote:
> > >>
> > >> bin/ebuild-helpers/emake: force SHELL to be set
> > >>
> [snip]
> > >>
> > >> diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
> > >> index 60718a2e4..21da85845 100755
> > >> --- a/bin/ebuild-helpers/emake
> > >> +++ b/bin/ebuild-helpers/emake
> > >> @@ -12,7 +12,7 @@
> > >> source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
> > >>
> > >> cmd=(
> > >> -${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
> > >> +${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@"
> > >> ${EXTRA_EMAKE}
> > >> )
> > >>
> > >> if [[ ${PORTAGE_QUIET} != 1 ]] ; then
> > >>
> > >
> > > I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh
> > > to avoid using
> > > an ancient host sh?
> >
> > I was about to write the same (also using EPREFIX, but EBROOT seems what
> > you want, as you figured).
> >
> > But then I wondered if "make SHELL=$BROOT/bin/sh" wouldn't override
> > explicitly set SHELL values in Makefiles. Assume a package has
> >
> > SHELL = /bin/zsh
> >
> > in one of its Makefiles. Then emake would reset this to 'sh'. Which
> > appears like it could cause build issues.
> >
> > If this is the case, then I am not sure what we can do about it. It
> > appears fragile, if not impossible, to ask 'make' which value for SHELL
> > it would assume, so that emake could adjust the path. Another option
> > could be that affected packages define a variable in their ebuild, e.g.
> > EMAKE_SHELL="zsh", which emake could extend with BROOT before passing
> > the resulting value as SHELL to make.
>
> So, I can also envision we drop this patch, and I see if I can patch
> make(1) to use $EPREFIX/bin/sh instead of /bin/sh by default. Not sure,
> but this would retain the behaviour Portage is doing now for non-Prefix,
> and would get the behaviour we want in Prefix.
>
> On an alternative note, there is CONFIG_SHELL (used for setting which shell
> to use with configure), which I think in many cases bleeds through to
> make, but should there be a MAKE_SHELL perhaps as well? Then the
> default would be pretty much ok.
>
> (We never ran into any problems forcing SHELL to bash in Prefix, but
> perhaps that's not a representative test for the whole of Gentoo.)
I've been looking around in make, and it seems we can set a default
shell there, would be pretty simple, I guess. It doesn't solve,
however, a bigger problem, which is what make's source also mentions,
lots of Makefiles having SHELL=/bin/sh. In other words, setting a
default in make makes no difference in preventing it from using /bin/sh
(which is the main aim here).
Therefore, I would like to consider the possibility to override SHELL
this way via emake, as it seems like the fix we need (for Prefix at
least) afterall.
Overriding should be safe, I think. SHELL=/bin/zsh makes little sense
to me, SHELL=/bin/csh would be more of a thing, but is there any package
out there that needs it? And if it does, wouldn't it be acceptable to
just handle that in that package? It would require a dep on that shell
anyway (so you could consider it a kludgy sanity check as well).
I think using bash like the original patch did isn't quite nice, it
should use sh instead. However, it cannot hardcode EPREFIX/bin/sh
without ensuring that binary exists, else we break bootstraps.
So I was thinking: how about something like SHELL=$(type -P sh)? To be
completely safe here, it could become an if such that if there's no sh,
it falls back to ${BASH}.
Would this approach be acceptable? Should it be perhaps conditional
based on whether an offset prefix is active?
Thanks,
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] phase-functions: make ED, EROOT read-only
On 25-07-2022 19:43:21 -0400, Mike Gilbert wrote: > On Mon, Jul 25, 2022 at 1:03 PM Fabian Groffen wrote: > > > > bin/phase-functions.sh: make ED and EROOT read-only too > > > > Like D, make ED and EROOT read-only vars. > > Makes sense. https://github.com/gentoo/portage/pull/870 Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] misc-functions: Prefix fixes
On 25-07-2022 19:42:51 -0400, Mike Gilbert wrote: > On Mon, Jul 25, 2022 at 12:47 PM Fabian Groffen wrote: > > > > bin/misc-functions.sh: some Prefix fixes > > > > - ED needs not to exist, whereas D does, so ensure we check for that, > > and create ED if absent, necessary for further checks to succeed > > - use EPREFIX in INSTALL_MASK > > Seems good to me. https://github.com/gentoo/portage/pull/870 -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] world-writable: tune for Prefix
On 25-07-2022 19:39:38 -0400, Mike Gilbert wrote: > On Mon, Jul 25, 2022 at 12:41 PM Fabian Groffen wrote: > > > > bin/install-qa-check.d/90world-writable: include EPREFIX in reports > > > > It is much less confusing and consistent to report full paths including > > the leading EPREFIX. > > Makes sense to me. https://github.com/gentoo/portage/pull/869 thanks -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] multilib-strict: fix for Prefix
On 25-07-2022 19:38:57 -0400, Mike Gilbert wrote:
> On Mon, Jul 25, 2022 at 12:26 PM Fabian Groffen wrote:
> >
> > bin/install-qa-check.d/80multilib-strict: use file/find from Prefix
> >
> > diff --git a/bin/install-qa-check.d/80multilib-strict
> > b/bin/install-qa-check.d/80multilib-strict
> > index afd223250..3db4ecce3 100644
> > --- a/bin/install-qa-check.d/80multilib-strict
> > +++ b/bin/install-qa-check.d/80multilib-strict
> > @@ -1,7 +1,7 @@
> > # Strict multilib directory checks
> > multilib_strict_check() {
> > if has multilib-strict ${FEATURES} && \
> > - [[ -x /usr/bin/file && -x /usr/bin/find ]] && \
> > + [[ -x ${EPREFIX}/usr/bin/file && -x ${EPREFIX}/usr/bin/find ]]
> > && \
> >[[ -n ${MULTILIB_STRICT_DIRS} && -n ${MULTILIB_STRICT_DENY} ]]
> > then
> > rm -f "${T}/multilib-strict.log"
>
> Seems good to me.
https://github.com/gentoo/portage/pull/868
Thanks
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets
On 26-07-2022 04:01:37 +0100, Sam James wrote:
>
>
> > On 26 Jul 2022, at 00:33, Mike Gilbert wrote:
> >
> > On Mon, Jul 25, 2022 at 11:38 AM Fabian Groffen wrote:
> >>
> >> bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects
> >>
> >> Check for dylib on Darwin, so on everything else.
> >>
> >> Signed-off-by: Fabian Groffen
> >>
> >> diff --git a/bin/install-qa-check.d/80libraries
> >> b/bin/install-qa-check.d/80libraries
> >> index 8dc35bb87..a477ec9cb 100644
> >> --- a/bin/install-qa-check.d/80libraries
> >> +++ b/bin/install-qa-check.d/80libraries
> >> @@ -140,7 +140,9 @@ lib_check() {
> >>local abort="no"
> >>local a s
> >>for a in "${ED%/}"/usr/lib*/*.a ; do
> >> - s=${a%.a}.so
> >> + [[ ${CHOST} == *-darwin* ]] \
> >> + && s=${a%.a}.dylib \
> >> + || s=${a%.a}.so
> >
> > I would find this much easier to read if you converted it to an
> > if/else statement instead of chaining && and ||.
>
> Yes, please.
https://github.com/gentoo/portage/pull/867
Thanks
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
On 25-07-2022 19:29:35 -0400, Mike Gilbert wrote: > Could you please create a PR at https://github.com/gentoo/portage so > that the CI system can test the changes for this patch series? https://github.com/gentoo/portage/pull/866 Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
On 25-07-2022 19:50:41 +0200, Ulrich Mueller wrote:
> >>>>> On Mon, 25 Jul 2022, Fabian Groffen wrote:
>
> > @@ -50,6 +51,16 @@ if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}"
> > ]]; then
> > __helpers_die "${helper} used with \${D} or \${ED}"
> > exit 1
> > fi
> > +if [[ -n ${EPREFIX} && \
> > + ${_E_INSDESTTREE_#${EPREFIX}} != "${_E_INSDESTTREE_}" ]];
>
> The semicolon is redundant.
removed, thanks
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
On 26-07-2022 09:03:18 +0200, Florian Schmaus wrote:
> On 26.07.22 05:00, Sam James wrote:
> >> On 25 Jul 2022, at 16:28, Fabian Groffen wrote:
> >>
> >> bin/ebuild-helpers/emake: force SHELL to be set
> >>
[snip]
> >>
> >> diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
> >> index 60718a2e4..21da85845 100755
> >> --- a/bin/ebuild-helpers/emake
> >> +++ b/bin/ebuild-helpers/emake
> >> @@ -12,7 +12,7 @@
> >> source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
> >>
> >> cmd=(
> >> - ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
> >> + ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
> >> )
> >>
> >> if [[ ${PORTAGE_QUIET} != 1 ]] ; then
> >>
> >
> > I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh to
> > avoid using
> > an ancient host sh?
>
> I was about to write the same (also using EPREFIX, but EBROOT seems what
> you want, as you figured).
>
> But then I wondered if "make SHELL=$BROOT/bin/sh" wouldn't override
> explicitly set SHELL values in Makefiles. Assume a package has
>
> SHELL = /bin/zsh
>
> in one of its Makefiles. Then emake would reset this to 'sh'. Which
> appears like it could cause build issues.
>
> If this is the case, then I am not sure what we can do about it. It
> appears fragile, if not impossible, to ask 'make' which value for SHELL
> it would assume, so that emake could adjust the path. Another option
> could be that affected packages define a variable in their ebuild, e.g.
> EMAKE_SHELL="zsh", which emake could extend with BROOT before passing
> the resulting value as SHELL to make.
So, I can also envision we drop this patch, and I see if I can patch
make(1) to use $EPREFIX/bin/sh instead of /bin/sh by default. Not sure,
but this would retain the behaviour Portage is doing now for non-Prefix,
and would get the behaviour we want in Prefix.
On an alternative note, there is CONFIG_SHELL (used for setting which shell
to use with configure), which I think in many cases bleeds through to
make, but should there be a MAKE_SHELL perhaps as well? Then the
default would be pretty much ok.
(We never ran into any problems forcing SHELL to bash in Prefix, but
perhaps that's not a representative test for the whole of Gentoo.)
Thanks,
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] phase-functions: make ED, EROOT read-only
bin/phase-functions.sh: make ED and EROOT read-only too Like D, make ED and EROOT read-only vars. Signed-off-by: Fabian Groffen diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh index ccf7eeea7..212b19fc1 100644 --- a/bin/phase-functions.sh +++ b/bin/phase-functions.sh @@ -12,7 +12,7 @@ PORTAGE_READONLY_METADATA="BDEPEND DEFINED_PHASES DEPEND DESCRIPTION PDEPEND RDEPEND REPOSITORY RESTRICT SLOT SRC_URI" PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE EBUILD_PHASE_FUNC \ - EBUILD_SH_ARGS EMERGE_FROM FILESDIR MERGE_TYPE \ + EBUILD_SH_ARGS ED EMERGE_FROM EROOT FILESDIR MERGE_TYPE \ PM_EBUILD_HOOK_DIR \ PORTAGE_ACTUAL_DISTDIR PORTAGE_ARCHLIST PORTAGE_BASHRC \ PORTAGE_BINPKG_FILE PORTAGE_BINPKG_TAR_OPTS PORTAGE_BINPKG_TMPFILE \ -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] misc-functions: Prefix fixes
bin/misc-functions.sh: some Prefix fixes
- ED needs not to exist, whereas D does, so ensure we check for that,
and create ED if absent, necessary for further checks to succeed
- use EPREFIX in INSTALL_MASK
Signed-off-by: Fabian Groffen
diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index c8bac08e7..8fcc23588 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -19,6 +19,8 @@ source "${PORTAGE_BIN_PATH}/ebuild.sh" || exit 1
install_symlink_html_docs() {
if ! ___eapi_has_prefix_variables; then
local ED=${D}
+ else
+ [[ ! -d ${ED} && -d ${D} ]] && dodir /
fi
cd "${ED}" || die "cd failed"
# Symlink the html documentation (if DOC_SYMLINKS_DIR is set in
make.conf)
@@ -83,7 +87,7 @@ install_qa_check() {
local EPREFIX= ED=${D}
fi
- cd "${ED}" || die "cd failed"
+ cd "${D}" || die "cd failed"
# Collect the paths for QA checks, highest prio first.
paths=(
@@ -367,7 +718,7 @@ preinst_mask() {
local f x
for f in man info doc; do
if has no${f} ${FEATURES}; then
- INSTALL_MASK+=" /usr/share/${f}"
+ INSTALL_MASK+=" ${EPREFIX}/usr/share/${f}"
fi
done
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] world-writable: tune for Prefix
bin/install-qa-check.d/90world-writable: include EPREFIX in reports
It is much less confusing and consistent to report full paths including
the leading EPREFIX.
Signed-off-by: Fabian Groffen
diff --git a/bin/install-qa-check.d/90world-writable
b/bin/install-qa-check.d/90world-writable
index 820683bd6..90b961a86 100644
--- a/bin/install-qa-check.d/90world-writable
+++ b/bin/install-qa-check.d/90world-writable
@@ -2,7 +2,7 @@
world_writable_check() {
# Now we look for all world writable files.
- local unsafe_files=$(find "${ED}" -type f -perm -2 | sed -e
"s:^${ED}:/:")
+ local unsafe_files=$(find "${ED}" -type f -perm -2 | sed -e
"s:^${D}:/:")
local OLDIFS x prev_shopts=$-
OLDIFS=$IFS
@@ -19,7 +21,7 @@ world_writable_check() {
eqawarn
fi
- local unsafe_files=$(find "${ED}" -type f '(' -perm -2002 -o -perm
-4002 ')' | sed -e "s:^${ED}:/:")
+ local unsafe_files=$(find "${ED}" -type f '(' -perm -2002 -o -perm
-4002 ')' | sed -e "s:^${D}:/:")
if [[ -n ${unsafe_files} ]] ; then
eqawarn "QA Notice: Unsafe files detected (set*id and world
writable)"
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] multilib-strict: fix for Prefix
bin/install-qa-check.d/80multilib-strict: use file/find from Prefix
diff --git a/bin/install-qa-check.d/80multilib-strict
b/bin/install-qa-check.d/80multilib-strict
index afd223250..3db4ecce3 100644
--- a/bin/install-qa-check.d/80multilib-strict
+++ b/bin/install-qa-check.d/80multilib-strict
@@ -1,7 +1,7 @@
# Strict multilib directory checks
multilib_strict_check() {
if has multilib-strict ${FEATURES} && \
- [[ -x /usr/bin/file && -x /usr/bin/find ]] && \
+ [[ -x ${EPREFIX}/usr/bin/file && -x ${EPREFIX}/usr/bin/find ]] && \
[[ -n ${MULTILIB_STRICT_DIRS} && -n ${MULTILIB_STRICT_DENY} ]]
then
rm -f "${T}/multilib-strict.log"
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets
bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects
Check for dylib on Darwin, so on everything else.
Signed-off-by: Fabian Groffen
diff --git a/bin/install-qa-check.d/80libraries
b/bin/install-qa-check.d/80libraries
index 8dc35bb87..a477ec9cb 100644
--- a/bin/install-qa-check.d/80libraries
+++ b/bin/install-qa-check.d/80libraries
@@ -140,7 +140,9 @@ lib_check() {
local abort="no"
local a s
for a in "${ED%/}"/usr/lib*/*.a ; do
- s=${a%.a}.so
+ [[ ${CHOST} == *-darwin* ]] \
+ && s=${a%.a}.dylib \
+ || s=${a%.a}.so
if [[ ! -e ${s} ]] ; then
s=${s%usr/*}${s##*/usr/}
if [[ -e ${s} ]] ; then
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
bin/ebuild-helpers/emake: force SHELL to be set
On Prefix systems /bin/sh can be anything, including very ancient. So
ensure we're running with bash, since that's what Gentoo Linux is
expecting /bin/sh to be (by default, at least).
Provide a fallback for the (near impossible) case that we use a bash
that doesn't set BASH, or when we don't use bash at all. This is not
expected, though, as we explicitly require bash throughout all Portage,
so we don't really care about using a non-Prefixed one, for this really
shouldn't happen.
Signed-off-by: Fabian Groffen
diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
index 60718a2e4..21da85845 100755
--- a/bin/ebuild-helpers/emake
+++ b/bin/ebuild-helpers/emake
@@ -12,7 +12,7 @@
source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
cmd=(
- ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
+ ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
)
if [[ ${PORTAGE_QUIET} != 1 ]] ; then
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] usage of /bin/bash in shebangs
On 25-07-2022 08:09:02 -0700, Zac Medico wrote: > On 7/24/22 23:17, Fabian Groffen wrote: > > On 24-07-2022 13:58:31 -0700, Zac Medico wrote: > >> On 7/24/22 12:29, Fabian Groffen wrote: > >>> Hi, > >>> > >>> Quick question, I noticed that portage uses /bin/bash hardcoded in > >>> shebang of scripts, while it uses /usr/bin/env python for python > >>> executable files. > >>> > >>> Is there anything against using /usr/bin/env bash for shell scripts? > >>> Changing this would help for Prefix. > >> > >> > >> I can't think of any reason not to do this. > > > > Do you want to see a patch, or is it OK to push this myself? > > It's OK to just push this yourself. Thanks! Pushed, thanks, b716e3591 -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
bin/ebuild-helpers/doins: fix D check, add EPREFIX check
ED = D/EPREFIX, so checking for ED includes EPREFIX, which when this is
absent fails to check for D. Simply check for D instead, which catches
both the case for D and ED.
Add check for usage of EPREFIX, like for using D with helpers.
Signed-off-by: Fabian Groffen
diff --git a/bin/ebuild-helpers/doins b/bin/ebuild-helpers/doins
index 24fe48121..4315a038f 100755
--- a/bin/ebuild-helpers/doins
+++ b/bin/ebuild-helpers/doins
@@ -42,7 +42,7 @@ if ! ___eapi_has_prefix_variables; then
export ED="${D}"
fi
-if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" ]]; then
+if [[ ${_E_INSDESTTREE_#${D}} != "${_E_INSDESTTREE_}" ]]; then
__vecho "---" 1>&2
__vecho "You should not use \${D} or \${ED} with helpers." 1>&2
__vecho " --> ${_E_INSDESTTREE_}" 1>&2
@@ -50,6 +51,16 @@ if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" ]];
then
__helpers_die "${helper} used with \${D} or \${ED}"
exit 1
fi
+if [[ -n ${EPREFIX} && \
+ ${_E_INSDESTTREE_#${EPREFIX}} != "${_E_INSDESTTREE_}" ]];
+then
+ __vecho "---" 1>&2
+ __vecho "You should not use \${EPREFIX} with helpers." 1>&2
+ __vecho " --> ${_E_INSDESTTREE_}" 1>&2
+ __vecho "---" 1>&2
+ __helpers_die "${helper} used with \${EPREFIX}"
+ exit 1
+fi
if ___eapi_doins_and_newins_preserve_symlinks; then
DOINS_ARGS+=( --preserve_symlinks )
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] usage of /bin/bash in shebangs
On 24-07-2022 13:58:31 -0700, Zac Medico wrote: > On 7/24/22 12:29, Fabian Groffen wrote: > > Hi, > > > > Quick question, I noticed that portage uses /bin/bash hardcoded in > > shebang of scripts, while it uses /usr/bin/env python for python > > executable files. > > > > Is there anything against using /usr/bin/env bash for shell scripts? > > Changing this would help for Prefix. > > > I can't think of any reason not to do this. Do you want to see a patch, or is it OK to push this myself? Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] usage of /bin/bash in shebangs
Hi, Quick question, I noticed that portage uses /bin/bash hardcoded in shebang of scripts, while it uses /usr/bin/env python for python executable files. Is there anything against using /usr/bin/env bash for shell scripts? Changing this would help for Prefix. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] Changing the VDB format
Hi, I've recently been thinking about this too. On 13-03-2022 18:06:21 -0700, Matt Turner wrote: > The VDB uses a one-file-per-variable format. This has some > inefficiencies, with many file systems. For example the 'EAPI' file > that contains a single character will consume a 4K block on disk. > I recommend json and think it is the best choice because: [snip] > - json provides the smallest on-disk footprint > - json is part of Python's standard library (so is yaml, and toml will > be in Python 3.11) > - Every programming language has multiple json parsers > -- lots of effort has been spent making them extremely fast. I would like to suggest to use "tar". The reason behind this is a bit convoluted, but I try to be as clear and sound as I can: - "new style" bin-packages use tar too - tar-file allows to keep all individual files/members, e.g. for legacy tools to unpack and look at the VDB that way - tar-file allows streaming, so single file read, for efficient retrieval - single tar-file for entire VDB, allows to make it "atomic", one can modify tar archives lateron to add new vdb entries, or perform updates -- again, without inplace (e.g. memory backing) this could be done atomic) - tar-file could be used for (rsync) tree metadata (md5-cache) in the same way, e.g. re-use streaming approach, or unpack for legacy tools - tar-file could be used for Packages file, instead of flat file with keys, basically just write VDB entries with some additional keys, very similar in practise. - tar-files are slightly easier to manage from command line, tools to do so exist for a long time and are installed. (jq isn't pulled in by @system these days, I think) - tar-files can easily (optionally) be compressed retaining streaming abilities (this is for these usages very likely to pay off), a much higher dictionary benefit for a single tar vs many files. - single tar-file is much more efficient to GPG-sign (which would allow some securing of the VDB, not sure if useful though) - going back to the first point, vdb entry from binary package could simply be dropped into the vdb tar, and vice-versa - going back to metadata, dep-resolving could simply load the entire system available/installed packages with two reads in memory (if it has enough of that -- pretty common these days), which should allow for vast speedups, especially on cold(ish) filesystems. > I think we would have a significant time period for the transition. I > think I would include support for the new format in Portage, and ship > a tool with portage to switch back and forth between old and new > formats on-disk. Maybe after a year, drop the code from Portage to > support the old format? Here I believe that with tar-format, initially code could be written to instead of accessing a file directly, it could open up the tar-file, locate the member it needs, and then retrieve that instead. This is a bit naive, but probably sort of managable, and allows to having a switch that specifies which format to write. It's easy to detect which form you have automatically. E.g. nothing has to change for users unless they actively make a change for it. Like you, I think the main reason for doing this should be performance, basically allowing faster operations. I feel though that we should aim to use a single solution to maintain a number of "trees" that we have: metadata, vdb, Packages/binpkgs, for they all seem to exhibit a similar (IO) behaviour when being employed. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal
On 04-12-2021 10:24:23 +0100, Michał Górny wrote: > On Sat, 2021-12-04 at 10:15 +0100, Fabian Groffen wrote: > > On 04-12-2021 10:13:09 +0100, Michał Górny wrote: > > > On Sat, 2021-12-04 at 09:56 +0100, Fabian Groffen wrote: > > > > Why don't you change your color.map instead of changing the default for > > > > everyone? > > > > > > Why should we keep a stupid default? Should we optimize Gentoo for > > > people who don't want to be able to read Portage's output? > > > > You're assuming everyone uses the terminal in the way you do. I simply > > don't think that's how the world looks like. > > On the other hand, you're assuming that everyone uses the terminal > in the way you do. It eludes me how you came to that conclusion. > > No need for calling things stupid, IMO. > > Using dark blue on black background is stupid. ... then don't use black background or dark blue text? Now, if you would make a supported claim that all terminals we install use a black background by default, your change becomes more valid. However, we then still don't know if people leave that default or use something else, but we could make some educated guess about the amount of people not changing the default. My point, because I think this wasn't clear to you, is and always was, how many people is this change going to be disruptive to. And should we make a hint to users when they install this version of Portage that they can revert/change this by altering color.map (and how)? Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal
On 04-12-2021 10:13:09 +0100, Michał Górny wrote: > On Sat, 2021-12-04 at 09:56 +0100, Fabian Groffen wrote: > > Why don't you change your color.map instead of changing the default for > > everyone? > > Why should we keep a stupid default? Should we optimize Gentoo for > people who don't want to be able to read Portage's output? You're assuming everyone uses the terminal in the way you do. I simply don't think that's how the world looks like. No need for calling things stupid, IMO. -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] portage.output: Replace darkblue colors with teal
Why don't you change your color.map instead of changing the default for
everyone?
Thanks,
Fabian
On 04-12-2021 05:48:49 +0100, Michał Górny wrote:
> The "darkblue" color is often barely visible on dark terminals which
> makes reading emerge output really hard (I basically have to copy-paste
> it a lot in order to be able to read it at all). Replace it with teal
> that does not seem to have any significant use in the output.
>
> Signed-off-by: Michał Górny
> ---
> lib/portage/output.py | 6 +++---
> man/color.map.5 | 6 +++---
> 2 files changed, 6 insertions(+), 6 deletions(-)
>
>
> The before-after screenshot can be found on the PR:
> https://github.com/gentoo/portage/pull/776
>
>
> diff --git a/lib/portage/output.py b/lib/portage/output.py
> index 42f487f8a..77375b012 100644
> --- a/lib/portage/output.py
> +++ b/lib/portage/output.py
> @@ -157,7 +157,7 @@ _styles["UNMERGE_WARN"] = ("red",)
> _styles["SECURITY_WARN"] = ("red",)
> _styles["MERGE_LIST_PROGRESS"] = ("yellow",)
> _styles["PKG_BLOCKER"] = ("red",)
> -_styles["PKG_BLOCKER_SATISFIED"] = ("darkblue",)
> +_styles["PKG_BLOCKER_SATISFIED"] = ("teal",)
> _styles["PKG_MERGE"] = ("darkgreen",)
> _styles["PKG_MERGE_SYSTEM"] = ("darkgreen",)
> _styles["PKG_MERGE_WORLD"] = ("green",)
> @@ -165,8 +165,8 @@ _styles["PKG_BINARY_MERGE"] = ("purple",)
> _styles["PKG_BINARY_MERGE_SYSTEM"] = ("purple",)
> _styles["PKG_BINARY_MERGE_WORLD"] = ("fuchsia",)
> _styles["PKG_UNINSTALL"] = ("red",)
> -_styles["PKG_NOMERGE"] = ("darkblue",)
> -_styles["PKG_NOMERGE_SYSTEM"] = ("darkblue",)
> +_styles["PKG_NOMERGE"] = ("teal",)
> +_styles["PKG_NOMERGE_SYSTEM"] = ("teal",)
> _styles["PKG_NOMERGE_WORLD"] = ("blue",)
> _styles["PROMPT_CHOICE_DEFAULT"] = ("green",)
> _styles["PROMPT_CHOICE_OTHER"] = ("red",)
> diff --git a/man/color.map.5 b/man/color.map.5
> index 288bf7fbd..92a1baa91 100644
> --- a/man/color.map.5
> +++ b/man/color.map.5
> @@ -40,7 +40,7 @@ Defines color used for numbers indicating merge progress.
> \fBPKG_BLOCKER\fR = \fI"red"\fR
> Defines color used for unsatisfied blockers.
> .TP
> -\fBPKG_BLOCKER_SATISFIED\fR = \fI"darkblue"\fR
> +\fBPKG_BLOCKER_SATISFIED\fR = \fI"teal"\fR
> Defines color used for satisfied blockers.
> .TP
> \fBPKG_MERGE\fR = \fI"darkgreen"\fR
> @@ -63,10 +63,10 @@ package.
> Defines color used for world packages planned to be merged using a binary
> package.
> .TP
> -\fBPKG_NOMERGE\fR = \fI"darkblue"\fR
> +\fBPKG_NOMERGE\fR = \fI"teal"\fR
> Defines color used for packages not planned to be merged.
> .TP
> -\fBPKG_NOMERGE_SYSTEM\fR = \fI"darkblue"\fR
> +\fBPKG_NOMERGE_SYSTEM\fR = \fI"teal"\fR
> Defines color used for system packages not planned to be merged.
> .TP
> \fBPKG_NOMERGE_WORLD\fR = \fI"blue"\fR
> --
> 2.34.1
>
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [RFC] LTS branch of Portage
In all fairness, there haven't been that much incidents with Portage in the past under Zac's supervision, isn't it a bit overkill to bureaucratise the release model just after one incident? It appears to me that changes to Portage need to be considered very carefully, always, since it affects everyone. Thanks, Fabian On 05-10-2021 10:31:40 +0200, Michał Górny wrote: > Hi, everyone. > > I've been thinking about this for some time already, and the recent > FILESDIR mess seems to confirm it: I'd like to start a more stable LTS > branch of Portage. > > Roughly, the idea is that: > > - master becomes 3.1.x, and primary development happens there > > - 3.0.x becomes the LTS branch and only major bugfixes are backported > there > > As things settle down in the future, master would become 3.2.x, 3.1.x > would become LTS, 3.0.x will be discontinued and so on. > > WDYT? > > -- > Best regards, > Michał Górny > > > -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] In what phase are file "merged"?
On 23-06-2021 08:47:58 +0200, Ulrich Mueller wrote: > >>>>> On Wed, 23 Jun 2021, Joakim Tjernlund wrote: > > > In PMS 9.2 is says: > > The call order for upgrading, downgrading or reinstalling a package is: > > > pkg_pretend (only for EAPIs listed in table 9.2), which is called > > outside of the normal call order process. > > pkg_setup > > src_unpack > > src_prepare (only for EAPIs listed in table 9.3) > > src_configure (only for EAPIs listed in table 9.4) > > src_compile > > src_test (except if RESTRICT=test) > > src_install > > pkg_preinst > > pkg_prerm for the package being replaced > > pkg_postrm for the package being replaced > > pkg_postinst > > > It does not say where in this list new files merged and old files > > unmerged, can anyone enlighten me? > > It's somewhat hidden, but it's there: > https://projects.gentoo.org/pms/8/pms.html#x1-950009.1.10 > >9.1.10 pkg_preinst >... immediately before merging the package to the live filesystem. ... > >9.1.11 pkg_postinst >... immediately after merging the package to the live filesystem. ... Aha, so does this mean pkg_prerm and pkg_postrm are run with replacing package in place, e.g. if they refer to scripts installed by the replaced package they may no longer exist or be the same? Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] KeyError: 'ED' ?
On 09-06-2021 14:19:38 -0400, Mike Gilbert wrote: > On Thu, May 6, 2021 at 6:54 AM Joakim Tjernlund > wrote: > > > > On Wed, 2021-05-05 at 16:47 +, Joakim Tjernlund wrote: > > > I am upgrading an old portage(2.3.76) to 3.0.18 using qmerge(binary pkg) > > > and I get this: > > > qmerge -OK sys-apps/portage > > > [R] sys-apps/portage-3.0.18 > > > * Checking for suitable kernel configuration options... > > > [ ok ] > > > * Using python3.8 in global scope > > > FEATURES variable contains unknown value(s): disabled > > > Traceback (most recent call last): > > > File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main > > > return _run_code(code, main_globals, None, > > > File "/usr/lib/python3.8/runpy.py", line 87, in _run_code > > > exec(code, run_globals) > > > File > > > "/var/tmp/qmerge/sys-apps/portage-3.0.18/image/usr/lib/python3.8/site-packages/portage/_compat_upgrade/default_locations.py", > > > line 93, in > > > main() > > > File > > > "/var/tmp/qmerge/sys-apps/portage-3.0.18/image/usr/lib/python3.8/site-packages/portage/_compat_upgrade/default_locations.py", > > > line 63, in main > > > config_path = os.path.join(os.environ['ED'], > > > GLOBAL_CONFIG_PATH.lstrip(os.sep), 'make.globals') > > > File "/usr/lib/python3.8/os.py", line 675, in __getitem__ > > > raise KeyError(key) from None > > > KeyError: 'ED' > > > > > > so portage fails to install properly, I am guessin this is becuse I have > > > an old portage to begin with? > > > Can it be fixed? > > > > The error goes away when I do: ED=/ qmerge -OK sys-apps/portage > > Does that mean that it is qmerge(aka portage-utils) that needs to set ED > > during merge? which PHASES ? > > According to PMS, a package manager must define ED in src_install, > pkg_preinst, and pkg_postinst. Perhaps qmerge should use export for these vars. In any case this seems tracked in bug 792273. -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] _get_lock_fn: support multiprocessing spawn start method (bug 758230)
Thanks Zac!
On 04-12-2020 14:58:22 -0800, Zac Medico wrote:
> Ensure that _get_lock_fn arguments to multiprocessing.Process will
> successfully pickle, as required by the spawn start method, which
> is the default for macOS since Python 3.8.
>
> Since file descriptors are not inherited unless the fork start
> method is used, the subprocess should only try to close an
> inherited file descriptor for the fork start method.
>
> Bug: https://bugs.gentoo.org/758230
> Signed-off-by: Zac Medico
> ---
> lib/portage/locks.py | 36 +++-
> 1 file changed, 23 insertions(+), 13 deletions(-)
>
> diff --git a/lib/portage/locks.py b/lib/portage/locks.py
> index 1073343be..193045c03 100644
> --- a/lib/portage/locks.py
> +++ b/lib/portage/locks.py
> @@ -44,18 +44,7 @@ def _get_lock_fn():
> if _lock_fn is not None:
> return _lock_fn
>
> - def _test_lock(fd, lock_path):
> - os.close(fd)
> - try:
> - with open(lock_path, 'a') as f:
> - fcntl.lockf(f.fileno(),
> fcntl.LOCK_EX|fcntl.LOCK_NB)
> - except EnvironmentError as e:
> - if e.errno == errno.EAGAIN:
> - # Parent process holds lock, as expected.
> - sys.exit(0)
>
> - # Something went wrong.
> - sys.exit(1)
>
> fd, lock_path = tempfile.mkstemp()
> try:
> @@ -64,8 +53,16 @@ def _get_lock_fn():
> except EnvironmentError:
> pass
> else:
> - proc = multiprocessing.Process(target=_test_lock,
> - args=(fd, lock_path))
> + proc = multiprocessing.Process(
> + target=_subprocess_test_lock,
> + args=(
> + # Since file descriptors are not
> inherited unless the fork start
> + # method is used, the subprocess should
> only try to close an
> + # inherited file descriptor for the
> fork start method.
> + fd if
> multiprocessing.get_start_method() == "fork" else None,
> + lock_path,
> + ),
> + )
> proc.start()
> proc.join()
> if proc.exitcode == os.EX_OK:
> @@ -80,6 +77,19 @@ def _get_lock_fn():
> _lock_fn = fcntl.flock
> return _lock_fn
>
> +def _subprocess_test_lock(fd, lock_path):
> + if fd is not None:
> + os.close(fd)
> + try:
> + with open(lock_path, 'a') as f:
> + fcntl.lockf(f.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB)
> + except EnvironmentError as e:
> + if e.errno == errno.EAGAIN:
> + # Parent process holds lock, as expected.
> + sys.exit(0)
> +
> + # Something went wrong.
> + sys.exit(1)
>
> _open_fds = {}
> _open_inodes = {}
> --
> 2.26.2
>
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] Speeding up Tree Verification
On 30-06-2020 13:13:29 -0500, Sid Spry wrote: > On Tue, Jun 30, 2020, at 1:20 AM, Fabian Groffen wrote: > > Hi, > > > > On 29-06-2020 21:13:43 -0500, Sid Spry wrote: > > > Hello, > > > > > > I have some runnable pseudocode outlining a faster tree verification > > > algorithm. > > > Before I create patches I'd like to see if there is any guidance on > > > making the > > > changes as unobtrusive as possible. If the radical change in algorithm is > > > acceptable I can work on adding the changes. > > > > > > Instead of composing any kind of structured data out of the portage tree > > > my > > > algorithm just lists all files and then optionally batches them out to > > > threads. > > > There is a noticeable speedup by eliding the tree traversal operations > > > which > > > can be seen when running the algorithm with a single thread and comparing > > > it to > > > the current algorithm in gemato (which should still be discussed here?). > > > > I remember something that gemato used to use multiple threads, but > > because it totally saturated disk-IO, it was brought back to a single > > thread. People were complaining about unusable systems. > > > > I think this is an argument for cgroups limits support on the portage process > or > account as opposed to an argument against picking a better algorithm. That is > something I have been working towards, but I am only one man. But this requires a) cgroups support, and b) the privileges to use it. Shouldn't be a problem in the normal case, but just saying. > > In any case, can you share your performance results? What speedup did > > you see, on warm and hot FS caches? Which type of disk do you use? > > > > I ran all tests multiple times to make them warm off of a Samsung SSD, but > nothing very precise yet. > > % gemato verify --openpgp-key signkey.asc /var/db/repos/gentoo > [...] > INFO:root:Verifying /var/db/repos/gentoo... > INFO:root:/var/db/repos/gentoo verified in 16.45 seconds > > sometimes going higher, closer to 18s, vs. > > % ./veriftree.py > 4.763171965983929 > > So roughly an order of magnitude speedup without batching to threads. That is kind of a change. Makes one wonder if you really did the same work. > > You could compare against qmanifest, which uses OpenMP-based > > paralllelism while verifying the tree. On SSDs this does help. > > > > I lost my notes -- how do I specify to either gemato or qmanifest the GnuPG > directory? My code is partially structured as it is because I had problems > doing > this. I rediscovered -K/--openpgp-key in gemato but am unsure for qmanifest. qmanifest doesn't do much magic out of the standard gnupg practices. (It is using gpgme.) If you want it to use a different gnupg dir, you may change HOME, or GNUPGHOME. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] Speeding up Tree Verification
ass
>
> return 0
>
> if __name__ == '__main__':
> sys.exit(main())
> ```
>
> ```python (wkd.py, likely unneeded but I didn't want to redo these files yet)
> #!/usr/bin/env python3
> import sys, hashlib
> import dns
> from dns import (
> name, query, dnssec,
> message, resolver, rdatatype
> )
> import shutil, requests
>
> def check_domain_signature(domain: str) -> bool:
> response = dns.resolver.query(domain, dns.rdatatype.NS)
> nsname = response.rrset[0]
> response = dns.resolver.query(str(nsname), dns.rdatatype.A)
> nsaddr = response.rrset[0].to_text()
>
> # DNSKEY
> request = dns.message.make_query(domain,
> dns.rdatatype.DNSKEY, want_dnssec=True)
> response = dns.query.udp(request, nsaddr)
> if response.rcode() != 0:
> raise Exception('Unable to request dnskey.')
>
> answer = response.answer
> if len(answer) != 2:
> raise Exception('Malformed answer to dnskey query.')
>
> name = dns.name.from_text(domain)
> try:
> dns.dnssec.validate(answer[0], answer[1], {name: answer[0]})
> except dns.dnssec.ValidationFailure as exc:
> # Validation failed. The raise causes python to abort with status 1.
> #raise exc
> return False
> except AttributeError as exc:
> # Validation may have failed; DNSKEY missing signer attribute. dig
> may report
> # domain as valid.
> #
> # TODO: Additional state where subdomain of valid domain may fail
> with 3 nested
> # KeyErrors. Avoid temptation to wildcard catch. Safer to put in
> process?
> #raise exc
> return False
> else:
> return True
>
> def hash_localpart(incoming: bytes) -> str:
> '''Z-base32 the localpart of an e-mail address
>
>
> https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-08#section-3.1
> describes why this is needed.
>
> See https://tools.ietf.org/html/rfc6189#section-5.1.6 for a
> description of the z-base32 scheme.
> '''
> zb32 = "ybndrfg8ejkmcpqxot1uwisza345h769"
>
> b = hashlib.sha1(incoming).digest()
> ret = ""
> assert(len(b) * 8 == 160)
> for i in range(0, 160, 5):
> byte = i // 8
> offset = i - byte * 8
> # offset | bits remaining in k+1 | right-shift k+1
> # 3 | 0 | x
> # 4 | 1 | 7
> # 5 | 2 | 6
> # 6 | 3 | 5
> # 7 | 4 | 4
> if offset < 4:
> n = (b[byte] >> (3 - offset))
> else:
> n = (b[byte] << (offset - 3)) + (b[byte + 1] >> (11 - offset))
>
> ret += zb32[n & 0b1]
> return ret
>
> def build_web_key_uri(address: str) -> str:
> local, remote = address.split('@')
> local = hash_localpart(local.encode('utf-8'))
> return 'https://' + remote + '/.well-known/openpgpkey/hu/' + \
> local
>
> def stream_to_file(uri: str, fname: str) -> None:
> with requests.get(uri, verify=True, stream=True) as r:
> from pprint import pprint
> pprint(r.headers)
> with open(fname, 'wb') as f:
> shutil.copyfileobj(r.raw, f)
> ```
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] Add caching to a few commonly used functions
Hi Chun-Yu, On 26-06-2020 23:34:12 -0700, Chun-Yu Shei wrote: > Hi, > > I was recently interested in whether portage could be speed up, since > dependency resolution can sometimes take a while on slower machines. > After generating some flame graphs with cProfile and vmprof, I found 3 > functions which seem to be called extremely frequently with the same > arguments: catpkgsplit, use_reduce, and match_from_list. In the first > two cases, it was simple to cache the results in dicts, while > match_from_list was a bit trickier, since it seems to be a requirement > that it return actual entries from the input "candidate_list". I also > ran into some test failures if I did the caching after the > mydep.unevaluated_atom.use and mydep.repo checks towards the end of the > function, so the caching is only done up to just before that point. > > The catpkgsplit change seems to definitely be safe, and I'm pretty sure > the use_reduce one is too, since anything that could possibly change the > result is hashed. I'm a bit less certain about the match_from_list one, > although all tests are passing. > > With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world" > speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup. "emerge > -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 sec > (2.5% improvement). Since the upgrade case is far more common, this > would really help in daily use, and it shaves about 30 seconds off > the time you have to wait to get to the [Yes/No] prompt (from ~90s to > 60s) on my old Sandy Bridge laptop when performing normal upgrades. > > Hopefully, at least some of these patches can be incorporated, and please > let me know if any changes are necessary. This sounds like a good job to me! Do you have any idea what the added memory pressure for these changes are? Thanks, Fabian > > Thanks, > Chun-Yu > > > -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] Use env to find python
On 16-06-2020 23:10:28 -0500, Sid Spry wrote: > On Tue, Jun 16, 2020, at 3:57 PM, Michał Górny wrote: > > '/usr/bin/env python' (with no extra options) is the portable shebang. > > > > I added `-S` to preserve the options passed via the shebang line. It seems > they can be left off, does anyone know otherwise? -S is not supported on some platforms. In any case, I think this patch is wrong. In the Prefix case, the shebangs are set during install, in the non-Prefix case I think there is machinery in place to replace the shebangs during install. It seems the problem is already solved, why do you need these shebangs changed? Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] Change BINPKG_COMPRESS default from bzip2 to xz
On 26-04-2020 21:29:42 +0200, Michał Górny wrote: > On Sun, 2020-04-26 at 09:55 -0700, Matt Turner wrote: > > Bug: https://bugs.gentoo.org/715108 > > Signed-off-by: Matt Turner > > --- > > Strawman patch. Bikeshed away. > > > > xz is generally slow and doesn't do parallel good. If we want to change > this, we should go for something cool like zstd that scales better. I'd go for zstd too. It seems to be the best of both worlds, good compression at a good speed. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.
On 13-12-2019 14:24:33 -0500, Michael Orlitzky wrote: > On 12/13/19 9:28 AM, Fabian Groffen wrote: > > > > We are providing those patches, maybe. In reality very often the > > patches originate from somewhere else though. And you don't want to > > have to respin all of those just because. At least that's what I feel. > > > > Just because... the context changed? A new "!" in a line of context can > be the difference between letting someone log in with the right password > and letting them log in with the wrong password. You should at least > have to stop and verify that the patch does what you think it does when > it "gains" fuzz. And at that point, git diff will give you a clean > version of it. Counter argument is that we've been doing this for decades, and always relied on maintainers to check the contents of their patches, without problems. We didn't introduce a predictable random number generator or something. Your very specific example just illustrates the niche this proposal is targetting. As with many of the proposals lately, they just seem to aim at more work for individual maintainers, with a very low gain ratio. Better, allow this to be a FEATURE, or whatever, that devs should enable, but don't spit this in the user's face by default. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] eapply: Drop -s option for patch.
On 13-12-2019 15:24:40 +0100, Michał Górny wrote: > > > ...and why do we consider it correct to apply patches when the context > > > doesn't match? If our only goal is to make things 'easier' for > > > 'everyone', then we could just pass -F and ignore all the context. > > > > > > Though I don't understand why include any context in the first place if > > > you don't care about it matching. Sounds like a waste of space to me! > > > > The patch command defaults to -F2. If that makes no sense, why is it > > the upstream default? > > > > You should ask upstream, not me. But if I were to guess, the answer > would be because patch(1) is used by random people trying to apply > random patches they've found somewhere. We on the other hand are > applying patches that *we* are supposed to provide. We are providing those patches, maybe. In reality very often the patches originate from somewhere else though. And you don't want to have to respin all of those just because. At least that's what I feel. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [RFC] Adding extra vars to md5-cache, for QA&tooling purposes
Hi, On 25-07-2019 14:20:50 +0200, Michał Górny wrote: > Hi, > > TL;DR: I'd like to make it possible for ebuilds to define additional > variables that will be stored in md5-cache. This would be useful for CI > and other tooling that right now has to parse ebuilds for other data. Only downside I can think of, is a diskspace increase for the md5-cache. Not sure if this is going to be substantial, but given things like PYTHON_COMPAT, perhaps a quick calculation of extra "cost" can be made. Should diskspace become a problem, one could consider to use a separate file/dir, that users could rsync-exclude, since Portage won't need it to operate properly. Thanks, Fabian > > > The idea is to add a new incremental ebuild/eclass variable (technical > name: QA_EXTRA_CACHE_VARS) that would define additional data to be > stored in cache. For example, python*-r1 eclasses would define > 'PYTHON_COMPAT', acct-user would define 'ACCT_USER_ID', etc. > > When regenerating cache, the PM would read this variable, and store > the values of all defined variables into md5-cache. As a result, > programs needing those variables can get them straight from cache > without having to attempt to run or parse ebuilds (which is both slow > and prone to bugs). > > This would benefit e.g. gpyutils that right now need to attempt to parse > PYTHON_COMPAT from ebuilds. It would also benefit writing future > pkgcheck checks for user/group ID collisions. > > > Notes: > > - since md5-cache uses key-value format and allows for future > extensions, the new values can be added without breaking anything; > > - md5-cache is not specified in the PMS, and the whole thing can be > implemented without need for EAPI bump, > > - I would like to have this implemented consistently both in Portage > and pkgcore, > > - we will need to clearly define how to dump arrays. > > > What do you think? > > -- > Best regards, > Michał Górny > -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval
Pushed, thanks
On 10-01-2019 20:37:16 -0800, Zac Medico wrote:
> On 1/10/19 7:30 AM, Fabian Groffen wrote:
> > The reporting of files remaining can look somewhat odd since the report
> > interval is hardcoded to be per 1000 objects. Adjust this interval to
> > be time based. This means that modern (fast) machines likely will never
> > see the countdown messages at all. On slow setups the message will be
> > informative that there is progress, albeit rather slowly. While at it,
> > report percentage done.
> >
> > Output before this patch:
> >
> > * checking 6158 files for package collisions
> > 5158 files remaining ...
> > 4158 files remaining ...
> > 3158 files remaining ...
> > 2158 files remaining ...
> > 1158 files remaining ...
> > 158 files remaining ...
> >
> > Possible output after this patch on a slower machine:
> >
> > * checking 6158 files for package collisions
> > 48% done, 3145 files remaining ...
> > 96% done, 192 files remaining ...
> > 100% done
> >
> > Signed-off-by: Fabian Groffen
> > ---
> > lib/portage/dbapi/vartree.py | 15 +--
> > 1 file changed, 13 insertions(+), 2 deletions(-)
> >
> > diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> > index 4b91caea8..909c0e473 100644
> > --- a/lib/portage/dbapi/vartree.py
> > +++ b/lib/portage/dbapi/vartree.py
> > @@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
> > 'portage.util.install_mask:install_mask_dir,InstallMask',
> > 'portage.util.listdir:dircache,listdir',
> > 'portage.util.movefile:movefile',
> > + 'portage.util.monotonic:monotonic',
> > 'portage.util.path:first_existing,iter_parents',
> > 'portage.util.writeable_check:get_ro_checker',
> > 'portage.util._xattr:xattr',
> > @@ -3475,13 +3476,21 @@ class dblink(object):
> > symlink_collisions = []
> > destroot = self.settings['ROOT']
> > totfiles = len(file_list) + len(symlink_list)
> > + previous = monotonic()
> > + progress_shown = False
> > + report_interval = 1.7 # seconds
> > + falign = len("%d" % totfiles)
> > showMessage(_(" %s checking %d files for package
> > collisions\n") % \
> > (colorize("GOOD", "*"), totfiles))
> > for i, (f, f_type) in enumerate(chain(
> > ((f, "reg") for f in file_list),
> > ((f, "sym") for f in symlink_list))):
> > - if i % 1000 == 0 and i != 0:
> > - showMessage(_("%d files remaining
> > ...\n") % (totfiles - i))
> > + current = monotonic()
> > + if current - previous > report_interval:
> > + showMessage(_("%3d%% done, %*d files
> > remaining ...\n") %
> > + (i * 100 / totfiles,
> > falign, totfiles - i))
> > + previous = current
> > + progress_shown = True
> >
> > dest_path = normalize_path(
> > os.path.join(destroot,
> > f.lstrip(os.path.sep)))
> > @@ -3570,6 +3579,8 @@ class dblink(object):
> > break
> > if stopmerge:
> > collisions.append(f)
> > + if progress_shown:
> > + showMessage(_("100% done\n"))
> > return collisions, dirs_ro, symlink_collisions,
> > plib_collisions
> >
> > def _lstat_inode_map(self, path_iter):
> >
>
> Looks good!
> --
> Thanks,
> Zac
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH v5] collision_protect: use dynamic report interval
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects. Adjust this interval to
be time based. This means that modern (fast) machines likely will never
see the countdown messages at all. On slow setups the message will be
informative that there is progress, albeit rather slowly. While at it,
report percentage done.
Output before this patch:
* checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...
Possible output after this patch on a slower machine:
* checking 6158 files for package collisions
48% done, 3145 files remaining ...
96% done, 192 files remaining ...
100% done
Signed-off-by: Fabian Groffen
---
lib/portage/dbapi/vartree.py | 15 +--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..909c0e473 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
+ 'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3475,13 +3476,21 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+ previous = monotonic()
+ progress_shown = False
+ report_interval = 1.7 # seconds
+ falign = len("%d" % totfiles)
showMessage(_(" %s checking %d files for package
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- if i % 1000 == 0 and i != 0:
- showMessage(_("%d files remaining
...\n") % (totfiles - i))
+ current = monotonic()
+ if current - previous > report_interval:
+ showMessage(_("%3d%% done, %*d files
remaining ...\n") %
+ (i * 100 / totfiles,
falign, totfiles - i))
+ previous = current
+ progress_shown = True
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
@@ -3570,6 +3579,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+ if progress_shown:
+ showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions,
plib_collisions
def _lstat_inode_map(self, path_iter):
--
2.20.1
Re: [gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval
On 09-01-2019 21:22:26 -0800, Zac Medico wrote:
> > + if tnow + tinterv < ninterv:
> > + showMessage(_("100% done\n"))
>
> It took me a moment to understand the calculation here. How about if we
> do something like this instead:
>
>
> previous = monotonic()
> progress_shown = False
You're right. I should've gone for readability in the first place.
There is no point in trying to save anything here.
Thanks,
Fabian
> for f in files:
> current = monotonic()
> if current - previous > tinterv:
> showMessage(...)
> previous = current
> progress_shown = True
>
> if progress_shown:
> showMessage(_("100% done\n"))
>
>
>
> > return collisions, dirs_ro, symlink_collisions,
> > plib_collisions
> >
> > def _lstat_inode_map(self, path_iter):
> >
>
>
> --
> Thanks,
> Zac
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH v4] collision_protect: use dynamic report interval
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects. Adjust this interval to
be time based. This means that modern (fast) machines likely will never
see the countdown messages at all. On slow setups the message will be
informative that there is progress, albeit rather slowly. While at it,
report percentage done.
Output before this patch:
* checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...
Possible output after this patch on a slower machine:
* checking 6158 files for package collisions
48% done, 3145 files remaining ...
96% done, 192 files remaining ...
100% done
Signed-off-by: Fabian Groffen
---
lib/portage/dbapi/vartree.py | 13 +++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..df192e6fc 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -35,6 +35,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util.install_mask:install_mask_dir,InstallMask',
'portage.util.listdir:dircache,listdir',
'portage.util.movefile:movefile',
+ 'portage.util.monotonic:monotonic',
'portage.util.path:first_existing,iter_parents',
'portage.util.writeable_check:get_ro_checker',
'portage.util._xattr:xattr',
@@ -3475,13 +3476,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+ tnow = monotonic()
+ tinterv = 2 # seconds
+ ninterv = tnow + tinterv
+ falign = len("%d" % totfiles)
showMessage(_(" %s checking %d files for package
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- if i % 1000 == 0 and i != 0:
- showMessage(_("%d files remaining
...\n") % (totfiles - i))
+ if monotonic() > ninterv:
+ showMessage(_("%3d%% done, %*d files
remaining ...\n") %
+ (falign, i * 100 /
totfiles, totfiles - i))
+ ninterv = monotonic() + tinterv
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
@@ -3570,6 +3577,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+ if tnow + tinterv < ninterv:
+ showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions,
plib_collisions
def _lstat_inode_map(self, path_iter):
--
2.20.1
Re: [gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval
On 08-01-2019 20:59:34 +, M. J. Everitt wrote:
> On 08/01/19 19:15, Zac Medico wrote:
> > On 1/8/19 5:42 AM, Fabian Groffen wrote:
> >> The reporting of files remaining can look somewhat odd since the report
> >> interval is hardcoded to be per 1000 objects. Adjust this interval to
> >> be time based. This means that modern (fast) machines likely will never
> >> see the countdown messages at all. On slow setups the message will be
> >> informative that there is progress, albeit rather slowly. While at it,
> >> report percentage done.
> >>
> >> Output before this patch:
> >>
> >> * checking 6158 files for package collisions
> >> 5158 files remaining ...
> >> 4158 files remaining ...
> >> 3158 files remaining ...
> >> 2158 files remaining ...
> >> 1158 files remaining ...
> >> 158 files remaining ...
> >>
> >> Possible output after this patch on a slower machine:
> >>
> >> * checking 6158 files for package collisions
> >> 48% done, 3145 files remaining ...
> >> 96% done, 192 files remaining ...
> >> 100% done
> >>
> >> Signed-off-by: Fabian Groffen
> >> ---
> >> lib/portage/dbapi/vartree.py | 11 +--
> >> 1 file changed, 9 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
> >> index 4b91caea8..244195fad 100644
> >> --- a/lib/portage/dbapi/vartree.py
> >> +++ b/lib/portage/dbapi/vartree.py
> >> @@ -3475,13 +3475,18 @@ class dblink(object):
> >>symlink_collisions = []
> >>destroot = self.settings['ROOT']
> >>totfiles = len(file_list) + len(symlink_list)
> >> + tnow = time.time()
> >> + tinterv = 2 # seconds
> >> + ninterv = tnow + tinterv
> >>showMessage(_(" %s checking %d files for package
> >> collisions\n") % \
> >>(colorize("GOOD", "*"), totfiles))
> >>for i, (f, f_type) in enumerate(chain(
> >>((f, "reg") for f in file_list),
> >>((f, "sym") for f in symlink_list))):
> >> - if i % 1000 == 0 and i != 0:
> >> - showMessage(_("%d files remaining
> >> ...\n") % (totfiles - i))
> >> + if time.time() > ninterv:
> >> + showMessage(_("%3d%% done, %d files
> >> remaining ...\n") %
> >> + (i * 100 / totfiles,
> >> totfiles - i))
> >> + ninterv = time.time() + tinterv
> >>
> >>dest_path = normalize_path(
> >>os.path.join(destroot,
> >> f.lstrip(os.path.sep)))
> >> @@ -3570,6 +3575,8 @@ class dblink(object):
> >>break
> >>if stopmerge:
> >>collisions.append(f)
> >> + if tnow + tinterv < ninterv:
> >> + showMessage(_("100% done\n"))
> >>return collisions, dirs_ro, symlink_collisions,
> >> plib_collisions
> >>
> >>def _lstat_inode_map(self, path_iter):
> >>
> > Please replace time.time() with portage.util.monotonic.monotonic().
Will do.
> It's a bit of a nit-pick, granted, but can we ensure that the count-down's
> remain padded/justified such that the numbers line up for easy at-a-glance
> inspection ? The optimal standard looks somewhat like the pre-merge Sizes:
>
> * Final size of build directory: 2696 KiB (2.6 MiB)
> * Final size of installed tree: 5372 KiB (5.2 MiB)
>
> Otherwise, I think this will be quite helpful. Thanks.
I think I understand what you mean, let me see if I can do something
without too much complexity.
Thanks,
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH v3] collision_protect: use dynamic report interval
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects. Adjust this interval to
be time based. This means that modern (fast) machines likely will never
see the countdown messages at all. On slow setups the message will be
informative that there is progress, albeit rather slowly. While at it,
report percentage done.
Output before this patch:
* checking 6158 files for package collisions
5158 files remaining ...
4158 files remaining ...
3158 files remaining ...
2158 files remaining ...
1158 files remaining ...
158 files remaining ...
Possible output after this patch on a slower machine:
* checking 6158 files for package collisions
48% done, 3145 files remaining ...
96% done, 192 files remaining ...
100% done
Signed-off-by: Fabian Groffen
---
lib/portage/dbapi/vartree.py | 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..244195fad 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,18 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+ tnow = time.time()
+ tinterv = 2 # seconds
+ ninterv = tnow + tinterv
showMessage(_(" %s checking %d files for package
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- if i % 1000 == 0 and i != 0:
- showMessage(_("%d files remaining
...\n") % (totfiles - i))
+ if time.time() > ninterv:
+ showMessage(_("%3d%% done, %d files
remaining ...\n") %
+ (i * 100 / totfiles,
totfiles - i))
+ ninterv = time.time() + tinterv
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
@@ -3570,6 +3575,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+ if tnow + tinterv < ninterv:
+ showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions,
plib_collisions
def _lstat_inode_map(self, path_iter):
--
2.20.1
Re: [gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval
On 08-01-2019 09:17:04 +0100, Ulrich Mueller wrote: > >>>>> On Tue, 08 Jan 2019, Fabian Groffen wrote: > > > Output before this patch: > > > * checking 6111 files for package collisions > > [...] > > > After: > > > * checking 6158 files for package collisions > > [...] > > Why does the total number of files change? First was python2, second python3. The result was indicative only, the number doesn't change of course. Sorry for the confusion. Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] collision_protect: use dynamic report interval
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects. Adjust this interval to
be regular towards the end. While at it, report percentage done.
Output before this patch:
* checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
files remaining ...
111 files remaining ...
After:
* checking 6158 files for package collisions
16% done, 5131 files remaining ...
33% done, 4104 files remaining ...
50% done, 3077 files remaining ...
66% done, 2050 files remaining ...
83% done, 1023 files remaining ...
100% done
Signed-off-by: Fabian Groffen
---
lib/portage/dbapi/vartree.py | 14 +++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..78f2b37f2 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+ bucksize = 1000
+ buckcnt = int(totfiles / bucksize)
+ if buckcnt == 0 or totfiles % bucksize > int(bucksize /
2):
+ buckcnt = buckcnt + 1
+ bucksize = int(totfiles / buckcnt) + 1
showMessage(_(" %s checking %d files for package
collisions\n") % \
- (colorize("GOOD", "*"), totfiles))
+ (colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- if i % 1000 == 0 and i != 0:
- showMessage(_("%d files remaining
...\n") % (totfiles - i))
+ if i % bucksize == 0 and i != 0:
+ showMessage(_("%3d%% done, %d files
remaining ...\n") %
+ (i * 100 / totfiles,
totfiles - i))
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
@@ -3570,6 +3576,8 @@ class dblink(object):
break
if stopmerge:
collisions.append(f)
+ if bucksize < totfiles:
+ showMessage(_("100% done\n"))
return collisions, dirs_ro, symlink_collisions,
plib_collisions
def _lstat_inode_map(self, path_iter):
--
2.20.1
[gentoo-portage-dev] [PATCH] lib/portage/dbapi/vartree: use dynamic report interval in _collision_protect
The reporting of files remaining can look somewhat odd since the report
interval is hardcoded to be per 1000 objects. Adjust this interval to
be regular towards the end. While at it, report percentage done.
Output before this patch:
* checking 6111 files for package collisions
5111 files remaining ...
4111 files remaining ...
3111 files remaining ...
2111 files remaining ...
files remaining ...
111 files remaining ...
After:
* checking 6158 files for package collisions
16% done, 5131 files remaining ...
33% done, 4104 files remaining ...
50% done, 3077 files remaining ...
66% done, 2050 files remaining ...
83% done, 1023 files remaining ...
Signed-off-by: Fabian Groffen
---
lib/portage/dbapi/vartree.py | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 4b91caea8..4d0bf2789 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -3475,13 +3475,19 @@ class dblink(object):
symlink_collisions = []
destroot = self.settings['ROOT']
totfiles = len(file_list) + len(symlink_list)
+ bucksize = 1000
+ buckcnt = int(totfiles / bucksize)
+ if buckcnt == 0 or totfiles % bucksize > int(bucksize /
2):
+ buckcnt = buckcnt + 1
+ bucksize = int(totfiles / buckcnt) + 1
showMessage(_(" %s checking %d files for package
collisions\n") % \
(colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
- if i % 1000 == 0 and i != 0:
- showMessage(_("%d files remaining
...\n") % (totfiles - i))
+ if i % bucksize == 0 and i != 0:
+ showMessage(_("%2d%% done, %d files
remaining ...\n") %
+ (i * 100 / totfiles,
totfiles - i))
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
--
2.20.1
Re: [RFC] gpkg format proposal v2 (was: Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format)
On 11-11-2018 21:53:33 +0100, Michał Górny wrote:
> Hi,
>
> Ok, here's the second version integrating the feedback received.
> The format is much simpler, based on nested tarballs inspired by Debian.
>
> The outer tarball is uncompressed and uses '.gpkg.tar' suffix. It
> contains (preferably in order but PM should also handle packages with
> mismatched order):
>
> 1. Optional (but recommended) "gpkg: ${PF}" package label that can be
> used to quickly distinguish Gentoo binpkgs from regular tarballs
> (for file(1)).
>
> 2. "metadata.tar${comp}" tarball containing binary package metadata
> as files.
>
> 3. Optional "metadata.tar${comp}.sig" containing detached signature
> for the metadata archive.
>
> 4. "contents.tar${comp}" tarball containing files to be installed.
>
> 5. Optional "contents.tar${comp}.sig" containing detached signature for
> the contents archive.
>
> Notes:
>
> a. ${comp} can be any compression format supported by binary packages.
> Technically, metadata and content archives may use different
> compression. Either or both may be uncompressed as well.
I'm wondering here, how much sense does it make to compress 2., 3.
and/or 4. if you compress the whole gpkg? I have the impression
compression on compression isn't beneficial here. Shouldn't just
compressing of the gpkg tar be sufficient?
As to allowing different compressors for a single gpkg, I think it would
be better to require all compressors to be the same, such that a PM or
tool can quickly see if it can "read" the file from the gpkg filename,
instead of having to fetch and open it first. Obviously, if you drop
compression of the inner tars, this point goes away.
Thanks,
Fabian
> b. While signatures are optional, the PM should have a switch
> controlling whether to expect them, and fail hard if they're not present
> when expected.
>
>
> Advantages
> --
> Guaranteed:
>
> + The binary package is still one file, so can be fetched easily.
>
> + File format is trivial and can be extracted using tar(1) + compressor.
>
> + The metadata and contents are compressed independently, and so can be
> easily extracted or modified independently.
>
> + The package format provides for separate metadata and content
> signatures, so they can be verified independently.
>
> + Metadata can be compressed now.
>
> Achieved by regular archives (but might be broken if modified by user):
>
> + Easy recognition by magic(1).
>
> + The metadata archive (and its signature) is packed first, so it may be
> read without fetching the whole binpkg.
>
>
> Why not .ar format?
> ---
> The use of .ar format has been proposed, akin to Debian. While
> the option is mostly feasible, and the simplicity of .ar format would
> reduce the outer size of binary packages, I think the format is simply
> too obscure. It lives mostly as static library format, and the tooling
> for it is part of binutils. LSB considers it deprecated. While I don't
> see it going away anytime soon, I'd rather not rely on it in order to
> save a few KiB.
>
>
> Is there anything left to address?
>
> --
> Best regards,
> Michał Górny
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH v2] rsync: quarantine data prior to verification (bug 660410)
+ self._remove_download(download_dir)
> if openpgp_env is not None:
> openpgp_env.close()
>
> -
> def _process_exitcode(self, exitcode, syncuri, out, maxretries):
> if (exitcode==0):
> pass
> @@ -561,7 +627,7 @@ class RsyncSync(NewBase):
> return rsync_opts
>
>
> - def _do_rsync(self, syncuri, timestamp, opts):
> + def _do_rsync(self, syncuri, timestamp, opts, download_dir):
> updatecache_flg = False
> is_synced = False
> if timestamp != 0 and "--quiet" not in opts:
> @@ -686,6 +752,12 @@ class RsyncSync(NewBase):
> elif (servertimestamp == 0) or (servertimestamp >
> timestamp):
> # actual sync
> command = rsynccommand[:]
> +
> + if self.repo.location != download_dir:
> + # Use shared hardlinks for files that
> are identical
> + # in the previous snapshot of the
> repository.
> + command.append('--link-dest=%s' %
> self.repo.location)
> +
> submodule_paths = self._get_submodule_paths()
> if submodule_paths:
> # The only way to select multiple
> directories to
> @@ -696,9 +768,10 @@ class RsyncSync(NewBase):
> # /./ is special syntax
> supported with the
> # rsync --relative option.
> command.append(syncuri + "/./"
> + path)
> - command.append(self.repo.location)
> else:
> - command.extend([syncuri + "/",
> self.repo.location])
> + command.append(syncuri + "/")
> +
> + command.append(download_dir)
>
> exitcode = None
> try:
> --
> 2.13.6
>
>
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)
On 01-04-2018 20:29:59 +0200, Michał Górny wrote: > > > This essentially looks like ~700 lines of code to try to workaround > > > broken networking. I would rather try to do that using 5 lines of code > > > but that's just me, and my programs aren't enterprise quality. I just > > > hope it actually solves as many problems as it's going to introduce. > > > > The vast majority of this code is generic and reusable, and I do intend > > to reuse it. For example, the executor support will be an essential > > piece for the asyncio.AbstractEventLoop for bug 649588. > > Sure it is and sure you will. But tell me: who is going to maintain it > all? Because as far as I can see, we're still dealing with a bus factor > of one and all you're doing is making it worse. More code, more > complexity, more creeping featurism and hacks. Well, well, well. This could be said about a lot of code, including your own. > Last time you went away, you left us with a horribly unmaintainable > package manager full of complexity, hacks and creeping featurism > and a Portage team whose members had barely any knowledge of the code. > Just when things started moving again, you came back and we're back to > square one. I don't see why this has to be made personal. In the olde days people just pushed whatever they needed, now it's reviewed and acked, so how can it be back to square one? > Today Portage once again is a one-developer project, full of more > complexity, more hacks and more creeping featurism. And we once again > have a bus factor of one -- one developer who apparently knows > everything, does everything and tries to be nice to everyone, except he > really ignores others, makes a lot of empty promises and consistently > makes the health of the project go from bad to worse. Errr, didn't you recently start your own fork with creeping featurism of its own because mainline was/is too slow? > So, please tell me: what happens when you leave again? How have you used > your time in the project? What have you done to make sure that > the project stays alive without you? Because as far as I can see, adding > few thousand of lines of practically unreviewed code every month does > not help with that. Why is this Zac's problem specifically? Isn't this a general problem of Gentoo? And isn't your attitude contributing in a large factor to the bus-factor getting down from 1 to 0? > I forked Portage because I didn't want to fight with you. When I forked > it, I declared that I will merge mainline changes regularly for > the benefit of my users. But after a week, I really start feeling like > that's going to be a really bad idea. Like it's time to forget about > mainline Portage as a completely dead end, and go our separate ways. So you fork. Like many forks, that is because people feel that it isn't going "their way". Yay. That doesn't make you "right". > I'm seriously worried about the future of Gentoo. I'd really appreciate > if you started focusing on that as well. I get that all this stuff looks > cool on paper but few months or years from now, someone will curse > 'whoever wrote that code' while trying to fix some nasty bug. Or get > things moving forward. Or implement EAPI 8. Perhaps it's time to look into the mirror. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH v2] __dyn_install: improve reporting of build and image sizes
On 27-08-2017 12:53:23 -0700, Zac Medico wrote: > On 08/27/2017 08:06 AM, Fabian Groffen wrote: > > Prior to this commit, the reported sizes would look like: > > > > * Final size of build directory: 34942 KiB > > * Final size of installed tree: 5627 KiB > > > > Because the sizes aren't aligned, it is hard to (visually) compare them. > > On top of this, because the numbers are sometimes bigger, print a human > > friendly size after the KiB size if applicable, like so: > > > > * Final size of build directory: 1906 KiB (1.8 MiB) > > * Final size of installed tree: 7 KiB > > > > It should be noted that in case both sizes have a human-readable > > variant, they are also aligned. > > > > The helper functions are defined and used in a subshell to avoid > > pollution of the caller's environment. > > --- > > bin/phase-functions.sh | 53 > > ++++++---- > > 1 file changed, 49 insertions(+), 4 deletions(-) > > Looks good. Please merge. Pushed, thanks! Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH v2] __dyn_install: improve reporting of build and image sizes
Prior to this commit, the reported sizes would look like:
* Final size of build directory: 34942 KiB
* Final size of installed tree: 5627 KiB
Because the sizes aren't aligned, it is hard to (visually) compare them.
On top of this, because the numbers are sometimes bigger, print a human
friendly size after the KiB size if applicable, like so:
* Final size of build directory: 1906 KiB (1.8 MiB)
* Final size of installed tree: 7 KiB
It should be noted that in case both sizes have a human-readable
variant, they are also aligned.
The helper functions are defined and used in a subshell to avoid
pollution of the caller's environment.
---
bin/phase-functions.sh | 53 ++
1 file changed, 49 insertions(+), 4 deletions(-)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index dfd8733c8..ce174ba91 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -598,10 +598,55 @@ __dyn_install() {
# record build & installed size in build log
if type -P du &>/dev/null; then
- local sz=( $(du -ks "${WORKDIR}") )
- einfo "Final size of build directory: ${sz[0]} KiB"
- sz=( $(du -ks "${D}") )
- einfo "Final size of installed tree: ${sz[0]} KiB"
+ local nsz=( $(du -ks "${WORKDIR}") )
+ local isz=( $(du -ks "${D}") )
+
+ # subshell to avoid polluting the caller env with the helper
+ # functions below
+ (
+ # align $1 to the right to the width of the widest of
$1 and $2
+ padl() {
+ local s1=$1
+ local s2=$2
+ local width=${#s1}
+ [[ ${#s2} -gt ${width} ]] && width=${#s2}
+ printf "%*s" ${width} "${s1}"
+ }
+
+ # transform number in KiB into MiB, GiB or TiB based on
size
+ human() {
+ local s1=$1
+ local units=( KiB MiB GiB TiB )
+
+ s1=$((s1 * 10))
+ while [[ ${s1} -gt 10240 && ${#units[@]} -gt 1
]] ; do
+ s1=$((s1 / 1024 ))
+ units=( ${units[@]:1} )
+ done
+
+ local r=${s1: -1}
+ s1=$((s1 / 10))
+ printf "%s.%s %s" "${s1}" "${r}" "${units[0]}"
+ }
+
+ size() {
+ local s1=$1
+ local s2=$2
+ local out="$(padl "${s1}" "${s2}") KiB"
+
+ if [[ ${s1} -gt 1024 ]] ; then
+ s1=$(human ${s1})
+ if [[ ${s2} -gt 1024 ]] ; then
+ s2=$(human ${s2})
+ s1=$(padl ${s1} ${s2})
+ fi
+ out+=" (${s1})"
+ fi
+ echo "${out}"
+ }
+ einfo "Final size of build directory: $(size ${nsz[0]}
${isz[0]})"
+ einfo "Final size of installed tree: $(size ${isz[0]}
${nsz[0]})"
+ )
__vecho
fi
--
2.14.1
Re: [gentoo-portage-dev] [PATCH] _collision_protect: report progress in work todo
On 26-08-2017 14:46:30 -0700, Zac Medico wrote:
> On 08/24/2017 06:28 AM, Fabian Groffen wrote:
> > Currently Portage reports its progress in checking collisions forward
> > every 1000th file like so:
> >
> > * checking 4149 files for package collisions
> > 1000 files checked ...
> > 2000 files checked ...
> > 3000 files checked ...
> > 4000 files checked ...
> >>>> Merging sys-apps/portage-2.3.8 to /
> >
> > Change it to countdown style so it is easier to anticipate what the
> > next action will be:
> >
> > * checking 4149 files for package collisions
> > 3149 files remaining ...
> > 2149 files remaining ...
> > 1149 files remaining ...
> > 149 files remaining ...
> >>>> Merging sys-apps/portage-2.3.8 to /
> > ---
> > pym/portage/dbapi/vartree.py | 5 +++--
> > 1 file changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> > index 7c8f150bb..04a40b732 100644
> > --- a/pym/portage/dbapi/vartree.py
> > +++ b/pym/portage/dbapi/vartree.py
> > @@ -3420,13 +3420,14 @@ class dblink(object):
> > dirs_ro = set()
> > symlink_collisions = []
> > destroot = self.settings['ROOT']
> > + totfiles = len(file_list) + len(symlink_list)
> > showMessage(_(" %s checking %d files for package
> > collisions\n") % \
> > - (colorize("GOOD", "*"), len(file_list) +
> > len(symlink_list)))
> > + (colorize("GOOD", "*"), totfiles))
> > for i, (f, f_type) in enumerate(chain(
> > ((f, "reg") for f in file_list),
> > ((f, "sym") for f in symlink_list))):
> > if i % 1000 == 0 and i != 0:
> > - showMessage(_("%d files checked ...\n")
> > % i)
> > + showMessage(_("%d files remaining
> > ...\n") % (totfiles - i))
> >
> > dest_path = normalize_path(
> > os.path.join(destroot,
> > f.lstrip(os.path.sep)))
> >
>
> Looks good. Please merge.
Pushed, thanks
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: Digital signature
[gentoo-portage-dev] [PATCH] _collision_protect: report progress in work todo
Currently Portage reports its progress in checking collisions forward
every 1000th file like so:
* checking 4149 files for package collisions
1000 files checked ...
2000 files checked ...
3000 files checked ...
4000 files checked ...
>>> Merging sys-apps/portage-2.3.8 to /
Change it to countdown style so it is easier to anticipate what the
next action will be:
* checking 4149 files for package collisions
3149 files remaining ...
2149 files remaining ...
1149 files remaining ...
149 files remaining ...
>>> Merging sys-apps/portage-2.3.8 to /
---
pym/portage/dbapi/vartree.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
index 7c8f150bb..04a40b732 100644
--- a/pym/portage/dbapi/vartree.py
+++ b/pym/portage/dbapi/vartree.py
@@ -3420,13 +3420,14 @@ class dblink(object):
dirs_ro = set()
symlink_collisions = []
destroot = self.settings['ROOT']
+ totfiles = len(file_list) + len(symlink_list)
showMessage(_(" %s checking %d files for package
collisions\n") % \
- (colorize("GOOD", "*"), len(file_list) +
len(symlink_list)))
+ (colorize("GOOD", "*"), totfiles))
for i, (f, f_type) in enumerate(chain(
((f, "reg") for f in file_list),
((f, "sym") for f in symlink_list))):
if i % 1000 == 0 and i != 0:
- showMessage(_("%d files checked ...\n")
% i)
+ showMessage(_("%d files remaining
...\n") % (totfiles - i))
dest_path = normalize_path(
os.path.join(destroot,
f.lstrip(os.path.sep)))
--
2.14.1
[gentoo-portage-dev] [PATCH] __dyn_install: improve reporting of build and image sizes
Prior to this commit, the reported sizes would look like:
* Final size of build directory: 34942 KiB
* Final size of installed tree: 5627 KiB
Because the sizes aren't aligned, it is hard to (visually) compare them.
On top of this, because the numbers are sometimes bigger, print a human
friendly size after the KiB size if applicable, like so:
* Final size of build directory: 1906 KiB (1.8 MiB)
* Final size of installed tree: 7 KiB
It should be noted that in case both sizes have a human-readable
variant, they are also aligned.
---
bin/phase-functions.sh | 49 +
1 file changed, 45 insertions(+), 4 deletions(-)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index dfd8733c8..af45a0d49 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -598,10 +598,51 @@ __dyn_install() {
# record build & installed size in build log
if type -P du &>/dev/null; then
- local sz=( $(du -ks "${WORKDIR}") )
- einfo "Final size of build directory: ${sz[0]} KiB"
- sz=( $(du -ks "${D}") )
- einfo "Final size of installed tree: ${sz[0]} KiB"
+ local nsz=( $(du -ks "${WORKDIR}") )
+ local isz=( $(du -ks "${D}") )
+
+ # align $1 to the right to the width of the widest of $1 and $2
+ padl() {
+ local s1=$1
+ local s2=$2
+ local width=${#s1}
+ [[ ${#s2} -gt ${width} ]] && width=${#s2}
+ printf "%*s" ${width} "${s1}"
+ }
+
+ # transform number in KiB into MiB, GiB or TiB based on size
+ human() {
+ local s1=$1
+ local units=( KiB MiB GiB TiB )
+
+ s1=$((s1 * 10))
+ while [[ ${s1} -gt 10240 && ${#units[@]} -gt 1 ]] ; do
+ s1=$((s1 / 1024 ))
+ units=( ${units[@]:1} )
+ done
+
+ local r=${s1: -1}
+ s1=$((s1 / 10))
+ printf "%s.%s %s" "${s1}" "${r}" "${units[0]}"
+ }
+
+ size() {
+ local s1=$1
+ local s2=$2
+ local out="$(padl "${s1}" "${s2}") KiB"
+
+ if [[ ${s1} -gt 1024 ]] ; then
+ s1=$(human ${s1})
+ if [[ ${s2} -gt 1024 ]] ; then
+ s2=$(human ${s2})
+ s1=$(padl ${s1} ${s2})
+ fi
+ out+=" (${s1})"
+ fi
+ echo "${out}"
+ }
+ einfo "Final size of build directory: $(size ${nsz[0]}
${isz[0]})"
+ einfo "Final size of installed tree: $(size ${isz[0]}
${nsz[0]})"
__vecho
fi
--
2.14.1
[gentoo-portage-dev] [PATCH] etc-update: fix hang when using_editor is set, bug #544440
Don't try to use an interactive editor to obtain the difference between
files. Fall back to plain `diff -q` in this case.
---
bin/etc-update | 19 +++
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/bin/etc-update b/bin/etc-update
index e0f7224dc..ea69f1478 100755
--- a/bin/etc-update
+++ b/bin/etc-update
@@ -649,10 +649,21 @@ do_distconf() {
elif [[ -L ${efile} || -L ${file} ]] ; then
# not the same file types
continue
- elif diff_command "${file}" "${efile}" &> /dev/null; then
- # replace identical copy
- mv "${file}" "${efile}"
- break
+ else
+ local ret=
+ if [[ ${using_editor} == 0 ]] ; then
+ diff_command "${file}" "${efile}" &> /dev/null
+ ret=$?
+ else
+ # fall back to plain diff
+ diff -q "${file}" "${efile}" &> /dev/null
+ ret=$?
+ fi
+ if [[ ${ret} == 0 ]] ; then
+ # replace identical copy
+ mv "${file}" "${efile}"
+ break
+ fi
fi
done
}
--
2.11.1
[gentoo-portage-dev] [PATCH] getgccversion: check for clang if gcc is not found
In Gentoo Prefix on Mac OS X systems, LLVM/clang is used as the
toolchain, so no gcc/gcc-config around. Probe for clang after we failed
to locate gcc, so we can show clang version instead.
---
pym/_emerge/actions.py | 39 ++-
1 file changed, 38 insertions(+), 1 deletion(-)
diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
index 59626ad..1d324aa 100644
--- a/pym/_emerge/actions.py
+++ b/pym/_emerge/actions.py
@@ -39,6 +39,7 @@ from portage import os
from portage import shutil
from portage import eapi_is_supported, _encodings, _unicode_decode
from portage.cache.cache_errors import CacheError
+from portage.const import EPREFIX
from portage.const import GLOBAL_CONFIG_PATH, VCS_DIRS,
_DEPCLEAN_LIB_CHECK_DEFAULT
from portage.const import SUPPORTED_BINPKG_FORMATS, TIMESTAMP_FORMAT
from portage.dbapi.dep_expand import dep_expand
@@ -2406,12 +2407,23 @@ def getgccversion(chost=None):
gcc_ver_command = ['gcc', '-dumpversion']
gcc_ver_prefix = 'gcc-'
+ clang_ver_command = ['clang', '--version']
+ clang_ver_prefix = 'clang-'
+
+ ubinpath = os.path.join('/', portage.const.EPREFIX, 'usr', 'bin')
+
gcc_not_found_error = red(
"!!! No gcc found. You probably need to 'source /etc/profile'\n" +
"!!! to update the environment of this terminal and possibly\n" +
"!!! other terminals also.\n"
)
+ def getclangversion(output):
+ version = re.search('clang version ([0-9.]+) ', output)
+ if version:
+ return version.group(1)
+ return "unknown"
+
if chost:
try:
proc = subprocess.Popen(["gcc-config", "-c"],
@@ -2439,7 +2451,7 @@ def getgccversion(chost=None):
return gcc_ver_prefix + myoutput
try:
- proc = subprocess.Popen(gcc_ver_command,
+ proc = subprocess.Popen([ubinpath + "/" + gcc_ver_command[0]] +
gcc_ver_command[1:],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
except OSError:
myoutput = None
@@ -2450,6 +2462,31 @@ def getgccversion(chost=None):
if mystatus == os.EX_OK:
return gcc_ver_prefix + myoutput
+ try:
+ proc = subprocess.Popen(
+ [ubinpath + "/" + chost + "-" + clang_ver_command[0]] +
clang_ver_command[1:],
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ except OSError:
+ myoutput = None
+ mystatus = 1
+ else:
+ myoutput = _unicode_decode(proc.communicate()[0]).rstrip("\n")
+ mystatus = proc.wait()
+ if mystatus == os.EX_OK:
+ return clang_ver_prefix + getclangversion(myoutput)
+
+ try:
+ proc = subprocess.Popen([ubinpath + "/" + clang_ver_command[0]]
+ clang_ver_command[1:],
+ stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+ except OSError:
+ myoutput = None
+ mystatus = 1
+ else:
+ myoutput = _unicode_decode(proc.communicate()[0]).rstrip("\n")
+ mystatus = proc.wait()
+ if mystatus == os.EX_OK:
+ return clang_ver_prefix + getclangversion(myoutput)
+
portage.writemsg(gcc_not_found_error, noiselevel=-1)
return "[unavailable]"
--
2.7.0
[gentoo-portage-dev] [PATCH] consider clang when looking up compiler version
This is a patch from prefix branch to make the compiler detection consider clang. Fabian Groffen (1): getgccversion: check for clang if gcc is not found pym/_emerge/actions.py | 39 ++- 1 file changed, 38 insertions(+), 1 deletion(-) -- 2.7.0
Re: [gentoo-portage-dev] [PATCH 1/2] bin/phase-helpers.shL Fix unpack() for deb2targz .xz types
On 07-12-2014 10:49:30 -0800, Zac Medico wrote: > On 12/07/2014 10:47 AM, Ulrich Mueller wrote: > >>>>>> On Sun, 07 Dec 2014, Zac Medico wrote: > > > >> On 12/07/2014 10:37 AM, Ulrich Mueller wrote: > >>> It's sort of trivial, but here is a patch: > >>> > >>> From c53e7057f94728d6e0c7d16c675702ca831b9a5a Mon Sep 17 00:00:00 2001 > >>> From: Ulrich Müller > >>> Date: Sun, 7 Dec 2014 19:33:52 +0100 > >>> Subject: [PATCH] Prefer GNU ar when unpacking .deb packages. > >>> > >>> [...] > > > >> LGTM. > > > > So shall I push it? > > Yes, please do. FYI: % portageq envvar CHOST x86_64-apple-darwin13 % ar --version ar: illegal option -- - usage: ar -d [-TLsv] archive file ... ar -m [-TLsv] archive file ... ar -m [-abiTLsv] position archive file ... ar -p [-TLsv] archive [file ...] ar -q [-cTLsv] archive file ... ar -r [-cuTLsv] archive file ... ar -r [-abciuTLsv] position archive file ... ar -t [-TLsv] archive [file ...] ar -x [-ouTLsv] archive [file ...] % ar x $EPREFIX/usr/portage/distfiles/realpath_1.16_i386.deb % ls control.tar.gz data.tar.gz debian-binary deb2targz should work (but I don't have it installed, so not sure if tested). Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] New Portage release
Hey all, I'm sorry if this has been discussed previously. I admit not having read all of the mails on this list recently. Long story short: what are the plans on releasing 2.2.9? Longer version: for Prefix I likely need a new release soon to get some fixes out of the door. Since I like to stay on par with mainline portage's versioning here, I've been going along with mainline portage's releases for a while, which worked fine. Now we haven't got a release for a long while, what's the plans here? Is there anyone at all who can/dares to spin a release? Apologies in advance if this has been discussed recently. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH 3/3] Add CPU model name to output of getportageversion as fifth element
On 14-08-2013 22:10:40 +0200, Mark Kubacki wrote: > From: W-Mark Kubacki > > It will read like this: > > Portage 2.1.13.7 (default/linux/amd64/13.0, gcc-4.6.2, glibc-2.18, > > 3.9.0-rc8-mark-signed+ x86_64, Intel(R) Core(TM) i7-3770T CPU @ 2.50GHz) > > That new fifth element will be the CPU model name of the host > running Portage. It is not the target architecture! FYI: % python3 Python 3.3.2 (default, Jul 24 2013, 11:14:02) [GCC 4.2.1 (Gentoo 4.2.1_p5666-r1, Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import platform >>> platform.release() '12.4.0' >>> platform.machine() 'x86_64' >>> platform.processor() 'i386' >>> % python Python 3.2.3 (default, May 6 2013, 21:19:05) [GCC 4.7.2] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import platform >>> platform.release() '5.11' >>> platform.machine() 'i86pc' >>> platform.processor() 'i386' >>> % python Python 3.3.2 (default, Jul 15 2013, 13:51:24) [GCC 4.7.2] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import platform >>> platform.release() '5.10' >>> platform.machine() 'sun4u' >>> platform.processor() 'sparc' >>> % python Python 3.2.5 (default, Jul 15 2013, 11:37:08) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import platform >>> platform.release() '3.8.13-gentoo' >>> platform.machine() 'x86_64' >>> platform.processor() 'AMD Athlon(tm) 64 X2 Dual Core Processor 3800+' >>> e.g. it seems to me only on Linux it gives fancy model output. Note that the first and second system were running a 64-bit Python as well as kernel. Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH 0/3] Add cross-prefix support
Hi Ruud, On 17-06-2013 09:21:41 +0200, Ruud Koolen wrote: > This patch series adds support for using a portage installed in one prefix > to build packages with a different prefix. Thanks so much for your work. I hope to have a look at this soon, and merge this. From your work I assume your patches don't "break" normal installs/bootstraps, right? (You tested that?) Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] Re: proj/portage:master commit in: man/, pym/portage/dbapi/, pym/portage/, cnf/
On 25-10-2012 02:45:41 -0700, Zac Medico wrote: > > can't you use os.fdatasync() or os.fsync() instead? > > os.fdatasync() or os.fsync() only work for single files, right? We don't > want to sync them individually, since that wastes disk bandwidth. It's > much more efficient to sync the whole filesystem at once, like syncfs does. You use syncfs, which according to the man-page requires a filedescriptor. Hence my suggestion. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] Re: proj/portage:master commit in: man/, pym/portage/dbapi/, pym/portage/, cnf/
On 25-10-2012 08:45:36 +, Zac Medico wrote: > URL: > http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=10b6d012 > > Add FEATURES=merge-sync, for bug #439584. > + if not self._device_path_map or \ > + "merge-sync" not in self.settings.features: > + return > + > + syncfs = self._get_syncfs() > + if syncfs is None: > + try: > + proc = subprocess.Popen(["sync"]) > + except EnvironmentError: > + pass > + else: > + proc.wait() > + else: > + for path in self._device_path_map.values(): > + try: > + fd = os.open(path, os.O_RDONLY) > + except OSError: > + pass > + else: > + try: > + syncfs(fd) can't you use os.fdatasync() or os.fsync() instead? -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] About adding a repoman check to test for missing files in FILESDIR
On 08-10-2012 20:47:41 +0200, Pacho Ramos wrote:
> El lun, 08-10-2012 a las 11:32 -0700, Zac Medico escribió:
> > Given the expressiveness of the bash language, it's really a non-trivial
> > thing to check without actually executing bash. See discussion here:
> >
> > https://bugs.gentoo.org/show_bug.cgi?id=431196
>
> Then, maybe I could have a custom repoman script that would check for
> them in my "files/" directory (either using bash to check for them or
> simply running "ls") :/
Yes, but still it is very hard to make it 100% reliable. If you stick
to simple "${FILESDIR}"/${PN}-x.y.patch patterns, it's very well
feasible, though. See for inspiration the fetch_files() function from [1].
[1]
http://overlays.gentoo.org/proj/alt/browser/trunk/prefix-overlay/scripts/eupdate.in
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: Digital signature
[gentoo-portage-dev] Re: proj/portage:master commit in: pym/portage/dbapi/
On 26-11-2011 16:56:41 +0530, Nirbheek Chauhan wrote: > On Sat, Nov 26, 2011 at 4:28 PM, Fabian Groffen wrote: > > On 26-11-2011 01:54:35 +, Arfrever Frehtes Taifersar Arahesis wrote: > >> commit: 1d4ac47c28706094230cb2c4e6ee1c1c71629aa0 > >> T> Org> > >> AuthorDate: Sat Nov 26 01:52:49 2011 + > >> Commit: Arfrever Frehtes Taifersar Arahesis gentoo > >> org> > >> CommitDate: Sat Nov 26 01:52:49 2011 + > >> URL: > >> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=1d4ac47c > >> > >> dblink.mergeme(): Merge files in alphabetic order. > > > > What's the advantage of this? I don't really like to pay for sorting a > > potentially huge list just for some eye-candy. (That's omitted by > > default these days anyway...) > > Any other opinions on this one? > > > > If it should be sorted[1], it should really be sorted in the reverse > order of distfile-download size. That would be extremely useful on > systems with slow internet connections. Too many times have I sat > waiting for libreoffice-bin to download while a webkit-gtk recompile > waits in the queue. > > We already have the information during dependency resolution with > --verbose, and it costs very little. Besides, sorting even 30,000 > entries (if you're merging every ebuild in portage) should not take > more than a few secs. A linux kernel has around that much of files, and I really wonder if it's worth waiting a couple of seconds (probably more on sparc and arm systems) just because then the files are in sorted order. > 1. I'm obviously assuming that dep nodes that do not depend on each > other would be sorted I think this is per package. I didn't watch closely enough the reply-to headers, the gentoo-portage-dev list was my original target, which obviously makes more sense for this context. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] Re: proj/portage:master commit in: pym/repoman/, pym/portage/tests/repoman/
On 21-10-2011 00:30:05 -0700, Zac Medico wrote: > On 10/21/2011 12:09 AM, Fabian Groffen wrote: > > On 21-10-2011 06:09:57 +, Zac Medico wrote: > >> UpdateChangeLog: split out/test copyright regex > >> > >> This also fixes a case where something like "Copyright 2011 " would be > >> replaced with "Copyright 2011-2011 ". > > > > I like this, but it should only happen when one would live in 1999 > > (that's why I added it unconditionally), see below. I'm affraid we need > > the 1999 to be configured, for legal reasons (though I don't claim to be > > an expert on this case, just judging from the comments from echangelog). > > I don't see anything like that in echangelog. What would be the point of > re-writing copyright start dates, when those should already be correctly > inherited from skel.ChangeLog anyway? Sure, but this is mainly intended for ebuilds and files (like init scripts?), I guess. I considered using skel.ebuild for .ebuild files, perhaps that's just the best to avoid hardcoding any years. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] Re: proj/portage:master commit in: pym/repoman/, pym/portage/tests/repoman/
On 21-10-2011 06:09:57 +, Zac Medico wrote: > UpdateChangeLog: split out/test copyright regex > > This also fixes a case where something like "Copyright 2011 " would be > replaced with "Copyright 2011-2011 ". I like this, but it should only happen when one would live in 1999 (that's why I added it unconditionally), see below. I'm affraid we need the 1999 to be configured, for legal reasons (though I don't claim to be an expert on this case, just judging from the comments from echangelog). > + def testCopyrightUpdate(self): > + test_cases = ( > + ( > + '2011', > + '# Copyright 1999-2008 Gentoo Foundation; > Distributed under the GPL v2', > + '# Copyright 1999-2011 Gentoo Foundation; > Distributed under the GPL v2', we should add a case like # Copyright 2010-2011 Gentoo Foundation; Distributed ... echangelog would have rewritten it to 1999-2011, because of legal reasons IIRC. > +def _update_copyright_year(year, line): > + """ > + These two regexes are taken from echangelog > + update_copyright(), except that we don't hardcode > + 1999 here (in order to be more generic). This comment makes little to no more sense, since the patterns taken from echangelog have changed considerably now :) -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH] repoman: update copyright on modified files
To retain the behaviour of echangelog, update the copyrights on modified
files (mostly ebuilds) when necessary. Also update the ChangeLog's
copyright.
diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
--- a/pym/repoman/utilities.py
+++ b/pym/repoman/utilities.py
@@ -523,9 +523,77 @@
return outvcs
+def update_copyright(fn_path, year, pretend):
+ """
+ Check file for a Copyright statement, and update its year. The
+ patterns used for replacing copyrights are taken from echangelog.
+ Only the first lines of each file that start with a hash ('#') are
+ considered, until a line is found that doesn't start with a hash.
+ """
+
+ try:
+ fn_hdl = io.open(_unicode_encode(fn_path,
+ encoding=_encodings['fs'], errors='strict'),
+ mode='r', encoding=_encodings['repo.content'],
errors='replace')
+ except EnvironmentError:
+ return
+
+ orig_header = []
+ new_header = []
+
+ for line in fn_hdl:
+ line_strip = line.strip()
+ orig_header.append(line)
+ if not line_strip or line_strip[:1] != '#':
+ new_header.append(line)
+ break
+
+ # these two regexes are taken from
+ # echangelog update_copyright()
+ line = re.sub(r'^(# Copyright \d+) ',
+ r'\1-%s ' % year, line)
+ line = re.sub(r'^(# Copyright) \d\d\d\d-\d\d\d\d',
+ r'\1 1999-%s' % year, line)
+ new_header.append(line)
+
+ difflines = 0
+ for line in difflib.unified_diff(orig_header, new_header,
+ fromfile=fn_path, tofile=fn_path, n=0):
+ util.writemsg_stdout(line, noiselevel=-1)
+ difflines += 1
+ util.writemsg_stdout("\n", noiselevel=-1)
+
+ # unified diff has three lines to start with
+ if difflines > 3 and not pretend:
+ # write new file with changed header
+ f, fnnew_path = mkstemp()
+ f = io.open(f, mode='w', encoding=_encodings['repo.content'],
+ errors='backslashreplace')
+ for line in new_header:
+ f.write(line);
+ for line in fn_hdl:
+ f.write(line)
+ f.close()
+ try:
+ fn_stat = os.stat(fn_path)
+ except OSError:
+ fn_stat = None
+
+ shutil.move(fnnew_path, fn_path)
+
+ if fn_stat is None:
+ util.apply_permissions(fn_path, mode=0o644)
+ else:
+ util.apply_stat_permissions(fn_path, fn_stat)
+ fn_hdl.close()
+
def UpdateChangeLog(pkgdir, category, package, new, removed, changed, \
msg, pretend, repodir):
- """ Write an entry to an existing ChangeLog, or create a new one. """
+ """
+ Write an entry to an existing ChangeLog, or create a new one.
+ Updates copyright year on changed files, and updates the header of
+ ChangeLog with the contents of skel.ChangeLog.
+ """
# figure out who to write as
if 'GENTOO_COMMITTER_NAME' in os.environ and \
@@ -548,6 +616,16 @@
logging.critical(err)
return None
+ year = time.strftime('%Y')
+ date = time.strftime('%d %b %Y')
+
+ # check modified files and the ChangeLog for copyright updates
+ # patches and diffs (identified by .patch and .diff) are excluded
+ for fn in new + changed:
+ if fn.endswith('.diff') or fn.endswith('.patch'):
+ continue
+ update_copyright(os.path.join(pkgdir, fn), year, pretend)
+
cl_path = os.path.join(pkgdir, 'ChangeLog')
clold_lines = []
clnew_lines = []
@@ -584,8 +662,9 @@
clold_lines.append(line)
if line_strip[:1] != '#':
break
- if clskel_file is None:
- clnew_lines.append(line)
+ line = re.sub(r'^(# Copyright)
\d\d\d\d-\d\d\d\d',
+ r'\1 1999-%s' % year, line)
+ clnew_lines.append(line)
if not line_strip:
break
elif clskel_file is not None:
@@ -597,7 +676,7 @@
line = line.replace('', category)
line = line.replace('', package)
line = re.sub(r'^(# Copyright
\d\d\d\d)-\d\d\d\d ',
- r'\1-%s ' % time.strftime('%Y'), line)
+ r'\1
Re: [gentoo-portage-dev] [PATCH 2 of 3] repoman: get ChangeLog header from skel.ChangeLog
On 19-10-2011 14:58:39 -0700, Zac Medico wrote: > On 10/19/2011 12:55 PM, Fabian Groffen wrote: > > + if clold_lines[-1].strip(): > > + f.write(clold_lines[-1]) > > If the old ChangeLog happens to be an empty file, then clold_lines[-1] > will raise IndexError. So, this is safer: > >if clold_lines and clold_lines[-1].strip(): I applied your fix, and moved it from the other patch in this one, where it belongs. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] sleep 1 in misc-functions.sh
The full context of this message is from a thread on gentoo-alt ml: http://archives.gentoo.org/gentoo-alt/msg_db73b1a140fd958efb88f2437170646d.xml Long story short, this person has to deal with a fileserver that sets the rw world bits on for all files that are created. While I don't think the suggested patch is fair (this is obviously a very badly configured fileserver), I do wonder if the "sleep 1" that is in the code is actually desired. In case we want it, I would suggest we move down the "sleep 1" so it's only done once for the entire list with files. Can we use eqawarn in bin/misc-functions.sh:177 instead and avoid the sleep? -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
[gentoo-portage-dev] [PATCH 1 of 3] repoman: get default to update changelog from layout.conf
Updating the ChangeLog file may be desirable in more repos than just the
one named 'gentoo', like e.g. the Prefix one. Hence, make this default
configurable though metadata/layout.conf.
This commit must go accompanied by a commit to
gentoo-x86/metadata/layout.conf that adds the following bit:
# Bug #337853 - gentoo's council says to enable
# --echangelog by default for the "gentoo" repo
update-changelog = true
diff --git a/bin/repoman b/bin/repoman
--- a/bin/repoman
+++ b/bin/repoman
@@ -644,10 +644,7 @@
print(prefix + line)
sys.exit(1)
-if options.echangelog is None and \
- repo_config.name == "gentoo":
- # Bug #337853 - gentoo's council says to enable
- # --echangelog by default for the "gentoo" repo
+if options.echangelog is None and repo_config.update_changelog:
options.echangelog = 'y'
if vcs is None:
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -48,7 +48,7 @@
'eclass_overrides', 'eclass_locations', 'format', 'location',
'main_repo', 'manifest_hashes', 'masters', 'missing_repo_name',
'name', 'priority', 'sign_manifest', 'sync', 'thin_manifest',
- 'user_location')
+ 'update_changelog', 'user_location')
def __init__(self, name, repo_opts):
"""Build a RepoConfig with options in repo_opts
@@ -126,6 +126,7 @@
self.create_manifest = True
self.disable_manifest = False
self.manifest_hashes = None
+ self.update_changelog = False
self.cache_format = None
def get_pregenerated_cache(self, auxdbkeys, readonly=True, force=False):
@@ -431,6 +432,9 @@
DeprecationWarning)
repo.manifest_hashes = manifest_hashes
+ if layout_data.get('update-changelog', '').lower() ==
'true':
+ repo.update_changelog = True
+
#Take aliases into account.
new_prepos = {}
for repo_name, repo in prepos.items():
[gentoo-portage-dev] [PATCH 2 of 3] repoman: get ChangeLog header from skel.ChangeLog
Use skel.ChangeLog from the repo to create the header of a new ChangeLog
file. Else, we just retain the original header of the ChangeLog. When
no skel.ChangeLog file exists, and this is a new ChangeLog, no header is
used.
diff --git a/bin/repoman b/bin/repoman
--- a/bin/repoman
+++ b/bin/repoman
@@ -2450,7 +2450,7 @@
new_changelog =
utilities.UpdateChangeLog(checkdir_relative, \
catdir, pkgdir, \
clnew, clremoved, clchanged, \
- changelog_msg, options.pretend)
+ changelog_msg, options.pretend, repodir)
if new_changelog is None:
writemsg_level("!!! Updating the ChangeLog
failed\n", \
level=logging.ERROR, noiselevel=-1)
diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
--- a/pym/repoman/utilities.py
+++ b/pym/repoman/utilities.py
@@ -523,7 +523,8 @@
return outvcs
-def UpdateChangeLog(pkgdir, category, package, new, removed, changed, msg,
pretend):
+def UpdateChangeLog(pkgdir, category, package, new, removed, changed, \
+ msg, pretend, repodir):
""" Write an entry to an existing ChangeLog, or create a new one. """
# figure out who to write as
@@ -550,7 +551,6 @@
cl_path = os.path.join(pkgdir, 'ChangeLog')
clold_lines = []
clnew_lines = []
- old_header_lines = []
header_lines = []
try:
@@ -559,41 +559,50 @@
mode='r', encoding=_encodings['repo.content'],
errors='replace')
except EnvironmentError:
clold_file = None
+
+ clskel_file = None
+ if clold_file is None:
+ # we will only need the ChangeLog skeleton if there is no
+ # ChangeLog yet
+ try:
+ clskel_path = os.path.join(repodir, 'skel.ChangeLog')
+ clskel_file = io.open(_unicode_encode(clskel_path,
+ encoding=_encodings['fs'], errors='strict'),
+ mode='r', encoding=_encodings['repo.content'],
+ errors='replace')
+ except EnvironmentError:
+ pass
f, clnew_path = mkstemp()
- # create an empty ChangeLog.new with correct header first
+ # construct correct header first
try:
- f = io.open(f, mode='w', encoding=_encodings['repo.content'],
- errors='backslashreplace')
-
- if clold_file is None:
- header_lines.append(_unicode_decode('# ChangeLog for
%s/%s\n' %
- (category, package)))
- year = time.strftime('%Y')
- header_lines.append(_unicode_decode('# Copyright 1999-'
- '%s Gentoo Foundation; Distributed under the
GPL v2\n' % year))
- header_lines.append(_unicode_decode('# $Header: $\n'))
- header_lines.append(_unicode_decode('\n'))
- else:
+ if clold_file is not None:
+ # retain header from old ChangeLog
for line in clold_file:
- line_strip = line.strip()
- if line_strip and line[:1] != "#":
- clold_lines.append(line)
+ line_strip = line.strip()
+ clold_lines.append(line)
+ if line_strip[:1] != '#':
break
- header_lines.append(line)
+ if clskel_file is None:
+ clnew_lines.append(line)
if not line_strip:
break
-
- # update the copyright year
- old_header_lines = header_lines[:]
- if len(header_lines) >= 2:
- header_lines[1] = re.sub(r'^(# Copyright
\d\d\d\d)-\d\d\d\d ',
- r'\1-%s ' % time.strftime('%Y'),
header_lines[1])
+ elif clskel_file is not None:
+ # read skel.ChangeLog up to first empty line
+ for line in clskel_file:
+ line_strip = line.strip()
+ if not line_strip:
+ break
+ line = line.replace('', category)
+ line = line.replace('', package)
+ line = re.sub(r'^(# Copyright
\d\d\d\d)-\d\d\d\d ',
+ r'\1-%s '
[gentoo-portage-dev] [PATCH 0 of 3] repoman changelog patches
The following three patches change repoman's echangelog feature slightly: 1) default for updating the changelog is retrieved from layout.conf 2) skel.ChangeLog is used for constructing the header of new ChangeLogs 3) modified files are checked for copyright years to be updated Please review before I push them.
[gentoo-portage-dev] [PATCH 3 of 3] repoman: update copyright on modified files
To retain the behaviour of echangelog, update the copyrights on modified
files (mostly ebuilds) when necessary. We also use this to update the
ChangeLog's copyright.
diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
--- a/pym/repoman/utilities.py
+++ b/pym/repoman/utilities.py
@@ -523,9 +523,84 @@
return outvcs
+def update_copyrights(pkgdir, files, year, pretend):
+ """
+ Check list of files for Copyright statements, and update them to
+ year. The patterns used for replacing copyrights are taken from
+ echangelog. Only the first lines of each file that start with a
+ hash ('#') are considered, until a line is found that doesn't start
+ with a hash.
+ patches and diffs (identified by .patch and .diff) are excluded
+ """
+
+ for fn in files:
+ if fn.endswith('.diff') or fn.endswith('.patch'):
+ continue
+
+ fn_path = os.path.join(pkgdir, fn)
+ try:
+ fn_hdl = io.open(_unicode_encode(fn_path,
+ encoding=_encodings['fs'], errors='strict'),
+ mode='r', encoding=_encodings['repo.content'],
errors='replace')
+ except EnvironmentError:
+ continue
+
+ orig_header = []
+ new_header = []
+
+ for line in fn_hdl:
+ line_strip = line.strip()
+ orig_header.append(line)
+ if not line_strip or line_strip[:1] != '#':
+ new_header.append(line)
+ break
+
+ # these two regexes are taken from
+ # echangelog update_copyright()
+ line = re.sub(r'^(# Copyright \d+) ',
+ r'\1-%s ' % year, line)
+ line = re.sub(r'^(# Copyright) \d\d\d\d-\d\d\d\d',
+ r'\1 1999-%s' % year, line)
+ new_header.append(line)
+
+ difflines = 0
+ for line in difflib.unified_diff(orig_header, new_header,
+ fromfile=fn_path, tofile=fn_path, n=0):
+ util.writemsg_stdout(line, noiselevel=-1)
+ difflines += 1
+ util.writemsg_stdout("\n", noiselevel=-1)
+
+ # unified diff has three lines to start with
+ if difflines > 3 and not pretend:
+ # write new file with changed header
+ f, fnnew_path = mkstemp()
+ f = io.open(f, mode='w',
encoding=_encodings['repo.content'],
+ errors='backslashreplace')
+ for line in new_header:
+ f.write(line);
+ for line in fn_hdl:
+ f.write(line)
+ f.close()
+ try:
+ fn_stat = os.stat(fn_path)
+ except OSError:
+ fn_stat = None
+
+ shutil.move(fnnew_path, fn_path)
+
+ if fn_stat is None:
+ util.apply_permissions(fn_path, mode=0o644)
+ else:
+ util.apply_stat_permissions(fn_path, fn_stat)
+ fn_hdl.close()
+
def UpdateChangeLog(pkgdir, category, package, new, removed, changed, \
msg, pretend, repodir):
- """ Write an entry to an existing ChangeLog, or create a new one. """
+ """
+ Write an entry to an existing ChangeLog, or create a new one.
+ Updates copyright year on changed files, and updates the header of
+ ChangeLog with the contents of skel.ChangeLog.
+ """
# figure out who to write as
if 'GENTOO_COMMITTER_NAME' in os.environ and \
@@ -548,6 +623,12 @@
logging.critical(err)
return None
+ year = time.strftime('%Y')
+ date = time.strftime('%d %b %Y')
+
+ # check modified files and the ChangeLog for copyright updates
+ update_copyrights(pkgdir, new + changed + ['ChangeLog'], year, pretend)
+
cl_path = os.path.join(pkgdir, 'ChangeLog')
clold_lines = []
clnew_lines = []
@@ -578,7 +659,8 @@
# construct correct header first
try:
if clold_file is not None:
- # retain header from old ChangeLog
+ # retain header from old ChangeLog, its copyright has
+ # already been updated by update_copyrights
for line in clold_file:
line_strip = line.strip()
clold_lines.append(line)
@@ -597,7 +679,7 @@
[gentoo-portage-dev] Re: proj/portage:master commit in: pym/repoman/
On 17-10-2011 00:14:14 +, Zac Medico wrote: > commit: b655137b7b5f87c11a7e742b7ef0e028a1139677 > AuthorDate: Mon Oct 17 00:11:08 2011 + > Commit: Zac Medico gentoo org> > CommitDate: Mon Oct 17 00:11:08 2011 + > URL: > http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=b655137b > > UpdateChangeLog: optimize and add unicode support > > Also: > * copy the old header from the old ChangeLog if it exists, in case it > contains a non-gentoo header Problem with this is that the copyright and package name aren't updated, when they should. I didn't consider repoman being used outside Gentoo (probably should have...) > * don't add a header if the old ChangeLog exists and doesn't contain > a header I'd prefer to actually add logic based on the repo for this. That is, do whatever I did for 'gentoo' repos (or some 'gentoo' property in layout.conf?), and possibly 'retain', and 'empty' as variations. What do you think? Thanks for applying and fixing for unicode. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH 2/2] prepstrip: add support for elfutils strip
On 11-10-2011 10:18:16 -0400, Mike Frysinger wrote: > > For the sake of it, can we try to detect GNU binutils here? Our > > profiles restrict strip, so no immediate problem here if not. > > > adding detection here will fix most, but i don't think all, of your troubles > > the code still unconditionally runs `strip -g` which i'm guessing won't work > for you ... > > also, we only run strip on ELF files. so if you aren't an ELF system, even > fixing the -g won't help. Actually, on Solaris (ELF) the way Portage uses strip will damage the files in such a way that they become unusable. And we're using GNU binutils there. So I guess it's ok to assume here. In the profiles stripping is just disabled everywhere it breaks. -- Fabian Groffen Gentoo on a different level signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH 2/2] prepstrip: add support for elfutils strip
On 11-10-2011 00:50:54 -0400, Mike Frysinger wrote:
> If people use strip from the elfutils package, take advantage of some of
> its neat features (like splitting + stripping in one step).
> +# See if we're using GNU binutils or elfutils for stripping
> +case $(${STRIP} --version) in
> +*elfutils*) # dev-libs/elfutils
> + # elfutils default behavior is always safe, so don't need to specify
> + # any flags at all
> + SAFE_STRIP_FLAGS=""
> + DEF_STRIP_FLAGS="--remove-comment"
> + SPLIT_STRIP_FLAGS="-f"
> + ;;
> +*) # assume binutils
For the sake of it, can we try to detect GNU binutils here? Our
profiles restrict strip, so no immediate problem here if not.
% strip --version
strip: unrecognized option: --version
Usage: strip [-AnuSXx] [-] [-d filename] [-s filename] [-R filename] [-o
output] file [...]
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: Digital signature
Re: [gentoo-portage-dev] EbuildProcess logs poll-error to already removed $T (on AIX)
portage/pym/_emerge/EbuildBuild.py",
> line 260, in _build_exit
> self.wait()
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line
> 41, in wait
> self._wait_hook()
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line
> 123, in _wait_hook
> self._exit_listener_stack.pop()(self)
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/CompositeTask.py",
> line 121, in _default_final_exit
> return self.wait()
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line
> 41, in wait
> self._wait_hook()
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line
> 123, in _wait_hook
> self._exit_listener_stack.pop()(self)
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/Scheduler.py", line
> 1523, in _build_exit
> self._schedule()
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/PollScheduler.py",
> line 103, in _schedule
> return self._schedule_tasks()
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/Scheduler.py", line
> 1719, in _schedule_tasks
> if q.schedule():
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/SequentialTaskQueue.py",
> line 55, in schedule
> task.start()
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line
> 23, in start
> self._start()
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/PackageMerge.py",
> line 43, in _start
> self.returncode = self.merge.merge()
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/MergeListItem.py",
> line 147, in merge
> retval = self._install_task.install()
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/EbuildBuild.py",
> line 348, in install
> rval = merge.execute()
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/EbuildMerge.py",
> line 26, in execute
> blockers=self.find_blockers)
> File "/big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py",
> line 4064, in merge
> mydbapi=mydbapi, prev_mtimes=prev_mtimes)
> File "/big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py",
> line 3915, in merge
> mydbapi=mydbapi, prev_mtimes=prev_mtimes)
> File "/big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py",
> line 3936, in _merge
> cleanup=cleanup, mydbapi=mydbapi, prev_mtimes=prev_mtimes)
> File "/big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py",
> line 3341, in treewalk
> others_in_slot=others_in_slot)
> File "/big5tk/local/gprefix/usr/lib/portage/pym/portage/dbapi/vartree.py",
> line 1700, in unmerge
> myebuildpath, "cleanrm")
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/Scheduler.py", line
> 737, in _dblink_ebuild_phase
> ebuild_phase.wait()
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line
> 40, in wait
> self._wait()
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/CompositeTask.py",
> line 72, in _wait
> task.wait()
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AsynchronousTask.py", line
> 40, in wait
> self._wait()
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/SubProcess.py",
> line 76, in _wait
> self.scheduler.schedule(self._reg_id)
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/PollScheduler.py",
> line 286, in _schedule_wait
> handler(f, event)
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/SpawnProcess.py",
> line 224, in _dummy_handler
> self._unregister_if_appropriate(event)
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AbstractPollTask.py", line
> 56, in _unregister_if_appropriate
> self._log_poll_exception(event)
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AbstractEbuildProcess.py",
> line 234, in _log_poll_exception
> (self.__class__.__name__, event,)])
> File
> "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/AbstractEbuildProcess.py",
> line 229, in _elog
> log_path=self.settings.get("PORTAGE_LOG_FILE"))
> File "/big5tk/local/gprefix/usr/lib/portage/pym/_emerge/PollScheduler.py",
> line 327, in _task_output
> mode='ab')
> IOError: [Errno 2] No such file or directory:
> '/big5tk/tmp/gprefix/portage/._unmerge_/app-misc/mime-types-8/temp/build.log'
>
--
Fabian Groffen
Gentoo on a different level
Re: [gentoo-portage-dev] Portage on AIX
On 24-11-2010 12:29:41 -0600, Perry Smith wrote: > > On Nov 24, 2010, at 12:22 PM, Fabian Groffen wrote: > > > Hi, > > > > On 24-11-2010 12:17:12 -0600, Perry Smith wrote: > >> I don't use Gentoo but I bumped into Portage on a recent quest. Long > >> story short, I'd like to try and port it to AIX. I'm writing this list to > >> see if you think that is possible or not. > > > > We use Portage Prefix on AIX, so I think we can say it works. Do you > > really need Portage (the package manager) or the entire set of > > applications provided by it? > > I'm assuming the latter does not exist. I've love to just say "install ruby" > right now and have it work but that isn't what I'm expecting. I'm hoping, > over time, to provide some of those packages over time. I hope I'm making > sense here... Gentoo Prefix[1] on AIX is sort of that. Not sure if ruby is "ported" yet, but at least the base system (the core you need to get Portage sanely running, and being able to emerge other stuff + a handful of indispensible tools) compiles. You'll find people on the gentoo-alt list[2] that maintain the AIX port. [1] http://prefix.gentoo.org/ [2] http://archives.gentoo.org/gentoo-alt/ -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] Portage on AIX
Hi, On 24-11-2010 12:17:12 -0600, Perry Smith wrote: > I don't use Gentoo but I bumped into Portage on a recent quest. Long story > short, I'd like to try and port it to AIX. I'm writing this list to see if > you think that is possible or not. We use Portage Prefix on AIX, so I think we can say it works. Do you really need Portage (the package manager) or the entire set of applications provided by it? -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] [API] First steps for creating an API for portage
On 18-06-2010 01:27:57 -0700, Brian Dolbec wrote: > Unless I'm way off the mark, your suggesting to re-code portage into "C" > then create a python interface to it. Others have started such a > project, but none have completed aside from paludis (C++). It's actually on my wishlist for a long time, just because Python is an extreme pain in my environments, and given its current maintainer it is going to be an extreme pain for a long time as well. > If there are other apps that would like to get info from portage via > this API, that is fine with me. I have intentions of making the "C" > interface available to all apps that desire to. If there is data that > they want to get from portage via the API, then they should send their > wish list in and we'll do our best to get it in there for them. > > Currently the API will be based on the packagekit portage backend code > that was produced last summer and other code from porthole, portato, > kportagetray. We are also going to put together a layman API for > consumer apps (guis frontends) to use to operate layman without the need > to run it in and parse terminal output. It too will hopefully be > available via a "C" interface for non-python apps. I don't want to push you into the C corner, but tools like qfile and qlist are in orders of magnitude faster than their equery variants for a good reason. Maybe this simple command gives you an idea why I think python and portage are slow. % time portageq envvar CHOST powerpc-apple-darwin8 0.488u 0.704s 0:03.09 38.1% 0+0k 25+70io 0pf+0w (three full seconds to return a very simple var) That said, if you design C APIs, please design them from a C point of view, initially implemented by your Python functionality doing the necessary wrapping to get a sane C structure. Then they can be replaced by native C code as RSI and time permits in the future. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] [API] First steps for creating an API for portage
On 18-06-2010 02:08:04 +0200, René 'Necoro' Neumann wrote: > In parallel (or thereafter), we build the C-bindings. The API for these > bindings probably look different -- but I guess they should be > implemented in terms of the library created above. > > By example: > > - Operation: get the list of categories > - Python-API: portage.api.categories() : Category list > - Implementation: def categories(): return > - C-API: category * categories() > - C-Implementation: some wrapper around portage.api.categories If you want to deliver a C implementation, I'd wonder why you wouldn't just make a full C implementation and create Python wrappers for them instead of the other way around. Might accidentially speed up Portage, and make tools like portage-utils happy. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] [PATCH] config_root variables in pym/portage/const.py
On 17-08-2009 23:06:01 -0700, Zac Medico wrote: > Zac Medico wrote: > > Fabian Groffen wrote: > >> The attached patch makes all variables against config_root relative (by > >> removing the leading '/'), with the result that all lstrip(os.sep) calls > >> in the code can be removed. Please note that all but two occurences did > >> use config_root. For one (pym/portage/__init__.py; sandbox stuff) I've > >> added it, for the other (in pym/portage/output.py) I don't know how to > >> get config_root, so substituted '/', which breaks Prefix by design. > > > > Your patch is in svn r13821. > > > > I've added a comment in output.py that we can use ObjectProxy to > > implement lazy initialization of codes and _styles, and add an > > init(config_root="/") function that can be called in order adjust > > the location that color.map is read from. > > I've added an _init(config_root="/") function like this in r14083. thanks, I've removed the FIXME comment in trunk, since it's fixed now -- Fabian Groffen Gentoo on a different level
[gentoo-portage-dev] [PATCH] config_root variables in pym/portage/const.py
In Prefix, there is a distinction between variables used with
config_root and target_root. Quoting from prefix' const.py:
# We have a most confusing situation here, which is most of all pretty
# weak for protecting us from making mistakes.
# First there is a config_root (PORTAGE_CONFIGROOT) which can be a path
# somewhere, from which all paths need to be relative (e.g.
# etc/portage), hence those constants do NOT have EPREFIX, because
# config_root contains EPREFIX by default -- overriding it loses the
# EPREFIX as one would expect.
# Second there is target_root (ROOT) which is to install somewhere
# completely else, in Prefix of limited use. Because this is an offset
# always given, the EPREFIX should always be applied in it. Those
# constants (like VDB_PATH) DO have EPREFIX.
# Unfortunately this file is ordered quite horrible in this respect.
The attached patch makes all variables against config_root relative (by
removing the leading '/'), with the result that all lstrip(os.sep) calls
in the code can be removed. Please note that all but two occurences did
use config_root. For one (pym/portage/__init__.py; sandbox stuff) I've
added it, for the other (in pym/portage/output.py) I don't know how to
get config_root, so substituted '/', which breaks Prefix by design.
How about this patch, and if it is a good idea, how about grouping all
of these config_root variables such that it's easier to see what is
what? I actually ran into this because Sebastian was hitting it with
his smolt-gentoo tool.
--
Fabian Groffen
Gentoo on a different level
Index: pym/portage/__init__.py
===
--- pym/portage/__init__.py (revision 13819)
+++ pym/portage/__init__.py (working copy)
@@ -1309,7 +1309,7 @@
if not config_profile_path:
config_profile_path = \
- os.path.join(config_root,
PROFILE_PATH.lstrip(os.path.sep))
+ os.path.join(config_root, PROFILE_PATH)
if os.path.isdir(config_profile_path):
self.profile_path = config_profile_path
else:
@@ -1325,7 +1325,7 @@
self.module_priority= ["user","default"]
self.modules= {}
self.modules["user"] = getconfig(
- os.path.join(config_root,
MODULES_FILE_PATH.lstrip(os.path.sep)))
+ os.path.join(config_root, MODULES_FILE_PATH))
if self.modules["user"] is None:
self.modules["user"] = {}
self.modules["default"] = {
@@ -1389,7 +1389,7 @@
self.profiles = []
if local_config and self.profiles:
custom_prof = os.path.join(
- config_root,
CUSTOM_PROFILE_PATH.lstrip(os.path.sep))
+ config_root, CUSTOM_PROFILE_PATH)
if os.path.exists(custom_prof):
self.user_profile_dir = custom_prof
self.profiles.append(custom_prof)
@@ -1465,7 +1465,7 @@
del rawpuseforce
make_conf = getconfig(
- os.path.join(config_root,
MAKE_CONF_FILE.lstrip(os.path.sep)),
+ os.path.join(config_root, MAKE_CONF_FILE),
tolerant=tolerant, allow_sourcing=True)
if make_conf is None:
make_conf = {}
@@ -1561,7 +1561,7 @@
self.configdict["defaults"]=self.configlist[-1]
self.mygcfg = getconfig(
- os.path.join(config_root,
MAKE_CONF_FILE.lstrip(os.path.sep)),
+ os.path.join(config_root, MAKE_CONF_FILE),
tolerant=tolerant, allow_sourcing=True,
expand=expand_map)
if self.mygcfg is None:
self.mygcfg = {}
@@ -1610,8 +1610,7 @@
self.pkeywordsdict = {}
self._plicensedict = {}
self.punmaskdict = {}
- abs_user_config = os.path.join(config_root,
- USER_CONFIG_PATH.lstrip(os.path.sep))
+ abs_user_config = os.path.join(config_root,
USER_CONFIG_PATH)
# locations for "categories" and "arch.list" files
locations = [os.path.jo
Re: [gentoo-portage-dev] prefix-portage-2.2.00.13734 with --nodeps merges in reverse order
On 02-07-2009 15:31:16 +0200, Michael Haubenwallner wrote: > Hi, > > seems prefix-portage as of r13734, when used with '--nodeps', does merge > packages in reverse order than given on cmdline, but I don't believe > this is a prefix issue... > > prefix-portage r13683 does merge in correct order. 13734 is after the _emerge split -- Fabian Groffen Gentoo on a different level
[gentoo-portage-dev] Portage and SIGHUP
I recently noticed this: (pegasus:portage/dev-libs/poppler) fabian% emerge --resume Calculating dependencies... done! *** Resuming merge... >>> Verifying ebuild manifests >>> Starting parallel fetch >>> Emerging (1 of 9) dev-libs/libgpg-error-1.7 * libgpg-error-1.7.tar.bz2 RMD160 SHA1 SHA256 size ;-) ... [ ok ] * checking ebuild checksums ;-) ... [ ok ] * checking auxfile checksums ;-) ...[ ok ] * checking miscfile checksums ;-) ... [ ok ] >>> Unpacking source... >>> Unpacking libgpg-error-1.7.tar.bz2 to >>> /net/pegasus.ins.cwi.nl/export/scratch/stripe/fabian/gentoo/prefix64/var/tmp/portage/dev-libs/libgpg-error-1.7/work * Running eautoreconf in '/net/pegasus.ins.cwi.nl/export/scratch/stripe/fabian/gentoo/prefix64/var/tmp/portage/dev-libs/libgpg-error-1.7/work/libgpg-error-1.7' ... * Running aclocal -I m4 -I /net/pegasus.ins.cwi.nl/export/scratch/stripe/fabian/gentoo/prefix64/usr/share/aclocal ... Hangup (pegasus:portage/dev-libs/poppler) fabian% e.g. Portage terminating immediately on a SIGHUP. If I sent SIGHUP myself to Portage from another shell it immediately stops as well. Can some one Linux try to reproduce this? I tested it by doing emerge -av portage and from another terminal SIGHUP that emerge. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] prefix portage chaining
On 20-03-2009 11:35:09 +0100, Markus Duft wrote: > i'll try and explain what i want in the first place: i'm porting things > to native windows. since windows isn't too cooperative, i'm unable to > merge most things (and with other things, i simply don't want to), and > thus i need to take those things from somewhere else (more or less the > complete @system). I _am_ able to build all those things for interix > (which is the host system in the windows case). So what i want is a > setup of two prefix instances with a certain relation to each other: the > native windows prefix should be able to recognize installed packages > from the other instance, and resolve dependencies by eventually using > the other .../var/db/... Since Windows and Interix seem to be two different things to me, can you explain why in this case Portage can resolve the dependencies from the Interix system to use for the Windows system? What dependencies are we talking about? Build tools? Libraries? Runtime dependencies? > This could be (and is) quite usefull for all other platforms too. For > exmaple i could use prefix chaining on a linux box. I could create a > prefix containing a base system, and then for testing of > i-don't-know-whatever, i could create another small prefix inheriting > all installed packages from the other one. this way new prefixes can > stay very slim, but still the "parent" prefix is not altered on merges. That potentially is useful, in the case where you want to upgrade a critical system package and test it out or something. Can't think of an example other than Portage itself for the moment, though. (And that one can be hairy, for instance if the vdb format changes NEEDED -> NEEDED.ELF.2) > one issue not handled by the current patch is, that prefixes can have > different CHOST/ARCH/... (which is the case with x86-interix and > x86-winnt for example). Then how does it work for you? -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] How to extract the version/revision of an installed package?
On 25-11-2008 16:15:09 -0800, Ned Ludd wrote:
> On Tue, 2008-11-25 at 18:05 +0200, Amit Dor-Shifer wrote:
> > Given the following:
> > # qlist -Iv sys-apps/portage
> > sys-apps/portage-2.1.4.5
> >
> > How do I safely extract the "2.1.4.5"?
> >
> > (I don't necessarily need to use qlist. Just want to get the version of
> > an installed package within a bash script)
>
> qatom $(qlist -ICv sys-apps/portage) | awk '{print $3}'
I happen to do it in bash for my update script, it hasn't broke yet:
get_ebuildversion() {
if [[ -z $1 ]] ; then
return
fi
# strip extension
t=${1%.ebuild}
# abort if this is not an ebuild
if [[ $t == $1 ]] ; then
return
fi
# strip package name
t=${t#*-}
# sometimes there are dashes in the package name
while [[ ${t:0:1} < "0" || ${t:0:1} > "9" ]] ; do
[[ $t != ${t#*-} ]] \
&& t=${t#*-} \
|| break
done
echo ${t}
}
you might want to remove the .ebuild check and strip the category ($1#*/})
--
Fabian Groffen
Gentoo on a different level
Re: [gentoo-portage-dev] search functionality in emerge
On 24-11-2008 10:34:28 +0100, René 'Necoro' Neumann wrote: > tvali schrieb: > > There is daemon, which notices about filesystem changes - > > http://pyinotify.sourceforge.net/ would be a good choice. > > Disadvantage: Has to run all the time (I see already some people crying: > "oh noez. not yet another daemon..."). ... and it is Linux only, which spoils the fun. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] Re: portage-2.2-rc3 parallel merges quit being parallel
On 28-07-2008 08:10:47 +, Duncan wrote: > René 'Necoro' Neumann <[EMAIL PROTECTED]> posted [EMAIL PROTECTED], > > A very basic thingy: > > watch qlop -cC > > > > qlop is part of portage-utils. And it seems to only work correctly here, > > when run as root :) > > Thanks! =8^) and when sandbox is enabled, as it looks for [sandbox] in the process list -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] [PATCH] Repoman subversion support
On 24-07-2008 17:38:51 +0200, Arfrever Frehtes Taifersar Arahesis wrote: > `svn list` returns only files which actually exist in the repository. It > doesn't check working copy. You should use `svn st`. Ok, you're right. But svn status simply sucks: [usr/portage/eclass] % touch x11.eclass.rej [usr/portage/eclass] % svn status ? files [usr/portage/eclass] % In other words, it auto-ignores stuff that really causes the messed up manifests if someone isn't paying attention. IIRC this ignorance isn't limited to .rej, but also other extentions, such as the trailing ~, which some editors leave behind. I guess we need to use --no-ignore, and then also recognise the "I" entries. Bah. -- Fabian Groffen Gentoo on a different level
Re: [gentoo-portage-dev] [PATCH] Repoman subversion support
On 14-05-2008 17:56:38 +0200, Fabian Groffen wrote:
> On 14-05-2008 16:27:28 +0200, Arfrever Frehtes Taifersar Arahesis wrote:
> > 2008-05-14 00:32 Marius Mauch <[EMAIL PROTECTED]> napisał(a):
> > > Merged in r10325 with some minor changes (removed the 'svn update' bit
> > > until someone remembers why exactly it's there
> >
> > During committing, only files, which are being committed, are being
> > updated, so `svn up` is certainly a good idea. Please add it.
>
> Details here are that the code did this:
>
> if vcs == "cvs":
> myf=open(checkdir+"/CVS/Entries","r")
> if vcs == "svn":
> myf=os.popen("svn update > /dev/null; svn list")
>
> genone's remark here refers to this `svn update`, of which I don't
> recall any more why I added it. All I know is that I think I needed it
> for some reason to get a correct output of "list". However, I have no
> proof here, so he removed the svn update, which gives a non-announced
> possible modification of the local repository.
I've seen numerous examples like bug 232825[1] now, and I'm almost
confident the svn update resolved that problem. Problem is that the
file (a .rej in the case of the bug) exists in the local dir, but
repoman doesn't notice it, since svn list doesn't return it (as '?').
Note that this sometimes happens, and sometimes not. I haven't found
the rule here yet.
[1] http://bugs.gentoo.org/show_bug.cgi?id=232825
--
Fabian Groffen
Gentoo on a different level
Re: [gentoo-portage-dev] [PATCH] Repoman subversion support
On 14-05-2008 16:27:28 +0200, Arfrever Frehtes Taifersar Arahesis wrote:
> 2008-05-14 00:32 Marius Mauch <[EMAIL PROTECTED]> napisał(a):
> > Merged in r10325 with some minor changes (removed the 'svn update' bit
> > until someone remembers why exactly it's there
>
> During committing, only files, which are being committed, are being
> updated, so `svn up` is certainly a good idea. Please add it.
Details here are that the code did this:
if vcs == "cvs":
myf=open(checkdir+"/CVS/Entries","r")
if vcs == "svn":
myf=os.popen("svn update > /dev/null; svn list")
genone's remark here refers to this `svn update`, of which I don't
recall any more why I added it. All I know is that I think I needed it
for some reason to get a correct output of "list". However, I have no
proof here, so he removed the svn update, which gives a non-announced
possible modification of the local repository.
--
Fabian Groffen
Gentoo on a different level
--
gentoo-portage-dev@lists.gentoo.org mailing list
Re: [gentoo-portage-dev] [PATCH] show binhosts as "repository"
On 14-05-2008 00:45:10 +0200, Marius Mauch wrote: > On Mon, 12 May 2008 20:53:35 +0200 > Fabian Groffen <[EMAIL PROTECTED]> wrote: > > > The following patch shows the url to the binhost in an emerge -av as > > repository name, instead of "unknown". Unfortunately the patch > > doesn't store the binhost url, such that portage can't show where the > > package comes from when unmerged. > > Not sure if this is the right thing to do, or if the reponame in those > cases should be the one from the source repo that was used to generate > those binary packages. Not saying that the binhost name here is > useless, quite the opposite, just a bit concerned about mixing > different things here. The idea behind it is that the repo_name of the binhost is not necessarily useful. In many cases it will contain the default repo_name value, since people just create binpkgs with a standard Portage install. I agree it is a hack. Perhaps a combination of the repo_name and its url would be the best. -- Fabian Groffen Gentoo on a different level -- gentoo-portage-dev@lists.gentoo.org mailing list
[gentoo-portage-dev] [PATCH] include status support for xterm-color and interix
Attached patch adds statusbar support for xterm-color and interix terminals. -- Fabian Groffen Gentoo on a different level --- ../../trunk/pym/portage/output.py 2008-05-12 19:25:13 +0200 +++ ./pym/portage/output.py 2008-05-08 21:17:50 +0200 @@ -246,7 +246,7 @@ if len(mystr) > max_len: mystr = mystr[:max_len] myt=os.environ["TERM"] - legal_terms = ["xterm","Eterm","aterm","rxvt","screen","kterm","rxvt-unicode","gnome"] + legal_terms = ["xterm","xterm-color","Eterm","aterm","rxvt","screen","kterm","rxvt-unicode","gnome","interix"] for term in legal_terms: if myt.startswith(term): if not raw:
[gentoo-portage-dev] [PATCH] show binhosts as "repository"
The following patch shows the url to the binhost in an emerge -av as
repository name, instead of "unknown". Unfortunately the patch doesn't
store the binhost url, such that portage can't show where the package
comes from when unmerged.
--
Fabian Groffen
Gentoo on a different level
--- ../../trunk/pym/_emerge/__init__.py 2008-05-12 19:25:21 +0200
+++ ./pym/_emerge/__init__.py 2008-05-12 17:16:49 +0200
@@ -1117,6 +1126,6 @@
pkg_chost = pkg.metadata.get("CHOST")
if pkg_chost and pkg_chost != pkgsettings["CHOST"]:
return False
if not portage.eapi_is_supported(pkg.metadata["EAPI"]):
return False
if not pkg.installed and \
@@ -4447,13 +4471,18 @@
pkg = x
metadata = pkg.metadata
ebuild_path = None
- repo_name = metadata["repository"]
+ if pkg_type == "binary":
+ repo_name =
self.roots[myroot].settings.get("PORTAGE_BINHOST")
+ else:
+ repo_name = metadata["repository"]
if pkg_type == "ebuild":
ebuild_path = portdb.findname(pkg_key)
if not ebuild_path: # shouldn't happen
raise
portage.exception.PackageNotFound(pkg_key)
repo_path_real =
os.path.dirname(os.path.dirname(
os.path.dirname(ebuild_path)))
+ elif pkg_type == "binary":
+ repo_path_real = repo_name
else:
repo_path_real =
portdb.getRepositoryPath(repo_name)
pkg_use = metadata["USE"].split()
@@ -5422,6 +5451,11 @@
self._repo_paths_real = [ os.path.realpath(repo_path) \
for repo_path in repo_paths ]
+ if root_config.settings.get("PORTAGE_BINHOST"):
+ binhost = root_config.settings.get("PORTAGE_BINHOST")
+ self._repo_paths.append(binhost)
+ self._repo_paths_real.append(binhost)
+
# pre-allocate index for PORTDIR so that it always has index 0.
for root_config in roots.itervalues():
portdb = root_config.trees["porttree"].dbapi
[gentoo-portage-dev] [PATCH] remove eselect compiler usage
eselect compiler has been removed from the tree, hence its usage can be
removed from portage.
--
Fabian Groffen
Gentoo on a different level
--- ../../trunk/pym/_emerge/__init__.py 2008-05-12 19:25:21 +0200
+++ ./pym/_emerge/__init__.py 2008-05-12 17:16:49 +0200
@@ -303,12 +318,6 @@
"!!! other terminals also.\n"
)
- mystatus, myoutput = commands.getstatusoutput("eselect compiler show")
- if mystatus == os.EX_OK and len(myoutput.split("/")) == 2:
- part1, part2 = myoutput.split("/")
- if part1.startswith(chost + "-"):
- return myoutput.replace(chost + "-", gcc_ver_prefix, 1)
-
mystatus, myoutput = commands.getstatusoutput("gcc-config -c")
if mystatus == os.EX_OK and myoutput.startswith(chost + "-"):
return myoutput.replace(chost + "-", gcc_ver_prefix, 1)
[gentoo-portage-dev] [PATCH] unpack .deb files with deb2tgz
Attached patch is necessary for some extreme platforms, as can be read
in the comments.
--
Fabian Groffen
Gentoo on a different level
--- ../../trunk/bin/ebuild.sh 2008-04-27 17:36:18 +0200
+++ ./bin/ebuild.sh 2008-04-13 11:41:55 +0200
@@ -372,9 +372,22 @@
LHa|LHA|lha|lzh)
lha xfq "${srcdir}${x}" || die "$myfail"
;;
- a|deb)
+ a)
ar x "${srcdir}${x}" || die "$myfail"
;;
+ deb)
+ # Unpacking .deb archives can not always be
done with
+ # `ar`. For instance on AIX this doesn't work
out. If
+ # we have `deb2targz` installed, prefer it over
`ar` for
+ # that reason. We just make sure on AIX
`deb2targz` is
+ # installed.
+ if type -P deb2targz > /dev/null; then
+ deb2targz "${srcdir}/${x}" || die
"$myfail"
+ mv "${srcdir}/${x/.deb/.tar.gz}"
data.tar.gz
+ else
+ ar x "${srcdir}/${x}" || die "$myfail"
+ fi
+ ;;
lzma)
if [ "${y}" == "tar" ]; then
lzma -dc "${srcdir}${x}" | tar xof -
${tar_opts}
