Re: [gentoo-dev] [EAPI 7] Cross-compile improvements (BDEPEND, BROOT, sysroot)

[Gregory M. Turner]

On Tue, 2015-12-01 at 22:58 +, James Le Cuirot wrote:
> Sorry
[snip]
>  gentle! :)

Not sure I have anything meaningful to contribute to the discussion
from a technical perspective at the moment, but I just wanted to voice
my support for what you're doing with cross: I think Gentoo is setting
some kind of standard -- or at least exerting a huge influence -- on
the worldwide group-think about what is possible and/or reasonable to
expect in cross-buildling frameworks and environments.

Unfortunately, the "standard" we are setting could clearly be higher.

Part of that comes down to a lack of manpower and testing at
the package level, which is not an easy problem to solve, as it
arises from the fundamentals of the market for open-source hacking
labor.

But what could be solved more readily is that the framework we have
today conflates some entities and concepts which are unified when
building locally, but divergent when cross-building.

Unravelling those conflations should help everyone  to experience less
cognitive dissonance and/or head-scratching-befuddlement, which, in
turn, could lead to a more productive, fun and easy-to-use cross-
building experience for users and developers alike, without expecting
any mysterious motivational forces to materialize from nowhere to
rescue us from the desire to watch TV or go on dates.

Anyhow, I say it's easier, but I know it's still pretty damn hard.  So,
thanks for working on this!

-gmt

Re: [gentoo-dev] impending c++11 clusterfuck?

[Gregory M. Turner]

On Wed, 2015-12-02 at 08:06 +0100, Michał Górny wrote:
> On Tue, 01 Dec 2015 18:12:17 -0800
> "Gregory M. Turner" <g...@be-evil.net> wrote:
> 
> > On Tue, 2015-12-01 at 07:18 -0500, Anthony G. Basile wrote:
> > > So on that bug we're talking about selectively
> > > adding -std=c++11 (or possibly gnu++11) to packages.  
> > 
> > Yes, this is the biggest real-world problem we face.  It requires
> > an
> > immediate solution in the ~* branches; the affected ebuilds just
> > dump a
> > bunch of gcc gobbldeygook and crash.
> > 
> > If I understand the generalized problem we are facing there, as a
> > package gets "c++11-ized", all of its reverse-BDEPs find themselves
> > in
> > the following situation:
> > 
> > Imagine cat/foo is a library.  cat/foo-3.0 is the newest non-c++11-
> > ized 
> > version (in its slot, perhaps).  Now cat/foo-3.1 comes along and
> > from
> > then on, anything BDEPENDing on it must be built with -std=c++11
> > (I'm
> > ignoring the c++11 vs. gnu++11 but presumably we'll eventually need
> > to
> > figure out some kind of game plan about that).
> 
> Just to be clear, this only happened because upstream started to use
> C++11 features in API. Normally libraries don't go that far, or use
> #ifs to support C++03.
> 
> > So, now, "everything" pulling in headers from  cat/foo finds itself
> > in
> > this situation:
> > 
> >  o if it pulls in headers from  >  o if it pulls in headers from >=cat/foo-3.1, we must add a CFLAG
> > 
> > I guess the reasonable way to achieve that sort of behavior is
> > pkgconfig (which is not really a rock-solid solution.  First, some
> > packages might fail to put it in there (but, OK we just add it
> > ourselves, let's say).  Second, how many of cat/foo's reverse BDEPs
> > side-step pkgconfig?  Sometimes this is pretty common, I'm afraid.
> >  
> 
> pkg-config is not a solution. -std= flags don't belong there.
> Furthermore, if you put -std= flags there, it is likely to override
> ${CXXFLAGS} and in particular make it really painful to force other
> standard on ebuild's end.

I see what you mean.  Maybe pkgconfig needs a new feature?  I wonder if
they are already thinking about it in pkgconfig-land.  I'll look around
on their ml/bugdb and see if I can find anything.

> > So, let's say lots of packages depending on cat/foo have this
> > pkgconfig
> > side-stepping problem, we could theoretically write some eclass to
> > inject it when appropriate, and expect those side-stepping ebuilds
> > to
> > consume it, no?  That seems pretty easy and hopefully wouldn't
> > require
> > any of the scary ideas that have been discussed in this thread.
> 
> So you're saying we should write a whole eclass to do:
> 
>   append-cppflags -std=c++11
> 
> ?

If we could just do "append-cppflags" somewhere, it'd be a non-bug as
we would have just done it.  That doesn't seem to be the case.

If I'm not mixed up, the difficulty is determining /when/ that's
appropriate.  Specifically, this c++11 requirement can bubble up
indirectly through deep dependencies or even non-DEPENDencies.  Source
file "X" just needs to #include header "Y" that #include's header "Z"
with the c++11 requirement, and "kablooey!".

So, yeah, an eclass that figures out when to append the flag (and then,
as you say, appends it) doesn't seem so crazy to me.

For example: if we can identify the dependencies, deep, hidden, or not,
that cause these failures (i.e., by waiting for bugzilla reports and
acting on them), we could pretty trivially fix them with such an
eclass.

So, just imagining one possible implementation of that, here's a
possible user-story -- which I mean, an eclass user (and therefore and
ebuild or eclass developer, but developer-story isn't a word afaik):

Bug wranglers observe that www-client/frobzilla ebuilds are crashing
due to headers, which are coming, ultimately, from net-libs/tldr. 

Assignees of the bug would:

o figure out which versions of net-libs/tldr trigger the problem
  and ensure they are selected by the hand-crafted list of
  DEPENDency specifiers that live in cxx11reqs.eclass.

o add the following code to all affected www-client/frobzilla ebuilds:

  CXX11REQ_CULPRITS=( net-libs/tldr )
  inherit cxx11reqs

This would cause the right thing to happen (CXXFLAGS augmentation) when
and only when the affected versions of tldr are installed.

Only thing left to do is to check that frobzilla doesn't strip the flag
back out, for some reason; if it did, perhaps it could add it back in,
by calling a xxx11reqs eclass API.

Note that frobzilla might not even DEPEND on tldr (not even indirectly
through several layers of DEPENDency); this would still do the trick;
the eclass could just use has_version to find out if the flag is
needed, according to a hand-crafted dependency specifier table.

I guess as far as bright ideas go, that's the best I've got, for now.

If a fully automagical solution is practical to implement, I haven't
yet imagined it.

-gmt

Re: [gentoo-dev] impending c++11 clusterfuck?

[Gregory M. Turner]

On Tue, 2015-12-01 at 07:18 -0500, Anthony G. Basile wrote:
> So on that bug we're talking about selectively
> adding -std=c++11 (or possibly gnu++11) to packages.

Yes, this is the biggest real-world problem we face.  It requires an
immediate solution in the ~* branches; the affected ebuilds just dump a
bunch of gcc gobbldeygook and crash.

If I understand the generalized problem we are facing there, as a
package gets "c++11-ized", all of its reverse-BDEPs find themselves in
the following situation:

Imagine cat/foo is a library.  cat/foo-3.0 is the newest non-c++11-ized 
version (in its slot, perhaps).  Now cat/foo-3.1 comes along and from
then on, anything BDEPENDing on it must be built with -std=c++11 (I'm
ignoring the c++11 vs. gnu++11 but presumably we'll eventually need to
figure out some kind of game plan about that).

So, now, "everything" pulling in headers from  cat/foo finds itself in
this situation:

 o if it pulls in headers from =cat/foo-3.1, we must add a CFLAG

I guess the reasonable way to achieve that sort of behavior is
pkgconfig (which is not really a rock-solid solution.  First, some
packages might fail to put it in there (but, OK we just add it
ourselves, let's say).  Second, how many of cat/foo's reverse BDEPs
side-step pkgconfig?  Sometimes this is pretty common, I'm afraid.  

So, let's say lots of packages depending on cat/foo have this pkgconfig
side-stepping problem, we could theoretically write some eclass to
inject it when appropriate, and expect those side-stepping ebuilds to
consume it, no?  That seems pretty easy and hopefully wouldn't require
any of the scary ideas that have been discussed in this thread.

-gmt

Re: [gentoo-dev] rfc: native multilib support in portage for eapi 7

[Gregory M. Turner]

On Tue, 2015-12-01 at 18:45 +0100, Michał Górny wrote:
> On Tue, 1 Dec 2015 11:38:08 -0600
> William Hubbs  wrote:
> 
> > I find the multilib eclasses and their separate multilib phase
> > functions
> > to be confusing, so I was wondering if we could discuss making
> > multilib
> > support native to portage in eapi 7 so that we can use the normal
> > phase
> > functions again?
> 
> That won't help you since the in-spec support would require special
> phase functions anyway.

It might help with other things, though.  I think it might be hard to
build sufficient consensus around this, but, if we could do so, it
could really be a boon.

As I recently pointed out in another thread, if we had a really easy-
to-use, high-performing dedicated vdb feature behind "multi-ness," we
might realize all kinds of benefits.

For example -- of course the following are hoplessly speculative but
just humor me for a moment -- in a sufficiently flexible
implementation, concievably things like prefix and crossdev could be
normalized as, respectively, user-configurable "EPREFIX" and "target"
"multi-" dimensions.

Not saying they should be, though, just making the point that the
concievable applications of a feature tend to correspond to the
feature's limitations and strengths from a practical perspective, and
our current multi- eclass frameworks are fairly limited in precisely
that way, despite being tremendously clever solutions to a very
difficult problem.

-gmt

Re: [gentoo-dev] impending c++11 clusterfuck?

[Gregory M. Turner]

On Sun, Nov 29, 2015 at 10:42 PM, Michał Górny <mgo...@gentoo.org> wrote:
> On Sun, 29 Nov 2015 19:56:04 -0800
> "Gregory M. Turner" <g...@be-evil.net> wrote:
> >  the mess gets magically cleaned up by robots somehow.
>
> Sadly := can't help here since gcc switches occur independently of
> package installs. And AFAIK revdep-rebuild doesn't help either.

Indeed... magical hand-wavey part is the only teeny-tiny flaw with my
proposal that the problem be magically, hand-wavically solved --
otherwise, it's rock-solid :)

OTOH when you're spending ten hours a week waiting for revdep-rebuild,
preserved-rebuild stops seeming like an intractable fantasy and more
like an interesting puzzle you'd might as well take a crack at
solving.

I dunno about y'all but my not-really-that-computationally-powerful
workstation has 2000 packages installed, about three-quarters of which
seem to have forked webkit (just kidding, but it sure feels that way).

emerge -e @world literally takes me two days if I'm lucky.  If I'm
not, more like a week, with my cores left mostly idling while my
productivity is thwarted by various non-webkit-compiling distractions
like using the toilet or working.

Point being, it's damn slow; I'm starting to feel like the guy waiting
for revdep-rebuild all week.  There's gotta be a way to do it.  If it
requires some encapsulation-breaking hacks to get it done (as
preserved-rebuild kinda did) it's probably well worth it.

-gmt

Greg Turner
g...@be-evil.net

[gentoo-dev] impending c++11 clusterfuck?

[Gregory M. Turner]

I'm quoting myself from bug #566328 here.  These were off-the-cuff
remarks that got away from me and became a call-to-arms...

(In reply to Michał Górny from comment #7)
> This is never this simple. C++11 can change the ABI. So the point kinda is,
> we need to ensure that all C++ libraries in a depgraph use the same C++
> version.

This is pretty awful when you really think about it.  I feel like I'm
watching a train-wreck in super slow motion.

I'm not sure we're taking this seriously enough -- sooner or later it
seems destined to become a major clusterfuck if we don't do something
proactive about it now while the drawing-board is relatively
uncluttered.

The only thing I can think of that has this kind of two-way depgraph
magic property are the major "abi" USE_EXPAND values (multilib-build
and python-r1, in other words).

But those rely on fancy framework-generated USE-flag deps, which seem
like overkill and likely to incur unjustifiable user-experience-costs.

Perhaps a solution to this cxx11 clusterfuck can be found that works
more like perl?  By that I mean, pick your poison (respectively, your
cxx11 ABI of preference or your major perl version of choice), rely on
inbuilt portage features do the trick most of the time, and, when it
breaks, run "magically-fix-everything.sh," grab a caffeinated beverage
or three and fire up your favorite VOD client while the mess gets
magically cleaned up by robots somehow.

-gmt

Greg Turner
g...@be-evil.net

[gentoo-dev] rfc: toolchain-funcs: target-has-split-usr API?

[Gregory M. Turner]

What do people think of something like this?  Obviously the equivalent 
patch to prefix would need to include a test for 
PREFIX_DISABLE_GEN_USR_LDSCRIPT:


Author: Gregory M. Turner gmturner...@ameritech.net
Date:   Fri Apr 12 11:13:21 2013 -0700

eclass/toolchain-funcs: Add target-has-split-usr API

Move the platform-filtering code from gen_usr_ldscript into its own
API so that ebuilds can reliably test whether gen_usr_ldscript does
something or not in the currently applicable environment.  See
in-source comments for more details.

Signed-off-by: Gregory M. Turner gmturner...@ameritech.net

diff --git a/eclass/toolchain-funcs.eclass b/eclass/toolchain-funcs.eclass
index 9b94512..7600aaf 100644
--- a/eclass/toolchain-funcs.eclass
+++ b/eclass/toolchain-funcs.eclass
@@ -604,6 +604,36 @@ gcc-specs-nostrict() {
 return $([[ ${directive/\{!fstrict-overflow:} != ${directive} ]])
 }

+# @FUNCTION: target-has-split-usr
+# @DESCRIPTION:
+# This function returns 0 (true) if, for the currently
+# targeted platform, portage is presuming that a split-usr
+# configuration must be maintained.  When target-has-split-usr
+# returns a nonzero (false) value, gen_usr_ldscript is a noop. This
+# may be handy in cases where an ebuild needs to decide, i.e., whether
+# it makes sense to move certain files from /usr/bin to /bin
+# in an ebuild.  In the future, the assumption that critical
+# stuff ends up in /bin and /$(libdir) whereas noncritical
+# stuff ends up in /usr/bin and /usr/$(libdir) may be removed
+# from portage entirely, or relegated to a legacy-support role.
+# As of April 2013, discussion was ongoing in the Gentoo developer
+# community as to what exactly to change and how (see bug #417451).
+# By predicating ebuild code on target-has-split-usr, which exists solely
+# to maintain a split-usr layout, ebuilds can future-proof themselves
+# against changes to the precise criteria that determine whether or not
+# split-usr is in effect for a given target and configuration.
+target-has-split-usr() {
+tc-is-static-only  return 1
+
+case ${CTARGET:-${CHOST}} in
+*-darwin*) return 0;;
+*linux*|*-freebsd*|*-openbsd*|*-netbsd*)
+use prefix  return 1
+return 0
+;;
+*) return 1 ;;
+esac
+}

 # @FUNCTION: gen_usr_ldscript
 # @USAGE: [-a] list of libs to create linker scripts for
@@ -620,19 +650,10 @@ gcc-specs-nostrict() {
 # the library (libfoo.so), as ldconfig should usually update it
 # correctly to point to the latest version of the library present.
 gen_usr_ldscript() {
+target-has-split-usr || return 0
 local lib libdir=$(get_libdir) output_format= auto=false 
suffix=$(get_libname)

 [[ -z ${ED+set} ]]  local ED=${D%/}${EPREFIX}/

-tc-is-static-only  return
-
-# Eventually we'd like to get rid of this func completely #417451
-case ${CTARGET:-${CHOST}} in
-*-darwin*) ;;
-*linux*|*-freebsd*|*-openbsd*|*-netbsd*)
-use prefix  return 0 ;;
-*) return 0 ;;
-esac
-
 # Just make sure it exists
 dodir /usr/${libdir}


--
gmt

Re: [gentoo-dev] eclass error-handling post-EAPI4

[Gregory M. Turner]

On 10/20/2012 4:07 PM, Zac Medico wrote:

On 10/20/2012 03:51 PM, Gregory M. Turner wrote:

Anyhow, given that eclasses can probably avoid auto-die() if they need
to, this leaves the issue of nonfatal allowing die to return to
callers... is there really any reason that die() needs to work this way?
  If helpers always used __helpers_die() instead of die(), then couldn't
die() be changed to ignore PORTAGE_NONFATAL?


Yes. It's a bug, as Ciaran said.


For anyone following this discussion, just a note that I've created a 
bug for this issue:


https://bugs.gentoo.org/show_bug.cgi?id=439356

A proposed patch is attached to the bug.

-gmt

Re: [gentoo-dev] eclass error-handling post-EAPI4

[Gregory M. Turner]

Thanks for your reply, but I still have some concerns about this.

On 10/19/2012 8:11 AM, Zac Medico wrote:

Regardless of EAPI, don't call the helpers that die in EAPI 4 unless you
want the function to die when the helpers fail, and use helper || die so
it behaves the same regardless of EAPI.


Took me a while, but I think I see why this is correct, now (mostly -- 
see below).  The source of my confusion was a mistaken assumption that 
die() would not respect PORTAGE_NONFATAL.  But isolated-functions.sh has 
roughly:


  die()
  {
  [[ ${PORTAGE_NONFATAL} == 1 ]]  {
  echo ${FUNCNAME[1]}: WARNING: $* 2
  return 1
  }
  .
  .

So -- correct me if I'm wrong -- this all works as follows:

  o helpers always handle errors by remembering the exit code of some
error-prone operation, and, if that exit code is nonzero, invoking
__helpers_die with some description of the problem, and finally
returning the remembered exit code.

In EAPI[0-3], this will dump the error text to stderr and return
the remembered exit code.

In =EAPI4, this will call isolated-functions.sh's die(), which will
swoop in and terminate everything before it has the chance to
go any further, effectively ensuring that __helpers_die never
returns -- /unless/ PORTAGE_NONFATAL is 1, in which case, it will
prepend __helpers_die: to the error message, but otherwise
behave as in EAPI[0-3].

  o die incantations in an ebuild or eclass will either directly or
indirectly call die() in isolated-functions.sh, which still works
as described above when used in these contexts -- which is to say,
that die is nonfatal when PORTAGE_NONFATAL is set.

  o PORTAGE_NONFATAL is only supposed to be manipulated by means of
the nonfatal API, which is only callable in =EAPI4, which means
that unless somebody cheated by directly messing with the
variable, in EAPI[0-3], die() never returns (like exit)


It's okay to die in older EAPIs, as long as you're not changing the

 behavior of a previously existing eclass function.

ACK


Previously existing eclass functions that only start to
die in EAPI 4 are okay, since they only start to die when the ebuild
developer bumps the EAPI.


ACK

If I indeed understand the facts correctly, I'm still a bit 
uncomfortable with your advice to just use helper || die in eclass 
code.  It seems to me that if I follow this recipe, =EAPI4 kinda works 
OK, but EAPI[0-3] doesn't.


Specifically, an EAPI[0-3] ebuild author will not have any means at his 
or her disposal to handle errors that occur in eclass utility functions. 
 The error-handling semantics of eclass utility functions would be like 
EAPI4, except without nonfatal around to provide relief from auto-die.


EAPI[0-3] conventions encourage the habit of writing ebuild code like 
normal bash script code, which is to say, that authors expect to be able 
to write code like


  invoke_fn || ... # handle error, probably by die()ing but maybe not

But eclasses that auto-die violate this expectation and create a 
situation where ebuild authors must be mindful of whether a particular 
function is a helper-function or an eclass utility function to correctly 
anticipate whether auto-death will occur.


This seems to be by design and may not be so horrible as I make it sound 
-- for example, it would be pretty weird to expect eclass code written 
before EAPI4 to suddenly start working differently in EAPI[0-3].


But should we really write new eclass code to behave this way as well?

Worse, eclasses authored in the pre-EAPI4 era ubiquitously assume that 
die() never returns, and do things like:


  efoo() {
  [[ ${EBUILD_PHASE} != prepare ]] 
  die efoo can only be used during src_prepare
  .
  .
  # stuff that isn't safe if ${EBUILD_PHASE} != prepare
  .
  .
  }

Since, in EAPI4, no means is provided to specify that a particular 
invocation of die() is unconditionally terminal, (except directly 
manipulating PORTAGE_NONFATAL, which doesn't seem to be encouraged), the 
only non-encapsulation-breaking non-EAPI-specific solution would be to 
re-code every presumed-terminal use of die() to look something like:


  failure_prone_action
  ret=$?
  if [[ ${ret} != 0 ]] ; then
  die Error message
  return ${ret}
  fi

I can imagine no reasonable way to avoid this without using aliases... 
or perhaps some construct like:


  do_or_die() {
  diemsg=$1
  shift
  $@ || { ret=$?; die ${diemsg}; return ${ret}; }
  }

  efoo() {
  do_or_die error message 1 \
  failure_prone_action1 arg1 arg2 arg3  \
  safe_action1 arg1  \
  safe_action2 arg1 arg2  \
  do_or_die error message 2 \
  failure_prone_action2 arg1 arg2  \
  .
  .

Hopefully I'm missing something here...?

-gmt

Re: [gentoo-dev] eclass error-handling post-EAPI4

[Gregory M. Turner]

On 10/20/2012 4:05 AM, Ciaran McCreesh wrote:

On Sat, 20 Oct 2012 03:52:49 -0700
Gregory M. Turner g...@malth.us wrote:

Took me a while, but I think I see why this is correct, now (mostly
-- see below).  The source of my confusion was a mistaken assumption
that die() would not respect PORTAGE_NONFATAL.


The source of your confusion is more the impression that there is such
a thing as PORTAGE_NONFATAL. You should be reading the spec, not code.


Seriously?  OK, let's try:

  http://dev.gentoo.org/~ulm/pms/4/pms.html#x1-1312.3.3.1

and

  http://dev.gentoo.org/~ulm/pms/4/pms.html#x1-13500012.3.3.6

do not comport with the actual behavior of portage.

Specifically, they very strongly suggest that nonfatal does not 
influence the behavior of die().  So depending on one's reading of the 
situation, either the PMS is misspecified or portage is broken.  Meanwhile,



http://dev.gentoo.org/~zmedico/portage/doc/ch05s03s05.html#package-ebuild-eapi-4-helpers-die-nonfatal

is dusting so much under the carpet that I can't possibly do my job as 
an eclass author without reverse-engineering, dry-run code analysis, or 
asking the SME's here on the mailing list how this is supposed to work 
in practice.


I've already asked the mailing list and so far the closest thing I have 
to an answer is Zac's response.  Although I appreciate his help, I've 
explained why his response doesn't fully address my concerns.  If you 
don't trust my dry-run analysis then we can try the reverse-engineering 
approach.  Give this ebuild a try:


8
# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI=4
DESCRIPTION=Foo
SLOT=0
KEYWORDS=~x86 ~amd64 ~x86-linux ~amd64-linux
IUSE=
DEPEND=
RDEPEND=${DEPEND}
utility_fn() { die uhoh; }
src_unpack() { mkdir -p ${S}; }
src_prepare() { nonfatal utility_fn; }
src_compile() { echo yaaay  foo.txt; }
src_install() { dodoc foo.txt; }
8

As you can see, my dry-run analysis appears to have been correct.

There is no documentation concerning best practices for eclass authors 
post-EAPI4, so I can't just RTFM -- there is no FM.  Furthermore, it 
appears that EAPI4 has broken a great many eclasses when invoked with 
the new nonfatal operator.  I guess I can just file a bug for that since 
so far nobody seems too concerned by it, but that still doesn't help me 
to write my eclass.


-gmt

Re: [gentoo-dev] eclass error-handling post-EAPI4

[Gregory M. Turner]

On 10/20/2012 11:16 AM, Zac Medico wrote:

On 10/20/2012 03:52 AM, Gregory M. Turner wrote:

EAPI[0-3] conventions encourage the habit of writing ebuild code like

   invoke_fn || ... # handle error, probably by die()ing but maybe not

But eclasses that auto-die violate this expectation and create a
situation where ebuild authors must be mindful of whether a particular
function is a helper-function or an eclass utility function to correctly
anticipate whether auto-death will occur.

should we really write new eclass code to behave this way as well?


Yeah, I think we should. If the eclass function is doing something that
might result in an undesirable die, then it can try to avoid the die by
first doing a sanity check. For example, it can check that a given file
path exists before passing it to the helper, and skip calling the helper
if the file path doesn't exist.


That seems reasonable -- in practice, I don't find myself needing to 
invoke helpers very much at all in eclasses, nor does it seem too 
terribly frequent in the existing eclasses (dodir is pretty common, but 
unlikely to fail unless something is seriously wrong anyhow).


Would it be feasible to lift the =EAPI4 restriction on nonfatal()?  I 
guess the problem would be that old portage's wouldn't have it.


Anyhow, given that eclasses can probably avoid auto-die() if they need 
to, this leaves the issue of nonfatal allowing die to return to 
callers... is there really any reason that die() needs to work this way? 
 If helpers always used __helpers_die() instead of die(), then couldn't 
die() be changed to ignore PORTAGE_NONFATAL?


It almost seems like the portage code was designed for nonfatal only to 
apply to helper invocations, and then the change to die() was added as 
an afterthought, to support the broader range of usages suggested 
(rather vaguely, I'd say) by the language in the PMS, ... yet somehow, 
the PMS ends up strongly suggesting -- I think most people would read 
the language as an explicit specification -- that ebuilds can rely on 
die() to terminate regardless of any usage of nonfatal.


-gmt

-gmt

[gentoo-dev] eclass error-handling post-EAPI4

[Gregory M. Turner]

I'm cooking up some eclass functions and it occurred to me that perhaps 
I had some responsibility regarding EAPI4's new error handling 
semantics.  After looking into it, it seems that, superficially, the 
answer to my question is no, the EAPI4 changes only apply to ebuild 
helpers.


Even so, this got me thinking: if ebuild helpers are going to warn(*) in 
EAPI4 and die in EAPI=4 (unless somebody used nonfatal), then, those 
error-handling semantics are going to bubble up through my eclass to its 
heirs (assuming my eclass uses helpers).


The problem I see with that is that a consumer of my eclass function is 
effectively required to anticipate these EAPI helper-function outcomes, 
which is fair-enough if s/he's consuming them directly, but pretty 
unreasonable if they're just consuming my eclass  In effect, this 
seems to create a situation where they can't know what to expect, 
without wading through my code, unless I go in and document in EAPI4+ 
it does X, but in EAPI3, it does Y.  Which, let's face it, most eclass 
authors probably won't get around to doing.


For example, suppose poop.eclass has a function epoop which performs 
the following steps:


  o Downloads the latest post from the
http://www.unnecessaryquotes.com/ website using wget
  o Strips out the top 3 posts from piped wget output using
a bunch of sed and awk magic.  Saves each post-processed post as
${S}/$(mktemp -u $( printf %100s |tr   X )).xml and
remembers these filenames in three hard-coded variables.
  o Installs the posts as documentation using dodoc statements
  o Returns $(( $(date +'%s') % 3 ))

Let's say I started writing this code before I knew anything about EAPI 
4, and the current epoop documentation states that:


 o errors in the wget and postprocessing steps will result in a return
   code of '3'
 o errors from dodoc will result in a return code of '3' + the
   return-code of dodoc
 o epoop() succeeds every third second and otherwise fails, returning
   the number of seconds ago that the API could have been invoked
   successfully as the error code (WTF: this is tounge-in-cheek:
   $(( $(date +'%s') % 3 )) is the current UNIX time in
   seconds, modulo 3, an extremely pointless and arbitrary success
   criterion for my hypothetical function that will cause it to
   randomly fail 66% of the times it's invoked and runs to completion.
   If you don't find this amusing, that's OK; it's not terribly
   important to the matter at hand.)

Well, now I have an awkward situation.  In EAPI's 0-3, the function 
works as documented.  In EAPI 4+, however, epoop() will die if any error 
occurs in dodoc.  I can fix this by replacing the dodoc code -- let's 
say it originally was coded like:


  dodoc ${docfile1} || { r=$?; eerror dodoc1; return $((r+3)); }
  dodoc ${docfile2} || { r=$?; eerror dodoc2; return $((r+3)); }
  dodoc ${docfile3} || { r=$?; eerror dodoc3; return $((r+3)); }

-- with code like:

  .
  .
  if has ${EAPI:-0} 0 1 2 3 ; then
  nonfatal dodoc ${docfile2}
  else
  dodoc ${docfile2}
  fi || { r=$?; eerror dodoc2; return $((r+3)); }
  .
  .

but I will have to do something like this for each of the three dodoc 
statements.  Seems like a real PITA.


Obviously, I've chosen a very silly example, and perhaps I've 
cherry-picked a solution that illustrates my point -- but it's not hard 
to imagine having to make fairly significant changes to an eclass in 
order to preserve its pre-EAPI4 semantics in EAPI4+.


The conclusion I'm tempted to draw is that it makes more sense to bring 
my eclass error-handling behavior into line with the EAPI-specific 
behavior of helper-functions than it does to gum up helper usage in my 
eclass with a bunch of EAPI-specific error-handling spaghetti-code.


i.e., returning to epoop: instead of the above solution, I might leave 
the dodoc code alone, and instead implement something vaguely like 
__helpers_die, i.e.:


  # note: stupidly clever -- not how i'd implement this irl
  _poop_die_maybe() {
  local ret=$? eapi=${EAPI:-0}
  eapi=${eapi::1}
  (( (3-eapi)*(PORTAGE_NONFATAL-1)  0 ))  die $@
  eerror $@
  return ${ret} # pass through pre-invocation $? unchanged
  }

With this in place, it's not terribly difficult to bring the 
error-handling/EAPI semantics of epoop() into line with those of the 
helper functions across all EAPIs.  For example, where we had


  return $(( $(date +'%s') % 3 ))

we might now put

  ret=$(( $(date +'%s') % 3 ))
  (( ret == 0 )) || \
  _poop_die_maybe epoop: invoked ${ret} seconds too late
  return ${ret}

Or, if it was the last statement in the epoop function, just:

  ( exit $(( $(date +'%s') % 3 )) ) || \
  _poop_die_maybe epoop: invoked $? seconds too late

I hope my gist isn't getting lost in the particulars of this silly 
example.  My point is: EAPI4 changes the behavior of eclasses if those 
eclasses consume helper functions, which could lead to some pretty 

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/14/2012 9:29 PM, Mike Frysinger wrote:

On Sunday 14 October 2012 04:49:28 Gregory M. Turner wrote:

Thirdly has been addressed ad nauseam in this thread and will be
solved by prepending the LDFLAG rather than appending, or, preferably,
by patching autotools (but only if I can find a simple, low-maintenance
approach that is likely to work without building any new per-platform
matrices or case-statements).


i'm fairly certain this isn't autotools.  i've poked around the python build
system before in the past and while it uses autoconf to do platform tests, it
doesn't use automake/libtool.  make is used to bootstrap python, and then they
descend into a horrible hack of a custom build system written in python.  i
suspect much of the pain you're seeing is coming from that last part.
-mike


I guess I should say in src_prepare rather than in autotools.

Specifically, I was thinking some kind of configure.in patch might be 
good, since configure.in seems to churn less than Makefile.in, a good 
thing if I want to produce a one-size-fits-all patch.


And, yeah, setup.py is definitely behind all this yucky.

Python clearly has an amazing community, so I hate to say anything 
negative... but I sometimes wish they would build less and buy more.


Anyhow, as everyone knows, bitching and moaning about FOSS is pointless 
so I'll stop there and spend my time fixing stuff instead :)


-gmt

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/12/2012 4:03 AM, Gregory M. Turner wrote:

First, something puts in all kinds of inappropriate amd64 multilib paths
(this ends up being harmless as wrong-arch libraries get rejected at
link-time and treated as non-matches for -lclauses... still, WTF?).

Secondly, something puts the built-in ld search paths into the
command-line ld search path (a practice which has been roundly
disparaged in this thread, and correctly so, IMO).

Thirdly, prefix x86-linux also clearly suffers from the original problem
I was seeing in cygwin, and which inspired this thread (putting -L.
after $(libdir) paths and therefore linking against the wrong
libpython.so).  I never suspected this applied to every prefix python
ebuild, but that's what the above seems to suggest... not cool.


Just wanted to update this thread with latest current triage and 
prognosis data for this patient:


Regarding etiology, for Firstly and Secondly above: it seems that 
these are by design (which is not to say that they should be left: in 
each case, the design is pretty bad and ought to be fixed, I think).


My hope is that Firstly and Secondly are needed only to accommodate 
prefix bootstrap, in which case we can fix them by detecting whether a 
prefix bootstrap is ongoing or not, and conditioning these hacks on that 
(improving Firstly to not be braindead would be a nice touch, although 
doing so in a way that doesn't break anything may be prohibitively 
difficult or beyond my abilities).


Thirdly has been addressed ad nauseam in this thread and will be 
solved by prepending the LDFLAG rather than appending, or, preferably, 
by patching autotools (but only if I can find a simple, low-maintenance 
approach that is likely to work without building any new per-platform 
matrices or case-statements).


Note that, even without this third fix, absent a PEBKAC placement of 
-L${EPREFIX}/usr/$(libdir) into LDFLAGS, either:


o it's a prefix bootstrap, in which case there probably is no
  ${EPREFIX}/usr/$(libdir)/libpython${PV}.so, or

o it's /not/ a prefix bootstrap, in which case the above-mentioned
  fixes would prevent the -L${EPREFIX}/usr/$(libdir) from
  getting into LDFLAGS, thus preventing the problem from arising.

So by fixing Firstly and Secondly, we have at least brought things 
sufficiently in line with expectations that we are only fixing 
user-level configuration mistakes instead of our own, as originally 
expected. **


I'm feeling ready, at this point, to file bugs for these issues; once I 
do, I'll post bug#'s into this thread and plan to take all off this 
chatter off-list, unless something more comes up that seems to require 
community review.


Thanks for everyone's help figuring this stuff out,

-gmt

** I suppose a third possibility is that someone is doing an in-place 
re-bootstrap or restarting a bootstrap build that failed mid-merge, 
but ... doesn't matter, it'll be academic once we fix the bug.

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/11/2012 2:40 PM, Marien Zwart wrote:

I'm going to do something potentially rude and comment on this without
having read the entire thread.

On Thu, Oct 11, 2012 at 10:39 PM, Gregory M. Turner g...@malth.us wrote:

Anyhow one thing I have figured out is how things can work correctly on
Linux wihtout -L.: on Linux, the python plugins aren't actually linked
against libpython.so!


Python can be built with and without a shared library
(libpythonx.y.so). If the shared library is built the interpreter
executable is linked against it (of course), and normally so are the
extension modules. This is useful because it makes the linker do the
right thing for applications that embed python (and end up loading
libpythonx.y via dlopen). If the shared library is not built extension
modules cannot link against it, and the python executable itself
exports the relevant symbols. So on a normal shared python build
extensions should end up linked against libpythonx.y.so.

However, at least one popular distro and presumably most of its
derivatives (Debian) has a system python built without the shared
library, plus the shared library built separately, for use by
applications that embed python. I am not entirely sure why they still
do this (if I understand correctly it was originally done for
performance reasons, but I'm not sure if they've checked those are
still valid). Their extension modules cannot link to libpythonx.y:
their python interpreter executable provides the same symbols
libpythonx.y provides, so loading such an extension module into the
regular interpreter would result in collisions.

The reason I mention this is Gentoo's python is built with the shared
library, so on Gentoo your extensions *should* end up linked to
libpythonx.y (if they use a more or less standard build system, at
least). If yours are not I'm curious what happened, and would
appreciate it if you gave me some more information off-list (marienz
on freenode or email).


Thanks a ton for this info, that explains a lot.  Actually, got this 
notion testing outside of Gentoo, building from a vanilla python 3.3.0 
tarball on a Fedora box.  I wanted to see what happens in a completely 
Gentoo-free environment so as to get some kind of baseline 
understanding of how upstream is propagating stuff through the Makefiles 
to setup.py.


Probably, I screwed up ./configure (I tried to approximate Gentoo's 
arguments), and requested a non-shlib build.  Guess it would have been 
clearer if I'd mentioned that in my post -- sorry if I worried you :)


I do, of course, intend to try it on my main Gentoo ~amd64 -- if I ever 
see it fail to do as you describe, I'll let you know, Marien.


My i686 prefix is up to date and I can confirm that it is linking 
against libpython there.


But... wow.  In my almost-vanilla i686 cross-prefix (hosted by my 
only-moderately-rice multilib ~amd64 workstation), in 
=dev-lang/python-2.7.3-r2, the modules are linked like so:


  i686-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 \
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-Wl,-O1 \
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-fno-strict-aliasing \
-O2 -march=i686 -pipe -m32 \
-fwrapv -DNDEBUG \
-I. -IInclude -I./Include \
-I${EPREFIX}/usr/include \

build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/mathmodule.o
 \

build/temp.linux-i686-2.7${EPREFIX}/var/tmp/portage/dev-lang/python-2.7.3-r2/work/Python-2.7.3/Modules/_math.o
 \
-L${EPREFIX}/lib -L${EPREFIX}/usr/lib \
-L/usr/lib32 -L/usr/lib64 -L/lib32 -L/lib64 \
-L. \
-lm -lpython2.7 \
-o build/lib.linux-i686-2.7/math.so

Which is just shameful.  In this case, I'm pretty sure all those 
-L{${EPREFIX},}{/usr,}/$(libdir) clauses are not coming from me:


  [PREFIX] $ emerge --info --verbose | grep /lib
  sys-devel/libtool:2.4.2::gentoo_prefix
  COLLISION_IGNORE=/lib/modules/* *.py[co]
  PKG_CONFIG_PATH=${EPREFIX}/usr/lib/pkgconfig:$( :;
  )${EPREFIX}/usr/share/pkgconfig
  PORTAGE_BIN_PATH=${EPREFIX}/usr/lib/portage/bin
  PORTAGE_PYM_PATH=${EPREFIX}/usr/lib/portage/pym
  UNINSTALL_IGNORE=/lib/modules/*

(snippets edited for readability obv).

Regarding the gcc-as-linker invocation above:

First, something puts in all kinds of inappropriate amd64 multilib paths 
(this ends up being harmless as wrong-arch libraries get rejected at 
link-time and treated as non-matches for -lclauses... still, WTF?).


Secondly, something puts the built-in ld search paths into the 
command-line ld search path (a practice which has been roundly 
disparaged in this thread, and correctly so, IMO).


Thirdly, prefix x86-linux also clearly suffers from the original problem 
I was seeing in cygwin, and which inspired this thread (putting -L

Re: [gentoo-dev] Proposal: removing server profile variants from profiles.desc

[Gregory M. Turner]

On 10/11/2012 3:31 PM, Ben Kohler wrote:

There are other ways to achieve a lighter system, but that's not really
what this is about.  The server profiles are not any lighter than the base
profiles.

To those in favor of keeping some kind of server profile around, how
would it differ from the base profile?  What would you enable or disable on
top of the base?  I am pretty sure that the current USE=-perl -python snmp
truetype xml is not what any of you would suggest.

In my opinion, removing /usr/portage/profiles/targets/server/make.defaults
and having the server target apply nothing over the base profiles, and
then dropping the warning from the server profiles, would be a better
situation than where we are now.

-Ben


That seems pretty reasonable.

Another possibility, I suppose, would be to combine Rich's idea with 
yours and get:


  base - minimal - server (noop)

Also super-easy if we just build the above cascade, take what's in 
server, move it into minimal, git rid of the awful warning, and call it 
fixed.


At least by clarifying what is meant by server, we might inspire 
someone to contribute more enhancements later.


Kind of the opposite of your original proposal, Ben :P

At least we've established broad consensus that the warning should be 
silenced.


-gmt

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/10/2012 9:14 PM, Mike Frysinger wrote:

On Wednesday 10 October 2012 23:37:26 Gregory M. Turner wrote:

(1) is worse than (2), but it does have some quasi-legitimate usages.
For example, prefix bootstrap does this (or used to), as do many of the
crossdev-wrapper scripts.  I've also resorted to such usage, myself,
when repairing a prefix after I've broken its compiler.  These cases
don't really seem completely correct or incorrect -- about the best
I can say for them it is that they are mostly efficacious, but prone to
problems.


and no, crossdev doesn't do this.  it is properly sysrooted.


You are right.  I guess I was thinking of this:

http://tinderbox.x86.dev.gentoo.org/misc/cmerge,

Which does use -L but looks pretty ancient.  I see now that crossdev's 
equivalent hacks up the linker scripts -- which seems as correct as 
one could reasonably hope for... pretty sexy.



further, you cannot do multilib with users putting -Lsystem libdir into
their LDFLAGS.  they just forced a single ABI and any other will simply fail.
the toolchain's compiler driver knows exactly where to find its own libraries.
if it doesn't, it's broken, and it should be fixed.


ACK


So technically speaking, LDFLAGS=-L. ${LDFLAGS} or similar is not a
complete abomination.  Still... I don't like it.


it's significantly better than the proposed code which tries to parse the
meaning of various flags and insert it at the right point.  i can't see it
being useful at all.  it's a recipe for fragility and being unmaintainable.


I think I meant the above differently than you interpreted it: I meant 
that solving library-search-path selection in autotools steps and/or 
Makefile.in would make the question of how to manipulate LDFLAGS 
academic, and seems more elegant.  Apparently you agree:



If the Makefiles are building against libraries expected to be in
${PWD}, it seems to me that the Makefiles should know to look there
automatically.


yes, so why is the ebuild specifying -L. for it ?  the package should already
have a -L to the right place, and if it wants to make sure it gets used before
the user's LDFLAGS, it should show up in the linking before it (or have the
build system prepend the value to LDFLAGS if it's appending currently).


If we decide that manipulating LDFLAGS is correct, whether we do this 
using my parser or LDFLAGS=-L. ${LDFLAGS} isn't particularly important.


My point above was that coming to that decision in the first place 
suggests some underlying issue exists that arguably ought to be fixed 
instead.



i'm not sure why this applies to the larger build system.  i can understand
that the python build itself might not be putting the -L. in a place where a
user's misconfigured LDFLAGS will cause a problem.


This only pertains to dev-lang/python AFAIK... there may be others 
lurking out there somewhere, but I'm not too worried about them.  The 
use-cases I have in mind are prefix bootstrap and system-recovery when 
gcc breaks.


In general, if python breaks, folks can end up in pretty bad shape due 
to chicken-and-egg issues with portage.  That's the only reason I've 
spent so much effort trying to think this fairly obscure corner-case 
through.


-gmt

P.S.: After poking around, the first commit where this appeared is this 
one: 
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-lang/python/python-2.5.1.ebuild?hideattic=0r1=1.4r2=1.5view=patch, 
a mere 5.5 years ago by uberlord.


The cvs ChangeLog references bug 177426 which doesn't seem particularly 
illuminating.  It also says:


  export LDFLAGS=-L. so we link modules correctly on FreeBSD
  and possibly other systems where python2.5 isn't installed yet.

Which at least seems to confirm that this was originally a problem not 
of search path ordering, but of complete failure to search ${S} in the 
first place.


Meanwhile, I've looked into the patches a bit.  In regular Gentoo 
nothing special happens to LDFLAGS, but in prefix, there is special 
handling that suggests some avenues of attack.  Before I do anything 
further, I want to figure out if it works right without LDPATH 
manipulation on linux, and, if so, whether I can't generalize whatever 
mechanism allows that to happen.

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/11/2012 8:50 AM, Mike Frysinger wrote:

On Thursday 11 October 2012 05:35:21 Gregory M. Turner wrote:

On 10/10/2012 9:14 PM, Mike Frysinger wrote:

it's not particularly important, but on one hand, the LDFLAGS parsing logic
should not be in the tree ever.


I've no major attachment to it.  Took all of five minutes to code up 
reading the ld manpage, and as you pointed out, it's probably solving a 
non-problem.



on the other, i can stomach much smaller one-
off hacks like prepending -L. to LDFLAGS if the maintainers of the python
ebuild really really want to add it.


If fixing the python builds proves too onerous then this is what I'll 
end up filing a bug for.



my [limited] understanding of the prefix compiler is that they wrap things with
a custom shell script to inject -L flags behind the back of the compiler at the
very last possible minute and point to the right place.


Pretty fuzzy on this myself.  Whatever binutils-config does works so 
well I've never had to look into it very deeply :)


Above, however, I'm referring to the prefix-specific cpython patch-sets, 
where I was sure I'd seen patches to project LDFLAGS into setup.py's 
compiler invocation tables.  But it seems I was mistaken.  Perhaps it 
was in the main gentoo-patches after all... or maybe I just need more 
coffee...


Anyhow one thing I have figured out is how things can work correctly on 
Linux wihtout -L.: on Linux, the python plugins aren't actually linked 
against libpython.so!


With any luck, this explains what's going on and suggests that platforms 
linking python modules against libpython.so just need to wedge -L${S} 
somewhere in configure.in, possibly do-able via sed script (unless we 
end up having to mess with setup.py, in which case, things may get a bit 
more complicated).


-gmt

Re: [gentoo-dev] Proposal: removing server profile variants from profiles.desc

[Gregory M. Turner]

On 10/11/2012 1:04 PM, Walter Dnes wrote:

On Thu, Oct 11, 2012 at 03:22:17PM -0400, Mike Frysinger wrote


sounds like something to fix rather than punt.  i don't know why
you think having server profiles is undesirable, but i certainly
desire it on many systems.  like servers.  the desktop and developer
profiles are not appropriate.


If you want a light
profile, I suggest doing what I do... start your USE variable in
make.conf with -*, and add any flags you need, either in package.use or
in make.conf.


popcorn

-gmt

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/6/2012 1:31 AM, Fabian Groffen wrote:

On 06-10-2012 00:47:57 -0700, Gregory M. Turner wrote:

In dev-lang/python*, we use

append-ldflags '-L.'

to ensure linking is performed against the built libpython.so in-tree,
rather than than in the one in $(libdir).  But, this doesn't work if
LDFLAGS contains -L$(libdir).

We could try to fix this like:

   export LDFLAGS=-L. ${LDFLAGS}

or so.  That would cover 99.9% of the cases out there.  But very rarely,
indiscriminately placing our '-L.' before every other clause in LDFLAGS
might cause an unanticipated side-effect.

I think it would make more sense in this case to just add one more patch
to Python, such that the build-system just inserts it.  I recall it's
necessary at least on FreeBSD, Darwin, Solaris, but I don't recall any
more why it works/worked on Linux fine.


I was thinking along these lines as well.  As it turned out, however, I 
was not able to find an automagical, PMS-compliant, non-crazy way to do 
it that resolved my test-case (more on that below), without effectively 
hooking econf().


Literally hooking a core API seems like a Pandora's box better left 
unopened, so this is what I came up with (note: the fpectl stuff was 
also in my overlay and the patches got entangled -- rather than editing 
the hunk by hand, I just left it.  It's orthogonal to this issue, 
however, and for present purposes can be ignored):


--8-
--- PORTAGE/dev-lang/python/python-2.7.3-r2.ebuild
+++ OVERLAY/dev-lang/python/python-2.7.3-r2.ebuild
@@ -218,13 +255,6 @@
# http://bugs.python.org/issue15506
export ac_cv_path_PKG_CONFIG=$(tc-getPKG_CONFIG)

-   # Set LDFLAGS so we link modules with -lpython2.7 correctly.
-   # Needed on FreeBSD unless Python 2.7 is already installed.
-   # Please query BSD team before removing this!
-   # On AIX this is not needed, but would record '.' as runpath.
-   [[ ${CHOST} == *-aix* ]] ||
-   append-ldflags -L.
-
local dbmliborder
if use gdbm; then
dbmliborder+=${dbmliborder:+:}gdbm
@@ -248,11 +278,18 @@
 myconf=${myconf} --enable-framework=${EPREFIX}/usr/lib \
|| myconf=${myconf} --enable-shared

+   if [[ ${CHOST} == *-cygwin* ]] ; then
+   fpeconfig=--without-fpectl
+   myconf=${myconf} ac_cv_func_bind_textdomain_codeset=yes
+   else
+   fpeconfig=--with-fpectl
+   fi
+
# note: for a framework build we need to use ucs2 because OSX
# uses that internally too:
# http://bugs.python.org/issue763708
-   OPT= econf \
-   --with-fpectl \
+   OPT= cpython_econf \
+   ${fpeconfig} \
$(use_enable ipv6) \
$(use_with threads) \
 		$( (use wide-unicode  use !aqua)  echo --enable-unicode=ucs4 
|| echo --enable-unicode=ucs2) \

--- PORTAGE/eclass/python.eclass
+++ OVERLAY/eclass/python.eclass
@@ -401,6 +401,64 @@
fi
 }

+# see http://thread.gmane.org/gmane.linux.gentoo.devel/80633/focus=80635
+_python_prepend_cwd_ldpath() {
+   local new=()
+   local f
+   local done=no
+   for f in ${LDFLAGS} ; do
+   case ${f} in
+   -Tbss=*|-Tdata=*|-Ttext=*|-Ttext-segment=*)
+   new+=( ${f} )
+   ;;
+   -L*|-T*|--library-path*|--script*)
+   if [[ ${done} == yes ]] ; then
+   new+=( ${f} )
+   elif [[ ${f} == -L. ]] ; then
+   # if it's somehow already there, don't 
duplicate it
+   new+=( -L. )
+   done=yes
+   else
+   new+=( -L. ${f} )
+   done=yes
+   fi
+   ;;
+   *)
+   new+=( ${f} )
+   ;;
+   esac
+   done
+   [[ ${done} == no ]]  new+=( -L. )
+   export LDFLAGS=${new[*]}
+}
+
+# @FUNCTION: cpython_econf
+# @DESCRIPTION:
+# econf() substitute for use in dev-lang/python ebuilds
+#
+# On some platforms, it is neccesary to prepend -L. to ldpath before
+# proceeding with python's configure process.  Using cpython_econf()
+# instead of econf() will ensure that this is taken care of correctly
+# before python's configure step can proceed.  This is not needed for
+# any python ebuilds other than dev-lang/python; any other ebuild
+# calling this function will receive an error.
+cpython_econf() {
+   if [[ ${CATEGORY}/${PN} != dev-lang/python ]] ; then
+   die cpython_econf can only be used by dev-lang/python ebuilds
+   fi
+   # econf will enforce ${EBUILD_PHASE} requirements, so we don't bother

Re: [gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

On 10/9/2012 2:26 PM, Mike Frysinger wrote:

On Saturday 06 October 2012 03:47:57 Gregory M. Turner wrote:

My god, I am a horrible self-editor.  Sorry.  Please ignore the magnum
opus above and allow me to try again.

In dev-lang/python*, we use

append-ldflags '-L.'

to ensure linking is performed against the built libpython.so in-tree,
rather than than in the one in $(libdir).  But, this doesn't work if
LDFLAGS contains -L$(libdir).


well, some notes here ...

for users: putting -Lsome-system-path is wrong.  it changes the meaning of
system paths when it comes to searching for linking by elevating it to user
specified path.  imo, you should be fixing the source rather than the symptom.

for packagers: using -L$(libdir) is almost always wrong and pointless.  the
toolchain itself knows the proper system path to search for libraries to link
against, and when you cross-compile, the --libdir you specify to configure is
not going to be valid when dereferenced on the build system.


Agreed on these points.  Not until we combine the doubly broken 
circumstances of (1) User specified -L/usr/$(libdir) in LDPATH and (2) 
an ebuild that puts -L. into LDPATH to fix Makefile recipes that appear 
more-or-less broken, do we get this conflict.  This leads to a certain 
odor of unsupportable circumstances which, I must admit, makes me 
slightly uncomfortable trying to fix it.


Breaking this down:

(1) is worse than (2), but it does have some quasi-legitimate usages. 
For example, prefix bootstrap does this (or used to), as do many of the 
crossdev-wrapper scripts.  I've also resorted to such usage, myself, 
when repairing a prefix after I've broken its compiler.  These cases 
don't really seem completely correct or incorrect -- about the best 
I can say for them it is that they are mostly efficacious, but prone to 
problems.


As for (2)... meh.  LDFLAGS, to my thinking, means something like:  Use 
these guidelines and parameters for linking, in preference to the 
default guidelines that the build process would normally use.  LDFLAGS 
coming from make.conf have a slightly different meaning, I guess, but 
the gist of it is the same.


So technically speaking, LDFLAGS=-L. ${LDFLAGS} or similar is not a 
complete abomination.  Still... I don't like it.


If the Makefiles are building against libraries expected to be in 
${PWD}, it seems to me that the Makefiles should know to look there 
automatically.


In the particular case of Gentoo's cpython, I have to admit I'm 
concerned that if I start mucking around with the Makefiles/autotool 
scaffolding, I'm going to break stuff.  But maybe it's worth taking that 
risk if it allows us to do things more correctly going forward... I 
could at least take a crack at it and see what people think of the results.


-gmt

[gentoo-dev] Re: [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

My god, I am a horrible self-editor.  Sorry.  Please ignore the magnum 
opus above and allow me to try again.


In dev-lang/python*, we use

  append-ldflags '-L.'

to ensure linking is performed against the built libpython.so in-tree, 
rather than than in the one in $(libdir).  But, this doesn't work if 
LDFLAGS contains -L$(libdir).


We could try to fix this like:

 export LDFLAGS=-L. ${LDFLAGS}

or so.  That would cover 99.9% of the cases out there.  But very rarely, 
indiscriminately placing our '-L.' before every other clause in LDFLAGS 
might cause an unanticipated side-effect.


The flag-o-matic patch in my previous post analyses LDPATH and finds the 
first statement that might possibly cause the ld command-line 
library-search-path to be impacted, and inserts the -L. just before 
that statement, hopefully achieving the desired result with the least 
possible side-effects.


I wonder: do people feel this distinction is meaningful enough to 
justify actually making it?  Or am I just being an anal-retentive nut-job?


If you tried to read my first post, thanks for not offing yourself like 
the guys who get stuck sitting next to Ted Striker in _Airplane!_


-gmt

[gentoo-dev] [PATCH/RFC] eclass/flag-o-matic.eclass: prepend-ldpath

[Gregory M. Turner]

[ eclass/flag-o-matic.eclass ]-8-
--- PORTAGE/eclass/flag-o-matic.eclass
+++ OVERLAY/eclass/flag-o-matic.eclass
@@ -117,6 +117,42 @@
return 0
 }

+# @FUNCTION: prepend-ldpath
+# @USAGE: path
+# @DESCRIPTION:
+# Place the specified ldpath into LDFLAGS before any options which could
+# add additional paths to ld's search path.  Specifically, it will place
+# the new path /foo into LDFLAGS as -L/foo just before the first
+# occurance matching any of the globs: '-L*', '-T', '--library-path*',
+# and '--script*', but not matching any of the globs: '-Tbss=*',
+# '-Tdata=*', '-Ttext=*', and '-Ttext-segment=*'.  If no such match is
+# found, then this is equivalent to append-ldflags -Lpath.
+prepend-ldpath() {
+   local new=()
+   local f
+   local done=no
+   for f in ${LDFLAGS} ; do
+   case ${f} in
+   -Tbss=*|-Tdata=*|-Ttext=*|-Ttext-segment=*)
+   new+=( ${f} )
+   ;;
+   -L*|-T*|--library-path*|--script*)
+   if [[ ${done} == yes ]] ; then
+   new+=( ${f} )
+   else
+   new+=( -L${1} ${f} )
+   done=yes
+   fi
+   ;;
+   *)
+   new+=( ${f} )
+   ;;
+   esac
+   done
+   [[ ${done} == no ]]  new+=( -L${1} )
+   export LDFLAGS=${new[*]}
+}
+
 # @FUNCTION: filter-lfs-flags
 # @DESCRIPTION:
 # Remove flags that enable Large File Support.
-8-

I think my code is probably fine, or if it's buggy, so be it.  A prior 
question is: does this have sufficient utility?


I need (not exactly... keep reading) something like this for python. 
Currently python*.ebuild contains:


  append-ldflags -L.

which is required in order to pull in the newly-rebuilt 
libpython${VERSION}.so at ebuild-time, instead of the system version 
(perhaps it's only so on obscure platforms?).


The problem is, LDFLAGS may already have some library paths coming in 
from the environment.  In this case, we end up with something like:


  LDFLAGS=-Wl,--engage-warp-engines -L/random/prefix/usr/lib -L.

and python goes ahead links everything against, i.e.:

  /random/prefix/usr/lib/libpython3.2.so,

or whatever is the version in question, so we achieve precisely nothing, 
effecting the outcome we are trying to prevent (the problem manifests in 
subtle, hard-to-diagnose ways, unfortunately).


In general, when we say append-ldflags -Lsomething, we may or may not 
mean always use something/libfoo, not $(libdir)/libfoo to link.  In 
the case where we do mean always, append-ldflags is a broken approach.


An alternative would be to just replace

  append-ldflags -L.

with

  export LDFLAGS=-L. ${LDFLAGS}.

however, there are problems with this:

First, it's aesthetically annoying that this spreads out the -L* 
arguments into two different zones in LDFLAGS.  No big deal, but it 
definitely doesn't make visually scanning the logs any easier.


Second, although I'm not aware of any ld arguments which act as 
modifiers to -L arguments (except the other -L arguments and 
ldscript arguments, which the above patch scans for), ld does use 
order-dependent patterns for other arguments.


For example: -lfoo -( -lbar -lbaz -) is not the same thing as -( 
-lfoo -lbar -) -lbaz.  So order-dependencies might emerge in some 
future binutils... in that case we'd still probably need to extend the 
case statement above to include the new order-affecting arguments, but 
at least we'd have a place in which to do so.


Third, although the meaning of -L* options may not be affected by other 
arguments in an order-dependant manner, the reverse is not so.  For 
example, see --mri-script, --script (which both can affect and be 
affected by the command-line library search path in an order-dependent 
manner!), -rpath-link (SunOS only), etc...


One could certainly argue that, in practice, points two and three amount 
to hopelessly obscure nitpicking and that nobody in their right mind 
should be relying on this type of stuff in their LDFLAGS.


I'm not sure I'd state it quite so strongly as that, (after all these 
might come just from profile.bashrc and be targeted to a particular 
ebuild) but justification is clearly on the thin side.


Indeed, if we're going to worry about side effects, it's not entirely 
clear that what what my patch does is safe.  For example, if the 
environment supplied -L/foo/bar --script baz, and python for some 
reason had a baz file in ${S}, then some kind of breakage would 
surely happen when we did


  prepend-ldpath .

Also, we might also legitimately worry that the presence of this 
function in flag-o-matic will somehow encourage people to do this 
more, and, I'd have to agree that it's a 

Re: [gentoo-dev] CVS - git, list of where non-infra folk can contribute

[Gregory M. Turner]

Brian Harring wrote:

1) We need a thin manifest - thick manifest converter.  Thin
manifests are used for git- they store just DIST entries.  Thick (also
known as 'full'), are what cvs/rsync users are familiar with- it holds
checksums for all content.

carebear is the current person volunteering to sort this
(help may be appreciated, talk to him/her/it).


heh :)

I'll read up, spend some time on IRC, and see what I can do to help here.



replay it into git via tailor;



Never knew about that tool... not sure about the wisdom of adding an 
extra moving part just to keep the lights on for those few hours... 
Given the 2G of history issue Diego mentioned, which if I understand 
correctly, effectively means that the future gentoo git can never rebase 
its commit history, why chance it?


In my last experience with cvs-git (at the time I was building a rsync 
(binutils cvs)-git mirror for a client), the most difficult thing about 
cvs-git was trying to scrub the identity data.


I don't remember the exact issue, but somehow, git had identity 
uniqueness constraints that cvs happily ignored, or something like that. 
 I never thought to try using svn as an intermediate -- but I like that 
idea a lot and wish I had thought of it when I needed to.


Anyhow, wrong ml for this, I'll subscribe to -scm.

-gmt

Re: [gentoo-dev] CIA replacement

[Gregory M. Turner]

On 10/1/2012 10:29 AM, Rich Freeman wrote:



Looking at the tracker [1], we need a pre-upload hook (I'm not quite
sure why), an rsync conversion script, the ability to validate the
converted tree, and documentation.  There is still an open bug for
commit signing, and I'm not quite sure why as this was implemented.


a job for gitolite? what does ogo use?


Am I missing some big concern that just isn't obvious in these bugs?


Things will break.  There will be much wringing of virtual hands. But in 
1 weeks' time, everyone will get over it and find ways to fix whatever 
broke.  There will be one less thing on the big to-do list, and most 
people will be happier overall.


It's one of those Social Security type of things it's hard to build a 
clean consensus around /when/ to make a break like this.  I think we 
basically need a headstrong asshole in a position of sufficient 
authority to say, fuck everyone, we are doing this, here is the 
changeover date.


Too much democracy, tactical planning and consensus-building will almost 
certainly perpetuate the status quo.


Look, Gentoo will probably fall to pieces for a few days.  The 
alternative, IMO, is a huge, elaborate multi-phase plan of some kind, 
like how Microsoft or NASA would do it: armies of 40-somethings toil for 
months or years in cubicles...  Elaborate Gantt charts and process 
diagrams are drawn up and approved by layer after layer of middle 
management.


Gentoo can tolerate a hiccup; it's survived way scarier challenges.  We 
have a huge community of very talented people highly motivated to keep 
this machinery working... there's literally zero chance that Gentoo will 
somehow have it's spirit broken by cvs-git.


In other words, devs -- imo -- just go for it.  Throw caution to the 
wind.  Break some bylaws if need be.  Ignore anyone saying we're not 
ready.  It'll be easier to get forgiveness than permission, and exactly 
what process issues needed to be smoothed over will frankly only truly 
be clear in the aftermath.


Of course, that's easy for me to say since I'm not the one whose door 
everyone will be beating down when TSHTF.  There are worse things than 
CVS (Visual SourceSafe comes to mind).  But if we don't bite the bullet 
now, we all know we'll be having this discussion again later.


-gmt

Re: [gentoo-dev] Discussing stuff that is not appropriate to discuss

[Gregory M. Turner]

If you're going to paint me and the other folks expressing opinions as 
entitled mouth-breathers, certainly you can't expect not to hear any 
reply because it's off-topic!


On 10/1/2012 2:00 PM, Diego Elio Pettenò wrote:

On 01/10/2012 13:54, Rich Freeman wrote:

I don't think we can keep the discussion off -dev forever though.  It
seems like we're close to being able to implement, which means lots of
changes that impact all devs.  I can't imagine that we'd want to
implement that without some kind of council vote.  Perhaps the
appropriate approach is to propose a GLEP?


No, the appropriate approach is first to _talk with Infra_. Guys I said
that before, but unless you actually factor in Infra when you want
infrastructure stuff done, you have to do the work yourself.
And it might not be nice and fun.


Well I definitely can't argue with the above, and I didn't know about 
-scm but, hey, you learn something new every day; anyhow I'm not sure 
it's makes it inappropriate to discuss.



So to close this in a few words: You walk the walk, you talk the talk.
And I have no intention to read another mail with what other awesome
thing we can do if we migrate to git and we don't even have to worry
about what it might happen on the serverside because git is just magical
and will sort itself out, okay?


OK, also agreed that git lacks entirely in magical properties -- I 
wasn't aware of having said anything suggesting otherwise.  I have some 
limited experience as a git admin and am aware of git's considerable 
limitations with respect to provisioning, security configuration, 
scalability and other non-optional features for a public-facing 
deployment.  There is more to be said on the matter but I'll take that 
to -scm.


Anyhow, I get it: administering the vcs for a huge project such as 
Gentoo is very hard work.  If I somehow gave some other impression, I'm 
sorry.   Perhaps Rich and I insensitively voiced our shared assumption 
that Gentoo's continued reliance on cvs stems from a lack of motivation 
and consensus, rather than a shortage of labor and resources.  Then 
again, if the folks in the trenches doing the work think I've slighted 
them, surely they are perfectly capable of chewing me out on their own 
behalfs and don't need you to do it for them?


I'm not looking for a fight, but after reading the above and some other 
remarks in this thread, by you and others, I did want to at least 
clarify my position: No, of course I couldn't possibly know all the 
repercussions of a change like this, but I also find it difficult to 
believe that whatever hurdles exist are so intractable that we'd might 
as well just throw in the towel.  Although I regret any bad feelings I 
may have caused, I stand by my statements.  To be clear, the magic of 
the Gentoo community -- not git -- is what I believe will make this 
doable, and yes, I appreciate that that magic is actually just a lot 
of people doing a lot of nasty, thankless chores.


As for the whole put-up/shut-up business, I'd be happy to help out any 
way I can (although tbh I'd still say whatever was on my mind even if I 
was too busy).  I'll contact Infrastructure to make sure they're aware 
of me.


-gmt

[gentoo-portage-dev] blech... (multijob/multiprocessing work-around for cygwin)

[Gregory M. Turner]

On cygwin, there is a problem with bi-directional pipe support in bash.

I used to solve this with an ugly reversion in portage and an 
ultra-simple stubbification patch for multiprocessing.eclass (both 
serialized everything).


However, this really sucked for numerous reasons, including the obvious 
one: it makes stuff slow as hell.


So I've reworked everything to use uni-directional named pipes.

In portage I have:

diff --git a/bin/helper-functions.sh b/bin/helper-functions.sh
index c7400fa..87f3120 100644
--- a/bin/helper-functions.sh
+++ b/bin/helper-functions.sh
@@ -7,6 +7,36 @@

 source ${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}/isolated-functions.sh

+# try real hard to figure out if this is a cygwin host; cache results.
+this_host_is_cygwin() {
+   if [[ -n ${_this_host_is_cygwin} ]] ; then
+   return $_this_host_is_cygwin
+   fi
+   [[ -x ${EPREFIX}/usr/bin/uname ]]  \
+   [[ $( ${EPREFIX}/usr/bin/uname -o 2/dev/null ) == Cygwin* ]] 
 \
+   export _this_host_is_cygwin=0  return 0
+   [[ -x /usr/bin/uname ]]  \
+   [[ $( /usr/bin/uname -o 2/dev/null ) == Cygwin* ]]  \
+   export _this_host_is_cygwin=0  return 0
+   [[ -x /bin/uname ]]  \
+   [[ $( /bin/uname -o 2/dev/null ) == Cygwin* ]]  \
+   export _this_host_is_cygwin=0  return 0
+   # hail-mary before we resort to envvars
+   [[ $( uname -o 2/dev/null ) == Cygwin* ]]  \
+   export _this_host_is_cygwin=0  return 0
+
+   [[ -n ${CHOST} ]]  \
+   [[ ${CHOST} == *-cygwin* ]]  \
+   export _this_host_is_cygwin=0  return 0
+   [[ -n ${CTARGET} ]]  \
+   [[ ${CTARGET} == *-cygwin* ]]  \
+   export _this_host_is_cygwin=0  return 0
+
+   # either it ain't cygwin or something is very broken.
+   export _this_host_is_cygwin=1
+   return 1
+}
+
 #
 # API functions for doing parallel processing
 #
@@ -19,25 +49,51 @@ numjobs() {

 multijob_init() {
# Setup a pipe for children to write their pids to when they finish.
-   mj_control_pipe=$(mktemp -t multijob.XX)
-   rm ${mj_control_pipe}
-   mkfifo ${mj_control_pipe}
-   redirect_alloc_fd mj_control_fd ${mj_control_pipe}
+   export mj_control_pipe=$(mktemp -t multijob.XX)
rm -f ${mj_control_pipe}
+   mkfifo ${mj_control_pipe}
+
+   if ! this_host_is_cygwin ; then
+   redirect_alloc_fd mj_control_fd ${mj_control_pipe}
+   rm -f ${mj_control_pipe}
+   fi

# See how many children we can fork based on the user's settings.
mj_max_jobs=$(numjobs)
mj_num_jobs=0
 }

+# make sure someone called multijob_init
+multijob_assert() {
+   if this_host_is_cygwin ; then
+   [[ -z ${mj_control_pipe} ]]  \
+   die multijob initialization required
+   [[ $( file -b ${mj_control_pipe} ) != fifo* ]]  \
+   die multijob fifo gone
+   else
+   [[ -z ${mj_control_fd} ]]  \
+   die multijob initialization required
+   fi
+}
+
 multijob_child_init() {
-   trap 'echo ${BASHPID} $? '${mj_control_fd} EXIT
+   multijob_assert
+   if this_host_is_cygwin ; then
+   trap 'echo ${BASHPID} $? '${mj_control_pipe} EXIT
+   else
+   trap 'echo ${BASHPID} $? '${mj_control_fd} EXIT
+   fi
trap 'exit 1' INT TERM
 }

 multijob_finish_one() {
local pid ret
-   read -r -u ${mj_control_fd} pid ret
+   multijob_assert
+   if this_host_is_cygwin ; then
+   read -r pid ret  ${mj_control_pipe}
+   else
+   read -r -u ${mj_control_fd} pid ret
+   fi
: $(( --mj_num_jobs ))
return ${ret}
 }
@@ -70,6 +126,9 @@ multijob_post_fork() {
 redirect_alloc_fd() {
local var=$1 file=$2 redir=${3:-}

+   [[ ${redir} ==  ]]  this_host_is_cygwin  \
+   die Cygwin bash has broken  bidirectional redirection 
support.
+
 	if [[ $(( (BASH_VERSINFO[0]  8) + BASH_VERSINFO[1] )) -ge $(( (4  
8) + 1 )) ]] ; then

# Newer bash provides this functionality.
eval exec {${var}}${redir}'${file}'
--
(Here's hoping Thunderbird didn't reflow the above; apologies in advance 
if it did)


In multiprocessing.eclass, I do something fairly similar, also relying 
on named pipes rather than file-descriptor cloning.


I was wondering if anyone in the know could comment on how correct or 
incorrect the above actually is?  For example, is either portage 
multijob or mutliprocessing.eclass expected to work if someone nests 
multijob_init's (because I'm pretty sure mine won't -- FTR, this seems 
pretty naughty, given that both get their parallelism defaults from 
MAKEOPTS)?


Finally, to be clear, I'm not wanting this to go in to git or 
anything... at least, certainly not 

Re: [gentoo-portage-dev] blech... (multijob/multiprocessing work-around for cygwin)

[Gregory M. Turner]

On 9/24/2012 9:08 AM, Zac Medico wrote:

On 09/24/2012 03:25 AM, Gregory M. Turner wrote:

  multijob_finish_one() {
  local pid ret
-read -r -u ${mj_control_fd} pid ret
+multijob_assert
+if this_host_is_cygwin ; then
+read -r pid ret  ${mj_control_pipe}
+else
+read -r -u ${mj_control_fd} pid ret
+fi
  : $(( --mj_num_jobs ))
  return ${ret}
  }

Since you re-open the pipe for each read, there's a race condition which
I've explained here:

http://permalink.gmane.org/gmane.linux.gentoo.devel/77528


Gross, thanks for this pointer, Zac.  Plenty of solutions one could 
concoct, but no point dumping effort into this black-hole when I can 
just wait for somebody else to get fed up and fix the underlying problem 
for me :)


Guess it's back to stubs, for now.

-gmt

Re: [gentoo-dev] Re: [PATCH] eutils: Warn on built_with_use usage

[Gregory M. Turner]

My main focus here is switching built_with_use to actively nagging

 people to stop using it; this includes nagging EAPI0/1 users of it.

Sans the implementation details, anyone got complaints with the
intent?


I have a concern about it, yes.  But, maybe there's a good answer to my 
concern, so please consider this a friendly ebuild development question 
disguised as a complaint :)


Unless I'm missing something, it seems that once we deprive the ebuild 
developer of this feature, there is no simple, supported way to retrieve 
the information except to depend on it.


The issue is that calculating dependencies is not the only reason we 
might want to know if a package was built with a particular USE-flag, 
and if we get rid of built_with_use, we literally cut ourselves off from 
retrieving this information in any officially sanctioned way (except to 
DEPEND on it, which may not be semantically correct).


I can think of all kinds of legitimate reasons we might want to know if 
the installed such-and-such package was built with so-and-so use-flags 
without depending on it.  i.e.:


 o if the current gcc falls within a certain range of version
   numbers and was built with graphite, we are going to trigger
   a compiler bug.  Suppose that there is no graphite support
   or dependency in ${P}, and that we can apply a patch which will work
   around the bug, but at a performance cost in ${P} we'd rather not
   pay unless we have to.

 o We need to modify a Makefile based on how a package we
   BDEPEND on was built -- but suppose there is no BDEPEND
   /limitation/ to enforce -- in other words, either way, our package
   will build, and there is no correlating reverse dependency to worry
   about at runtime.

Such needs are fairly unusual, but they do come up in real life.

My concern is that this will lead to people doing things like:

 o cut-pasting the old implementation of built_with_use into ebuilds,
   -- but that implementation will break if the portage database layout
   changes

 o creating bogus one-off use-flags as a way of performing these queries
   (and, thanks to the upcoming requirement that USE flags
   always appear in IUSE, exposing those flags to the end-user, perhaps
   with some confusing description like whether such-and-such was
   built with so-and-so).

 o creating BDEPENDs of -- and sketchy parsers for -- portage-utils or
   similar suites, just to ask this question.

Admittedly, it's hard to prevent people from doing

  built-with-use foo/bar baz || die ${P} needs foo/bar with baz

since, once upon a time, that was SOP, and we'd have to parse the bash 
code or something to qa warn for it automatically.


But any number of similar prohibitions are simply documented in the 
developer handbook, including this one.


Am I missing something, here?  I kinda think we should go the opposite 
direction and un-deprecate the API.  It seems like we are cutting off 
our nose to spite our face here.


-gmt

Re: [gentoo-dev] Re: [PATCH] eutils: Warn on built_with_use usage

[Gregory M. Turner]

On 9/17/2012 1:00 AM, Ciaran McCreesh wrote:

On Mon, 17 Sep 2012 00:58:02 -0700
Gregory M. Turner g...@malth.us wrote:

Unless I'm missing something, it seems that once we deprive the
ebuild developer of this feature, there is no simple, supported way
to retrieve the information except to depend on it.


has_version.


On 9/17/2012 1:10 AM, Ralph Sennhauser wrote: On Mon, 17 Sep 2012 
00:58:02 -0700


 has_version foo/bar[baz] can be used in EAPI 2 and later.


oh, duh... I guess I was just conflating the two, sorry for the noise :)

-gmt

Re: [gentoo-dev] Re: News item 1: changes to stages (make.conf and make.profile)

[Gregory M. Turner]

On 9/10/2012 10:39 PM, Duncan wrote:

Gregory M. Turner posted on Mon, 10 Sep 2012 20:29:53 -0700 as excerpted:


However, IIRC, /etc/make.conf is just ignored by portage if
/etc/portage/make.conf is present, so symlinking, or even better, if
possible, hardlinking those files would probably do the right thing
for legacy tools that don't know about the new location... unless I'm
mistaken, which is always plausible :)


Thanks.  Reasonable approach and good to know.


Well, I did warn about the likelihood I was wrong :)  Pretty clever 
'cause that way even when I'm wrong (as turns out to have been the 
case), I'm still right :P


In all seriousness, if both of them are sourced, then could one get away 
with something like this?


/etc/make.conf:
source /etc/portage/make.conf

/etc/portage/make.conf:
if [[ __GENTOO_MAKE_CONF_ONCE == gotit ]] ; then
__GENTOO_MAKE_CONF_ONCE=gotit
.
.
.
endif

or are conditionals disallowed?  As Zac mentions, hopefully it's 
harmless to duplicate things, but, personally, I would worry about the 
effect of duplicates on performance, and also in PORTDIR_OVERLAY.  Plus, 
it just seems dirty.


-gmt

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Gregory M. Turner]

On 9/11/2012 9:54 AM, Ian Stakenvicius wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 11/09/12 12:43 PM, Zac Medico wrote:

On 09/11/2012 09:36 AM, viv...@gmail.com wrote:

Dunno where to place this request, but if we go for something
like EJOBS could we also make it phase specific? So compile,
install and test could have a different number of jobs running.
Possibly three different variables that override a predefined
EJOBS.


Per-phase sounds a little to fine-grained. Instead, I'd suggest to
add an ELOADAVG variable that's analogous to make's --load-average
option. That should be enough to compensate for any differences
between phases.


I personally wonder about why this would be necessary from the
perspective of the user; if the user's system at emerge time can
handle X concurrent processes per emerge-job , i don't see why it
would matter what phase these jobs would be launched from.

At the ebuild level, certainly, but that's one of the reasons for
EJOBS in the first place, so that it can be overridden consistently
within a phase, if necessary for the ebuild (regardless of build
system type), right?


I would hope so.  There are definitely real-world reasons to want to 
restrict jobs -- usually to just one -- in a particular phase... several 
ebuilds, notably several in @system, do this already by injecting -j1 
somewhere.


To say it's all about performance seems to be forgetting a major reason 
somebody might want to do this as a user or an ebuild author.


Hmm...

I should preface this by saying that it really pains me to say this, 
because now this just starts to seem so fucking complicated -- but now 
that I think about it, this really seems to highlight a kind of semantic 
discrepancy that this thread has mostly dusted under the rug.


There are really TWO things we seem to care about -- one is some kind of 
global build-parallelism frob-set, and the other is the ability to turn 
off all parallelism for certain parts of an ebuild and those are 
actually kind-of orthogonal.


In other words, imagine if we had some kind of global parallellism 
arbiter thingy that looked at EJOBS or ELOADAVG or maybe some other 
things we haven't thought of yet, and decided things like:


  o should portage start a new parallel emerge process?
  o What is the appropriate MAKEOPTS for the emake
that foo-1.0.ebuild just issued?

or even, one could imagine, such things as:

  o should portage decide that parallelism has gotten out of control
somehow and suspend or kill a running ebuild for subsequent
resuming or restarting?

  o should portage re-nice some of its processes?

Note that, effectively, we have this already, and it's called portage.
But one could certainly make a case for modularizing it better, since, 
in truth, we are talking about a very common, very abstract problem here 
which portage shares with any number of batch-build systems.


Such an engine could very well do exactly the right thing if it were 
faced with a constraint that a certain part of a certain build needed to 
proceed without parallelism due to limitations coming from the build.


Also, there are very large parts of most builds -- configure comes to 
mind -- that don't parallelize even if, perhaps, they should.  In such 
cases, a really smart global parallelism arbiter could easily respond by 
spawning more jobs from other builds.


Not sure what I'm suggesting we do about it, exactly, but just pointing 
out that maybe a completely correct solution requires a much more 
elaborate implementation than just a bunch of syntactic sugar around 
what we currently call MAKEOPTS.  Whether or not Gentoo wants to take 
that all on, right now, as part of the next EAPI, is certainly debatable 
-- in fact, I'm inclined to say maybe it's not the best idea.


Perhaps we should really be asking: is the status-quo really so 
problematic/inelegant that it needs fixing? -- before we decide how best 
to fix it.


-gmt

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Gregory M. Turner]

On 9/12/2012 5:58 AM, Ian Stakenvicius wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 12/09/12 05:55 AM, Gregory M. Turner wrote:


Note that, effectively, we have this already, and it's called
portage. But one could certainly make a case for modularizing it
better, since, in truth, we are talking about a very common, very
abstract problem here which portage shares with any number of
batch-build systems.

Such an engine could very well do exactly the right thing if it
were faced with a constraint that a certain part of a certain build
needed to proceed without parallelism due to limitations coming
from the build.

Also, there are very large parts of most builds -- configure comes
to mind -- that don't parallelize even if, perhaps, they should.
In such cases, a really smart global parallelism arbiter could
easily respond by spawning more jobs from other builds.



So essentially what you're saying here is that it might be worthwhile
to look into parallelism as a whole and possibly come up with a
solution that combines 'emerge --jobs' and build-system parallelism
together to maximum benefit?


Yeah, couldn't have said it better myself ... apparently :)


Advanced HPC systems (sys-cluster/torque along with an appropriate
scheduler, for instance) can do such things with their jobs when the
jobs are properly built; I could see portage being able to handle this
as well given most of what is necessary is already known (ebuild
phases, build system type (via eclass), etc).   However, given the
limitations already put on parallelism in terms of emerge order, etc,
I could see this solution needing to be -very- complex and integration
needing to occur on multiple levels.  We'd also need to consider
distcc (and other cluster-shared compilation methods if there are
any??)..  It would be an interesting project, though.


ACK all of the above.

Tempting to think more deeply about this but probably the last thing I 
need to do right now is to talk myself into another speculative project.


I've hurt my wrist a bit -- probably an RSI -- so should help deter me :S

Only a few major sources of parallelism exist in portage: --jobs / 
--load-average in emerge opts, multiprocessing eclass  equiv. ebuild 
helper, distcc, and make...  Infrastructure is already in place for all 
of those, so perhaps a good holistic solution exists that isn't /too/ 
complicated.


...OK another f!#!%$^ brainstorm incoming :)

For JOBS syntax... what really seems missing in portage are:

  o a clean way to say dont parallelize this particular make
invocation in ebuilds

  o a clean way to globally say try to use this parallelization
strategy when emerging.

So what about something like:

  o EMERGE_JOBS and EMERGE_LOAD_AVERAGE make.conf vars equiv. to
--jobs and --load-average emerge options

  o EBUILD_JOBS and EBUILD_LOAD_AVERAGE make.conf vars

  o If the latter are not specified, they are copied respectively from
the former (debatable for *_JOBS, since now we get 16 processes when
we asked for four).

  o MAKEOPTS is auto-extended to reflect EBUILD_JOBS/EBUILD_LOAD_AVERAGE
if  only if -j|--jobs|-l|--load-average options aren't provided in
make.conf/profile/envvar MAKEOPTS

  o however, if MAKEOPTS override EBUILD_JOBS or EBUILD_LOAD_AVERAGE,
issue a conspicuous yellow-stars warning

  o extend emake to accept a --non-parallel option which will
strip all -j|--jobs|-l|--load-average options from MAKEOPTS;
perhaps support an equivalent EBUILD_NON_PARALLEL envvar as well,
with support for override in profile.bashrc. Don't warn about this
overriding EBUILD_JOBS -- treat as SOP.

  o debatable: respect EBUILD_NON_PARALLEL in multiprocessing, etc?
or, perhaps, something like:

EMAKE_NON_PARALLEL=${EMAKE_NON_PARALLEL:-${EBUILD_NON_PARALLEL:-no}}

could be used to distinguish between don't use any parallelism
and don't use GNU's make parallelism in emake.  Also maybe a
better name exists that doesn't use double-negatives.

?

Seems to me something vaguely like the above would provide

  o backward compatibility for ebuilds and make.conf

  o not so vastly different than what we have

  o a decent way to specify what we really want globally;
insofar as portage doesn't do the best job effecting the requested
parallelization strategy, more ambitious tactics could be
implemented later, hopefully without huge interface revisions.

-gmt

P.S.:

(Kind-of-crazy additional idea: put ceil(sqrt(EMERGE_JOBS)) into 
EBUILD_JOBS when only the former is specified, and then let 
effective_emerge_jobs equal floor(EMERGE_JOBS/EBUILD_JOBS) but maybe 
too much automagic for this to be a good idea.)

Re: [gentoo-dev] Re: News item 1: changes to stages (make.conf and make.profile)

[Gregory M. Turner]

On 9/9/2012 6:34 PM, Zac Medico wrote:

On 09/09/2012 05:59 PM, Duncan wrote:

To your knowlege (IOW have you tested) having /etc/make.conf either a
symlink to /etc/portage/make.conf or a simple one-line
source /etc/portage/make.conf?


I've tested them both just now, and they work for me. Why wouldn't they?


If both /etc/portage/make.conf and /etc/make.conf were evaluated, stuff like

  FOO=${FOO} bar

could cause, i.e., duplications... not sure what all the rules are 
limiting what one can and can't put in make.conf, but one could imagine 
all kinds of wacky stuff.


However, IIRC, /etc/make.conf is just ignored by portage if 
/etc/portage/make.conf is present, so symlinking, or even better, if 
possible, hardlinking those files would probably do the right thing 
for legacy tools that don't know about the new location... unless I'm 
mistaken, which is always plausible :)


-gmt

Re: [gentoo-dev] ROOT, EROOT, and EPREFIX in scripts (was: relative ROOT: correct behavior when ROOT=)

[Gregory M. Turner]

On 9/6/2012 5:56 AM, Ian Stakenvicius wrote:

On 06/09/12 03:55 AM, Ulrich Mueller wrote:

On Thu, 06 Sep 2012, Gregory M Turner wrote:

${ROOT:=/} EPREFIX=@GENTOO_PORTAGE_EPREFIX@
EROOT=${ROOT}${EPREFIX}



When ROOT is undefined or empty, this script will assign //foo
to EROOT and bad things will happen in Cygwin.



Several correct-ish solutions exist, i.e., in the above we could
change the concatenation statement to read:



EROOT=${ROOT}${EPREFIX#/}


I'd rather do it the other way around: EROOT=${ROOT%/}${EPREFIX}

Reason: EPREFIX is guaranteed to start with a slash, whereas for
ROOT I wouldn't be so sure that it always ends with one.


Not to mention that `man 5 ebuild` specifically defines EROOT to be
${ROOT%/}${EPREFIX}/ already , so all that's needed is the removal
of the final '/' if I'm reading this correctly..


Yeah, that trailing slash is kinda sneaky because it saves the day in 
the decidedly non-pathological normal case where ROOT is undefined and 
EPREFIX is empty.


That's another kind-of gross inconsistency that crops up with some 
regularity.  In portage python code, EROOT does, indeed, always end in a 
slash, just as man 5 ebuild specifies.


But in some places, i.e., in eselect, EROOT does not to end in a slash, 
or even has trailing slashes stripped from it.  In the case of eselect 
(which I'm just using as an example), this quirk has a bunch of 
consumers: the eselect plugins use code like:


foo=${EROOT}/usr/bar

which, if it were in an ebuild, would instead be properly written as:

foo=${EROOT}usr/bar

Indeed, most ebuilds do this correctly although occasionally I've seen 
exceptions.  My overlay is for Prefix so it lucks out if ebuild 
authors forget this and add the slash; but if/when non-prefix cygwin 
support is resuscitated in Gentoo, all of those extra slashes need to be 
fixed or else.


Anyhow, my point is that this tendency to effectively standardize subtly 
inconsistent usage of EROOT is pretty !@#$*ing gross.


I'd meant to save this argument for a separate thread, since a) now my 
original question isn't likely to get answered and b) I might seem to be 
bitching and moaning, as I don't, yet, have any patches to share -- but 
IMO if a bunch of scripts are going to call a variable EROOT in 
Gentoo, it seems like a no-brainer that the variable should have the 
same meaning in all of them.


This isn't the case right now, and has led to some real-world problems, 
so my feeling is that there ought to be some official documentation 
explaining in clear, simple terms:


 o what is the idea/point behind ROOT, EPREFIX and EROOT
 o where these variables (legitimately) come from, in other words,
   when can a user or developer inject them, and when should he
   or she be retrieving them from portage (i.e.: ebuilds).
 o the circumstances under which these variables must be changed
   in-flight, especially, for relative-ROOT activation
 o how to properly cook these variables when writing a script
   that will interact with portage (specified in a way that does
   the right thing for invokers and invokees, alike).
 o any EAPI interactions with the above (don't think we have any
   yet(?) but some recent proposals seem poised to change this).

Once we have that right, it's just a matter of changing all the scripts 
to do what the official documentation says -- if we can't, then 
obviously it's back to the drawing board to fix the specification, 
lather, rinse, repeat.


I'd be happy to do this myself but I'm not sure I know all the correct 
answers to the above questions (hence my original thread topic, for 
example), nor am I 100% sure what the correct place would be to document 
this (probably the dev manual or the wiki?)


An alternative to documenting and synchronizing code would be to build a 
source-able library script to automatically handle this type of thing. 
Maybe that's overkill, or maybe it's brilliant, I'd be interested to 
hear people's opinions.  I'm pretty sure we are just talking about a 
very few lines of simple code so my gut instinct is that it's overkill 
-- but if we expect things to get more complicated in the future, then 
centralizing the code, now, could save us some hassles, down the line.


tl;dr: we should standardize EROOT, but how?

-gmt

[gentoo-dev] relative ROOT: correct behavior when ROOT=

[Gregory M. Turner]

Hello, in my overlay I need to fix a bunch of issues that crop up when 
implementing EPREFIX construction in scripts due to Cygwin's 
idiosyncratic, but POSIX-compliant, handling of paths beginning with 
// (Cygwin does some arguably pathological stuff when such paths are 
used).


Almost all of these stem from the careless tacking of ROOT onto 
EPREFIX in bash-scripts, i.e., as in this (made-up) bash-script snippet:


${ROOT:=/}
EPREFIX=@GENTOO_PORTAGE_EPREFIX@
EROOT=${ROOT}${EPREFIX}

When ROOT is undefined or empty, this script will assign //foo to 
EROOT and bad things will happen in Cygwin.


Several correct-ish solutions exist, i.e., in the above we could change 
the concatenation statement to read:


EROOT=${ROOT}${EPREFIX#/}

Thats a matter for another thread, however.

As I've been reading code like the above, I've stumbled across some bugs 
and inconsistencies pertaining to relative-ROOT and I've been trying to 
file bugs to get these fixed.


One such issue pertains to the case where the ROOT environment variable 
is defined, but empty.  Portage and most python code seem to treat this 
as equivalent to ROOT=${PWD}, which is to say, that relative-root 
functionality is triggered by this circumstance.


However, several app-admin and sys-devel scripts disagree and treat 
this as equivalent to ROOT=/ (as does my example above).


So... which behavior is correct?  Since I'm aware of the issue, I'd 
might as well file bugs against the incorrect ones and get everything 
consistent.


-gmt

Re: [gentoo-dev] relative ROOT: correct behavior when ROOT=

[Gregory M. Turner]

On 9/6/2012 12:55 AM, Ulrich Mueller wrote:

On Thu, 06 Sep 2012, Gregory M Turner wrote:

Several correct-ish solutions exist, i.e., in the above we could change
the concatenation statement to read:



EROOT=${ROOT}${EPREFIX#/}


I'd rather do it the other way around:
EROOT=${ROOT%/}${EPREFIX}

Reason: EPREFIX is guaranteed to start with a slash, whereas for ROOT
I wouldn't be so sure that it always ends with one.


Good point, but EPREFIX is empty for non-prefix gentoo.  So, I guess:

EROOT=${ROOT%/}/${EPREFIX#/}

It's a contrived example anyhow, so it's mostly academic.

My main concern is the correct behavior when ROOT is defined, but empty 
-- I mostly included this explanation to preempt anyone inclined to ask 
what are you trying to achieve? :P


-gmt

Re: [gentoo-dev] Re: prune_libtool_files() and pkg-config dependency

[Gregory M. Turner]

On 8/31/2012 4:48 AM, Rich Freeman wrote:

On Fri, Aug 31, 2012 at 6:42 AM, Michał Górny mgo...@gentoo.org wrote:


So please introduce virtual/compiler, virtual/linker,
virtual/posix-system, virtual/sratatata and add them to DEPEND of every
single ebuild.


Every ebuild doesn't need all of those - that is the whole point.  As
Duncan already pointed out, reducing @system is a goal, but it doesn't
mean that we need to get there overnight.  However, we'll never get
there if we keep going backwards.


My 2c on this:

I'm reluctant to make sweeping statements like this, for any number of 
reasons, but -- well, I'm gonna.


IMO, getting there by slow evolution is not the right way.  At some 
point, @system becomes so primitive that bootstrapping must come to 
depend on more than @system, or we have to add add more phases to the 
bootstrap process, or split @system up into smaller sets or something.


The point is, we can't gradually reach a goal that's incompatible with 
the fundamental premise.  It's all well and good to say let's not put 
more stuff into @system because we want it to shrink, but, as it 
stands, there's a de-facto policy that @system includes everything 
needed to have a reasonably functional Gentoo, including all of the 
compilers, development tools and interpreters portage, gcc, and your rc 
system of choice rely on.  Until that fundamentally changes, IMO what 
belongs in @system is whatever best suits its current purpose.


For the record, I'm not saying we need to put pkgconfig in - I'm totally 
agnostic about that, as I am about whether it should be brought in as a 
dependency.


I just mean, probably the best way to fix the fat-@system problem is to 
create some kind of vision for a more-modular Gentoo of the Future 
first, and create a roadmap for getting there, second.


Its possible, perhaps even likely, that if we try to go the incremental 
route towards @system reduction, we will find, along the way, creative 
solutions to the various issues that have kept it fat-ish so far.  But 
that's likely to lead to a fairly ad-hoc patchwork of hacks, which imo 
would most likely be inferior to what could be achieved with some kind 
of destination in mind (even if that destination is subject to major 
revision as Gentoo progress toward it).


-gmt

[gentoo-dev] cygwin: gmp/mpc: package.use.mask static-libs? (was: supporting static-libs)

[Gregory M. Turner]

On 8/28/2012 4:05 PM, Diego Elio Pettenò wrote:

On 28/08/2012 15:36, Mart Raudsepp wrote:

static-libs is for installing static libraries IN ADDITION to shared
libraries, not instead.
USE=static is for what you have in mind there.

PE is not the same as ELF so on Windows you either build one or the
other for a number of reasons.


In this case, the issue is purportedly that the headers get generated 
differently, depending on whether you build static or dynamic.


I don't think too much, unless I have to, about what kind of 
libtool/gcc/gentoo magic makes it all possible, but by and large, static 
and dynamic libraries just work in my overlay.  So if gentoo-prefix 
would create both static and dynamic for a given library on elf, the 
same thing happens in my overlay (I mean that this typically happens 
without any hand-holding, on my part, to make it work).



Now on a different note, this is not even what USE=static is for — but
that's way behind what we were discussing before.


euse -i says:

  static - !!do not set this during bootstrap!! Causes binaries
   to be statically linked instead of dynamically

Pretty sure I've done the right thing in my overlay by masking, but it 
can't hurt to mention that I'm always happy to hear any suggestions or 
ideas (although of course I reserve the right to politely decide that 
your ideas are crap and ignore them :P).


My reasoning for the flag-mask is that Gentoo always builds the dynamic 
libs for these packages and therefore so should my overlay.  Unless I 
fix the underlying conflict or implement some kind of side-by-side 
scheme (a la libncursesw), I really have no choice but to forbid 
building statically (note that cygports adopts the same solution).


Masking the use-flag for the affected atoms accomplishes this 
prohibition exactly, exclusively, and without any kludge (except 
unfortunately I still have to patch the e-build, as, for some goofy 
reason, configure doesn't interpret the static/dynamic options the same 
way after applying the cygports patches -- hopefully I can fix that just 
by leaving some patches out; I just haven't gotten to it yet).


-gmt

Re: [gentoo-dev] supporting static-libs

[Gregory M. Turner]

On 8/28/2012 1:09 AM, Michał Górny wrote:

On Tue, 28 Aug 2012 02:15:40 +0200
hasufell hasuf...@gentoo.org wrote:

static-libs

pointless


I have to mask this flag for dev-libs/{gmp,mpc} in my cygwin overlay, 
where one can have static or dynamic, but not both, as per. upstream 
requirements (no idea why).  So FTR, this is not always a matter of 
personal taste.


-gmt

Re: [gentoo-dev] CWD-relative ROOT support in portage: misfeature?

[Gregory M. Turner]

On 8/18/2012 5:50 PM, Ian Stakenvicius wrote:


On 2012-08-17, at 11:00 PM, Gregory M. Turner g...@malth.us wrote:


  greg@fedora64vmw /tmp $ mkdir foo
  greg@fedora64vmw /tmp $ ROOT=foo portageq envvar ROOT
  /tmp/foo/

Does /anybody/ use this feature?


Sorry for the HTML response... am on the road.

I don't use the feature but I would fully expect said behavior. ie, going with 
the example above I would expect that I'd need   the / in front for the path to 
not be relative.


A user and maintainer of this (vapier) has emerged.  I pooh-poohed the 
relative-ROOT idea when I discovered it a few days ago, but I've 
flip-flopped.  I was concerned it would be exploitable by Bad 
People(tm), but I think it's no more exploitable than absolute-only 
ROOT, so long as its implemented correctly.


So far, nobody's turned up to advocate against the status quo (except 
me, but I'm fine with it now), so I think the matter can be considered 
resolved.


-gmt

Re: [gentoo-dev] Re: Questions about SystemD and OpenRC

[Gregory M. Turner]

On 8/16/2012 6:26 PM, Rich Freeman wrote:

On Thu, Aug 16, 2012 at 4:05 PM, Michael Mol mike...@gmail.com wrote:

The limited-visibility build feature discussed a week or so ago would
go a long way in detecting unexpressed build dependencies.


[snip]


If portage has the
dependency tree in RAM then you just need to dump all the edb listings
for those packages plus @system and feed those into sandbox.



That just requires reading a bunch of text files and no searching, so it
should be pretty quick.


Portage could hypothetically compile such a list while it crawls the 
package dependency tree, but I suspect the cost will not be small as you 
predict.



As far as I can tell the relevant calls to
check for read access are already being made in sandbox already, and
obviously they aren't taking forever.  We just have to see if the
search gets slow if the access list has tens of thousands of entries
(if it does, that is just a simple matter of optimization, but being
in-RAM I can't see how tens of thousands of entries is going to slow
down a modern CPU even if it is just an unsorted list).


I appreciate your optimism but I think you're underestimating the cost. 
 Can't speak for others, but my portage db's churn too much for comfort 
as is.  Once we start multiplying per-package-dependency iteration by 
the files-per-package iteration, that's going to be O(a-shit-load).


Of course, where there's a will there's a way.  I'd be surprised if some 
kind of delayed-evaluation + caching scheme wouldn't suffice, or, 
barring that, perhaps it's time to create an indexed-database-based 
drop-in replacement for the current portage db code.


I've enclosed some scripts you may find helpful in looking at the 
numbers.  They are kind-of kludgey (originally intended for 
in-house-only use and modified for present purposes) but may help shed 
some light, if they aren't too buggy, that is...


dumpworld slices and dices emerge -ep output to provide a list of 
atoms in the complete dependency tree of a given list of atoms (add 
'@system' to get the complete tree, dumpworld won't do so).


dumpfiles operates only on packages installed in the local system 
(non-installed atoms are silently dropped), and requires/assumes that 
'emerge -ep world' would not change anything if it is to give accurate 
information.  It takes a list of atoms, transforms them into the 
complete lists of atoms in their dependency tree via dumpworld, merges 
the lists together, and finds the number of files associated with each 
atom in portage.  Any collisions will be counted twice, since it doesn't 
keep track.  It also doesn't add '@system' unless you do.  By default it 
emits:


 o A list of package atoms and the files owned by each atom (stderr)
 o total atoms and files
 o average filename length

What is, perhaps, more discouraging than the numbers it reports is how 
long it takes to run (note: although I suspect an optimized python 
implementation could be made to do this faster by a moderate constant 
factor, I'm not sure if the big-oh performance characteristics can be 
significantly improved without database structure changes like the ones 
mentioned above).


My disturbingly bloated and slow workstation gives these answers (note: 
here it's even slower because it's running in an emulator):


greg@fedora64vmw ~ $ time bash -c 'dumpfiles @system 2/dev/null'
TOTAL: 402967 files (in 816 ebuilds, average path length: 66)


real15m33.719s
user13m18.909s
sys 2m8.436s
greg@fedora64vmw ~ $ time bash -c 'dumpfiles chromium 2/dev/null'
TOTAL: 401300 files (in 807 ebuilds, average path length: 66)


real15m28.900s
user13m15.126s
sys 2m8.088s

My workstation is surely an outlier as I have a lot of dependencies 
and files due to multilib, split-debug, and USE+=$( a lot ).  It's also 
got slow hardware Raid6 and the emulator only gives it 2G of ram to work 
with.  But I'm a real portage user; I'm sure there's other ones out 
there, if not many, with similar constraints.


-gmt
#!/bin/bash

if [[ x$(qlist -IC app-portage/portage-utils)x == xx || \
x$(qlist -IC app-portage/gentoolkit)x == xx ]] ; then
echo This utility requires both app-portage/portage-utils 2
echo and app-portage/gentoolkit.  Emerge them both and try again. 2
exit 1
fi

declare -a arguments atoms

arguments=( )
atoms=( )

verbose=yes
redic=no

for arg in $@ ; do
case $arg in
-q|--quiet) verbose=no ;;
-r|--redic) redic=yes ;; 
 *) arguments=( ${arguments[@]} $arg ) ;;
esac
done

[[ ${#arguments[*]} == 0 ]]  arguments=( '@world' )

for arg in ${arguments[@]} ; do
if [[ ${arg} == @* ]] ; then
newatoms=( ${arg} )
else
newatoms=( $( qlist -eICv ${arg} | sed 's/^/=/' ) )
fi
newatoms=( $( dumpworld ${newatoms[@]} ) )
result=$?
[[ ${result} != 0 ]]  { echo dumpworld failed, giving up. 2 ; 
exit 

Re: [gentoo-dev] remove system set?

[Gregory M. Turner]

On 8/16/2012 8:26 PM, Michael Mol wrote:


Ideally, you'd want as narrow a bootstrapping channel as possible.


I guess I tend to think that, too, and I'm pretty sure it's correct. But 
I don't normally think about why, and since you've prompted me to do so, 
perhaps it's a good moment to interject the fairly obvious, but 
amusingly contrary notion that there is too much of a good thing to be 
had in this dimension.


I suppose the main reason to want a minimal pre-bootstrap tool-chain is 
to maximize repeatability, minimize platform quirks, maximize 
time-stability in the face of changing code bases, and so forth.


Plus... you know... it's a lot more likely to make somebody say Wow! 
the narrow way.  Which maybe sounds like I'm poking fun, but I do 
actually think there's some intrinsic value in that.


However, there /are/ also reasons to make the bootstrap-er machine more 
fat and complex.  Most of them boiling down to there only being so much 
time in a day.


Otherwise we would boot-strap from stage -10, consisting only of a sed 
script and some architecture files to generate crude asm lexers 

:)

-gmt

[gentoo-dev] CWD-relative ROOT support in portage: misfeature?

[Gregory M. Turner]

It has come to my attention that gentoo supports relative ROOT, which 
is to say that, by design, portage will act as though (in bash terms):


  ROOT

equals

  ${PWD}/${ROOT}

when (again in bash terms):

  [[ $ROOT != /* ]]

at the moment execution crosses the boundary between a non-portage 
program and a portage program.  For example, I ran the following from a 
bash-prompt with PWD=/tmp in a portage-2.2 ~amd64 environment:


  greg@fedora64vmw /tmp $ mkdir foo
  greg@fedora64vmw /tmp $ ROOT=foo portageq envvar ROOT
  /tmp/foo/

Question: do we really want this behavior?

I have reason to believe that almost nobody uses this feature (namely, 
gcc-config and binutils-config are both broken under it for ages and 
nobody filed a bug or fixed it: see bugzilla #431104).


Does /anybody/ use this feature?  If not, I'd suggest that the portage 
team might ask itself whether the benefits of continuing to maintain it 
are greater than the hassle and potential for error it facilitates.


Just my 2c,

-gmt

Re: [gentoo-dev] Re: Questions about SystemD and OpenRC

[Gregory M. Turner]

On 8/16/2012 4:59 AM, Rich Freeman wrote:

On Wed, Aug 15, 2012 at 3:18 PM, Michael Mol mike...@gmail.com wrote:

It also sounds like something like that could be a benefit to shrinking @system.



I think the solution to the circular dependency issue isn't to make
Portage able to completely bootstrap the whole system, but rather just
to make it capable of coping with the issues and knowing when to raise
an alarm.

Gentoo will always involve extracting a tarball/etc for the initial
installation since you always need SOMETHING to start with.  You can't
even chroot into your install directory without a shell being there,
and typing emerge won't go so well if portage isn't actually
installed.

So, continue to build stages like we do right now - no doubt with
hard-coding and such to get around the dependencies.

As far as objections to listing gcc and such in every ebuild go, why
not?  We list all kinds of routine stuff in hundreds of ebuilds so
that we can install systems without them.  Why not just have a
toolchain virtual or something?

And since ssh was brought up - this is what happens with hacks like
this.  When you combine the default install with the minimum deps
for everything list you end up with an ssh you can't get rid of
without the package.provided hack (which really should be used for
stuff that is, well, provided).

It would be nice if people who want to build a server with Gentoo but
then reduce it to only true RDEPENDS could do so.  Obviously they'd
have to use binary packages to continue to maintain it (and even then
they'd need to keep portage on it), or they'd have to build another
one.  Actually, the trend in general is towards disposable servers
anyway so generating an entire new server every time one thing changes
is probably a desirable thing, since you probably want to be able to
do it every time you add a server anyway.


tldr: I like, approve and otherwise +1 the idea of somehow paring down 
or eliminating @system but I think it's going to be fairly challenging, 
so more discussion on this topic is warranted in my humble non-developer 
opinion :)


--

I really like everything you have to say here.  Unfortunately, 
assumptions of toolchain availability have gotten into the DNA of Gentoo 
in ways that make it nontrivial -- although probably not rocket science, 
either -- to implement these ideas.


I'd say it's the kind of thing where somebody needs to do the work.  I 
think there is demand for this, but when it comes down to brass tacks, 
people who really need features like this can just write a script to 
push some tarballs or files around in a way that's good enough for 
their purposes.  What is the cost/bene for a single sys-admin to do all 
the work and politics of making this change?


However, staying with the cost/bene theme, we have here a kind of 
externality, as they say in economics, (which is a fancy way, I guess, 
of saying a bad decision or a raw deal), because, in the aggregate, I 
think it's pretty clear that the cost/bene favors doing that work.


To be clear, I don't have religion about getting rid of @system, per-se, 
but I do have religion about the stuff Larry the Cow told me when I 
first visited the Gentoo homepage in 2001, or whenever, which was, 
basically, that the software I was using had a bunch of frobs that I 
couldn't touch, because I was running an rpm- or .deb-based system, and 
that Gentoo was going to let me frob them.


It's not a total disaster, even now -- a determined sysadmin can 
absolutely do this right now with features like prefix, ROOT, binpkg and 
so forth but /really/ fixing this, so that non-standard/minimal 
setups just work, would allow Gentoo to effectively address a whole 
bunch of really practical, real-world use-cases -- use-cases Gentoo is 
in many aspects uniquely suited to address, due to Larry the Cow's 
brilliant insights -- yet, by-and-large, due to precisely this @system 
thing and the package-management decisions that have stemmed from it, 
for which Gentoo has become unsuitable or impractical.


Specifically, I'm talking, here, about managed LAMP servers, big-data 
clusters, and embedded.


I suppose I'm not doing much to fix it by ranting and raving like this 
however.  So see first paragraph :)


-gmt

[gentoo-dev] a libtool + multilib gentoo host + 64-32 cross-prefix problem: a request for eyeballs

[Gregory M. Turner]

I've stumbled onto an interesting little problem trying to build my 
64-32 cross-prefix, which I bootstrapped with a no-multilib 32-bit 
crossdev toolchain.


Please see https://bugs.gentoo.org/show_bug.cgi?id=430722, especially 
the final two posts.


I'd appreciate input on the correctness/feasibility of the various 
solutions I propose there.


Apologies in advance for the verbose, stream-of-consciousness tone, it's 
more of a brainstorm than a bug at this point, and in retrospect, 
bugzilla probably wasn't the right place for it.


-gmt

Re: [gentoo-dev] Portage FEATURE suggestion - limited-visibility builds

[Gregory M. Turner]

On 7/26/2012 11:26 AM, Rich Freeman wrote:

I've been messing around with namespaces and some of what systemd has
been doing with them, and I have an idea for a portage feature.

But before doing a brain dump of ideas, how useful would it be to have
a FEATURE for portage to do a limited-visibility build?  That is, the
build would be run in an environment where the root filesystem appears
to contain everything in a DEPEND (including @system currently) and
nothing else?  It might be useful both in development/testing, and
also in production use (not sure how performance would work in the
real world - I was able in a script to get it to build an enviornment
in a few seconds for a few packages).


In practice I think it's going to be very hard to make this work in a 
platform-independent way; however in principle this is a ridiculously 
sexy idea that has crossed my mind more than once.


The challenge is that it requires either

  o Building very large sandboxes on a per-package basis

or

  o Python-level access to unionfs/aufs-style COW features.

Imagine the tree of dependencies which would need to be thrown together 
for, i.e.: kmail or firefox!  This makes the former approach seem damn 
nearly infeasible.  The latter approach holds more promise, I think, but 
represents a pretty big development effort.


Still very sexy idea, if a python-fs-layering API could be coded up.

One thing to consider: even if it does work, continuing to support the 
old way without fancy COW features is going to be required if portage 
is still going to support Gentoo/Alt in all of its flavors (either that, 
or unionfs/aufs features would need to be coded up for all those 
platforms that lack them).


-gmt

Re: [gentoo-dev] x32 release candidate

[Gregory M. Turner]

- Original Message -
 i'm pleased to announce the initial x32 release candidate:
 http://dev.gentoo.org/~vapier/x32/stage3-amd64-x32-20120605.tar.xz

Also pleased to hear this!  Thanks!  Can't wait to find the time to play with 
it.  Did you do all that work yourself?  Is there a wiki or forum thread 
somewhere where folks can gloat and/or commiserate?

OTBSLCI'm cautiously hopeful/enthusiastic about x32's prospects and a 
prefix seems like a fantastic way to bootstrap a community which could 
blueprint cheat sheets for prospective binary package maintainers 
(potentially driving adoption -- at least in my fantasies -- in the Grandma 
Zone)./LC/BS/OT

-gmt

Re: [gentoo-dev] cygwin prefix patch: request for eyeballs

[Gregory M. Turner]

- Original Message -
 On Mon, Mar 26, 2012 at 4:12 PM, Gregory M. Turner g...@malth.us
 wrote:
  https://github.com/gmt/gmt-cygwin-overlay/blob/master/sys-apps/portage/files/portage-2.2.01.20271-cygdll_protect.patch

 Consistency in the style would be nice.
 
 For instance in cygdll-update:
 
 # Here you have spaces after the $( and before the )
 extra_protection=$( portageq cygdll_protect_list ${EROOT} )
 
 # Later on you don't put spaces...
 eval $(portageq envvar -v CYGDLL_PROTECT PORTAGE_HOSTNAME \
 + PORTAGE_CONFIGROOT PORTAGE_INST_GID PORTAGE_INST_UID \
 + PORTAGE_TMPDIR EROOT USERLAND)

yes, good point, (the latter is cut-and-paste from etc-update, the former my 
own; I do that habitually as a way to avoid learning how tokenization works in 
bash :)

 Assuming $PORTAGE_BASH is always bash, you should use [[ consistently
 in your bash.

vs. [?  That sounds like cut-paste from etc-update again -- it seems to have 
been coded with sh support in mind.  Don't think that matters for cygwin (in 
fact it'd better not, the way I coded it), so, well spotted, I'll fix it.

 IMHO it is a common mistake to write your own argument processing, is
 there a reason getops is not appropriate (portability?)

Got me there -- I'm a getopts ignoramus.

 Onto the python:
 
 if os.environ.__contains__(PORTAGE_PYTHONPATH):
 
 Any reason why
 if PORTAGE_PYTHONPATH not in os.environ:
 was not used?

indeed, couldn't I just use 'if not os.environ[PORTAGE_PYTONPATH]:' (if the 
python executable were False or 0 would something stupid happen?)?  Also 
cut-paste from portage, but I did notice it, and had the exact same question.  
After my initial, unprintable reaction :)

Maybe the answer is buried somewhere in the docs ... afaik in 2.7, x in foo 
= foo.__contains__(x) (?).  Guess I'd need to double check 2.7 and 3.2 docs 
as well to be sure.

One idea: maybe meant as future-proofing against security bugs that could open 
up in future pythons.  I'll try to find blame (or whatever they call that in 
cvs) on this line of code, maybe we can ask the culprit himself.

 Python functions are not perl, don't pass 'argv' to the function.
 
 If the function takes to arguments, it should be:
 
 def func1(arg1, arg2)
 If some of the args have defaults:
 def func1(arg1, arg2=5)

Hm, are you looking at bin/portage_master_lock when you mention this?  It's 
cut-and-paste from portageq (starting to see a pattern yet? :) )

 I've never seen anyone use  before; I didn't even know it existed.
 Most folks use !=.

Sounds right, I'll seek and destroy.

 I can't really imagine why people do stuff like share
 /var/lib/portage
 across machines, so adding a bunch of complexity just to make it work
 seems nuts to me. That being said I have not worked on portage in
 years and Zac seems happier to support weird use cases.

Yikes, that's a pretty good critique, I'm afraid.  Almost painful to read :)

I decided to go ahead and implement this first, because I've seen some 
real-world file-sharing setups serving cygwin to clusters of development 
workstations, and secondly, because the consensus on #gentoo-portage was that 
this was a supported use-case.

Anyhow, seeing as how I already implemented it, it's too late for me to decide 
that it's too much work :)

In my defense, I thought it would be way less involved and it just kinda 
feature-creep'ed up on me.  By the time I figured out what a monster I'd 
created I was kinda pot-stuck.

-gmt

Re: [gentoo-dev] cygwin prefix patch: request for eyeballs

[Gregory M. Turner]

- Original Message -
  'if not os.environ[PORTAGE_PYTONPATH]:' 
 If PORTAGE_PYTHONPATH is not in os.environ then it will raise a
 KeyError, that is why we are doing a contains to begin with.

I somehow got the idea that the python gods had sprinkled magical syntax-sugar 
on bool(x[y])?  But maybe that doesn't make sense, it could cause gross 
surprises.  Anyhow I should rtfm rather than post my OT un-educated guesses...

-gmt

[gentoo-dev] cygwin prefix patch: request for eyeballs

[Gregory M. Turner]

Hello, I would appreciate if those of you with portage development experience 
and a moment to spare could please take a look at:

https://github.com/gmt/gmt-cygwin-overlay/blob/master/sys-apps/portage/files/portage-2.2.01.20271-cygdll_protect.patch

Not gunning for upstream or anything, but I would eventually like to start 
moving towards not a complete pile of crap status :)

Notes:

  o based against a-few-weeks-old rsync prefix-portage

  o may contain a very few hunks that should arguably go
upstream--I will make bugs for them and they are not
what (I feel) needs eyeballs.

  o trying to actually run this may be difficult.  let me
know where you're stuck and I'll try to help.

  o I must admit, I pretty much learned python as I went,
so... if you are thinking that looks like a total
rookie mistake it probably is, and I'd probably
appreciate your constructive critique.

  o I'll probably keep hacking on this so... moving target.
But it's mostly working, I think.

  o This doesn't make any big changes to upstream code.  So,
If it looks like it might be a big indentation change,
it probably is.

  o this is to solve the problem that, in cygwin, a running python
crashes as soon as you change any filesystem .dll's on which
it depends and fork() -- this is semi-orthogonal to the
typical cygwin rebasing problem and distinct from it.
The problem has nothing to do with locked files (except
that my solution does use some standard portage file-
locking features).

-gmt