Re: [gentoo-dev] Packages up for grabs
It's low maintenance; only thing needed is either to rebase to my libtransform work, or add proper xz support. Either way, any questions, let me know. ~brian On Mon, Jun 17, 2013 at 1:16 AM, Sergey Popov pinkb...@gentoo.org wrote: 16.06.2013 13:49, Pacho Ramos пишет: Due ferringb retirement the following packages are up for grabs: dev-util/bsdiff I will take care of it... -- Best regards, Sergey Popov Gentoo developer Gentoo Desktop-effects project lead Gentoo Qt project lead
Re: [gentoo-dev] [PATCH] Check for USE_PYTHON - PYTHON_TARGETS equiv.
On Tue, Nov 06, 2012 at 12:31:07PM -0500, Mike Gilbert wrote: On Mon, Nov 5, 2012 at 7:48 PM, Brian Harring ferri...@gmail.com wrote: Either way, I'm honestly not trying to piss folks off here nor stop the efforts to dig us out of the python.eclass mess. That said, *this time around* that eclass should actually have a plan so that we don't wind up in the same cluster fuck scenario as prior. It's a simple enough request; document your roadmap. Asking for reviews w/out that, frankly, is pointless (and sooner or later some asswipe like me is going to start -1'ing things till that's addressed). If anyone wants to help in drafting a GLEP, I would appreciate it. I'll try to start that this week. There will definitely be some holes, since I really don't have an all encompassing plan in my head. But it will be good to fill those in. Holes in the glep can be sorted/fleshed out; trust me, way more preferable sorting it there then in a live implementation the tree relies upon. :) Thanks- ~harring
Re: [gentoo-dev] [PATCH] Check for USE_PYTHON - PYTHON_TARGETS equiv.
On Mon, Nov 05, 2012 at 01:15:45PM +0100, Micha?? G??rny wrote: And in case anyone wondered, the output looks like this: * PYTHON_TARGETS - USE_PYTHON inconsistency found. This may result * in missing modules when trying to use Python packages. Please ensure * that the same implementations are listed in both variables. * * Implementation python2_5 disabled but 2.5 in USE_PYTHON * Implementation python3_1 disabled but 3.1 in USE_PYTHON * Implementation jython2_5 enabled but 2.5-jython not in USE_PYTHON If you're going to complain to users about flags being disabled in USE_PYTHON, use the actual flag name. Also... hopefully you typo'd, but 2.5-jython? Where was it decided the naming there was being shifted around? None of the code in the eclasses backs that text up from what I can tell. As floppym said; this flag isn't documented, so you're making it fairly hard to actually comment/review on what you're doing here- we just see glimpses of random patches, without knowing exactly what you intend overall. Documenting your end goal (moving people to USE_PYTHON? etc) is kind of necessary; normal rules of review reply, you want reviews, you make sure your reviewers know wtf you're trying to do and ultimately why. ~harring
Re: [gentoo-dev] [PATCH] Check for USE_PYTHON - PYTHON_TARGETS equiv.
On Mon, Nov 05, 2012 at 05:50:24PM -0500, Mike Gilbert wrote:
On Mon, Nov 5, 2012 at 5:25 PM, Brian Harring ferri...@gmail.com wrote:
On Mon, Nov 05, 2012 at 01:15:45PM +0100, Micha?? G??rny wrote:
And in case anyone wondered, the output looks like this:
* PYTHON_TARGETS - USE_PYTHON inconsistency found. This may result
* in missing modules when trying to use Python packages. Please ensure
* that the same implementations are listed in both variables.
*
* Implementation python2_5 disabled but 2.5 in USE_PYTHON
* Implementation python3_1 disabled but 3.1 in USE_PYTHON
* Implementation jython2_5 enabled but 2.5-jython not in USE_PYTHON
If you're going to complain to users about flags being disabled in
USE_PYTHON, use the actual flag name.
Those are the correct flag names.
Use flags can't have '.' in them, so no, that's not the correct flag
name; if you're going to tell folk blah is not in USE_PYTHON, be
*accurate*- meaning dropp the /_/./ substitution.
Also... hopefully you typo'd, but 2.5-jython? Where was it decided
the naming there was being shifted around? None of the code in the
eclasses backs that text up from what I can tell.
That's no typo.
2.5-jython is the PYTHON_ABI from python.eclass. It means version 2.5
of the python language as implemented by jython. The intent is to
allow for it to be matched by a glob pattern like 2.* in
RESTRICT_PYTHON_ABIS.
I'd forgotten that python eclass has PYTHON_ABI it ordered w/ the
python version first.
As floppym said; this flag isn't documented, so you're making it
fairly hard to actually comment/review on what you're doing here- we
just see glimpses of random patches, without knowing exactly what you
intend overall.
I think the scope of people who can review this is limited to those
that have some understanding of how python.eclass functions; a limited
subset indeed. ;-)
Y'all want reviews, then you need to help your reviewers. Really is
that simple.
Plus, bluntly, nobody wants another scenario of python.eclass w/ no
one understanding it.
Meaning what y'all are trying to develop/push, has to be
documented/understood by everyone. Strongly suspect this is why there
isn't any real review occurring for these patches coming in- no one
particularly knows wtf y'all are going with this.
USE_PYTHON is a variable in python.eclass that python devs use to
install python packages for any arbitrary python implementation,
without using eselect python. sys-apps/portage knows nothing about it,
and your typical user doesn't do anything with it.
Eh, two things;
1) info_vars; portage actually does report on it for emerge --info.
If it's internal (no users should be setting it), then you should
revert your addition of it to info_vars.
2) nothing in the tree actually uses it. Literally, niadda- just a
dead code section in python.eclass hinged on the var being exported.
Raising the question of why y'all are adding error reporting for this,
and what this is supposed to sort.
Not particularly digging the fact this warning y'all want to add is
ran everytime python_foreach_impl is fired also; meaning this will
spam logs if it ever hits.
Documenting your end goal (moving people to USE_PYTHON? etc) is kind
of necessary; normal rules of review reply, you want reviews, you make
sure your reviewers know wtf you're trying to do and ultimately why.
The end goal here is to migrate ebuilds to PYTHON_TARGETS. We can't
migrate every package in the tree at once, so this check is being
proposed to ensure that the user's settings for python.eclass
(USE_PYTHON or eselect python --python{2,3}) and python-r1.eclass
(PYTHON_TARGETS use-expand) match up.
If they are out-of-sync, users may get build errors about missing
modules. For instance, if dev-python/setuptools is installed with
PYTHON_TARGETS=python2_7, but the user has python3.2 selected via
eselect, they may get an error when they try to install packages using
python.eclass if they depend on dev-python/setuptools.
As we migrate more python libraries to python-r1, we can utilize
use-deps to move this into the dependency resolver. This is something
that was not possible with the old python.eclass.
This isn't quite what I'm asking for. I want y'all to literally
document thus:
1) What your finished solution is going to look like. Users control
which implementations are enabled via PYTHON_TARGETS, how y'all will
handle if no targets are currently specified (user hasn't made a
choice), etc.
2) How you're planning on getting there- literally, how things are
going to transition to your proposed replacement for python.eclass,
how the use deps will be structured, the potential gaps dependencye
wise as you go forward, etc.
As I indicated in my email, all folks see are changes coming in, no
sign of the actual end form you're intending, nor how you plan on
getting there.
I realize y'all may get pissed
Re: [gentoo-dev] [PATCH] Check for USE_PYTHON - PYTHON_TARGETS equiv.
On Mon, Nov 05, 2012 at 06:54:45PM -0500, Mike Gilbert wrote: On Mon, Nov 5, 2012 at 6:34 PM, Brian Harring ferri...@gmail.com wrote: This isn't quite what I'm asking for. I want y'all to literally document thus: 1) What your finished solution is going to look like. Users control which implementations are enabled via PYTHON_TARGETS, how y'all will handle if no targets are currently specified (user hasn't made a choice), etc. 2) How you're planning on getting there- literally, how things are going to transition to your proposed replacement for python.eclass, how the use deps will be structured, the potential gaps dependencye wise as you go forward, etc. As I indicated in my email, all folks see are changes coming in, no sign of the actual end form you're intending, nor how you plan on getting there. I realize y'all may get pissed at being asked to layout your actual plans... fact is, python.eclass got into this mess since people couldn't track where it was going ultimately leading to the source being spaghetti. Lay out where y'all are going w/ this, and *how*, so people can actually comment/contribute/avoid recreating another python.eclass where no one understands it. :) I will admit right now that I don't have a master plan in mind, and we are sort of making it up as we go along. I am not a software engineer; I just like to read/hack on code. I would love for some master architect to come along and document where we are headed, but that hasn't happened. And I'm not the person to do it -- it just doesn't interest me. mgorny has been writing lots of code and is looking for someone to review it. I'm very good at answering specific questions and offering an informed opinion, so I'm attempting to do so. Just zeroing in on this since my other questions aren't really getting addressed in the fashion I want- this however is the core point. Bluntly, there needs to be a plan- and it needs to be shared w/ folk. I wasn't kidding when I stated review requires making sure your reviewer understands your change and the intent; as is, all that's realistically occurring here is mgorny gets at best a comment or two bash wise, and that's it- nothing more since no one particularly knows where this is going and we just see random patchsets w/out much explanation as to how this all plugs together. I'm not demanding a point by point plan here; I'm frankly asking that the steps/end results be shared so that we don't wind up with another python.eclass, just this time w/ mgorny being the one who's got it all in his head. It's easy enough to write up a doc (glep if you want) laying out the intent and the roadmap. This should be done- if it has been already, link it in somewhere or start referencing it so the rest of us know wtf the plan is. Simple example, I'd ask why was PYTHON_TARGETS added when USE_PYTHON could've been fixed/coopted into the same thing?. I ask that now, I'll get flak/bitchyness frankly since it's already deployed/in use. Either way, I'm honestly not trying to piss folks off here nor stop the efforts to dig us out of the python.eclass mess. That said, *this time around* that eclass should actually have a plan so that we don't wind up in the same cluster fuck scenario as prior. It's a simple enough request; document your roadmap. Asking for reviews w/out that, frankly, is pointless (and sooner or later some asswipe like me is going to start -1'ing things till that's addressed). ~harring
Re: [gentoo-dev] Maintainer needed: dev-libs/icu
On Sun, Oct 28, 2012 at 10:35:01PM +0100, Arfrever Frehtes Taifersar Arahesis wrote: 2012-10-28 22:14:15 Mike Gilbert napisa??(a): This library is used for processing Unicode text in several high-profile packages, including Chromium and other Webkit browsers, PHP, boost, and many more. Fair warning: ICU tends to break several packages with every major release, so thorough testing is needed when bumping it. This package is currently being maintained by proxy by a former Gentoo developer, Arfrever. Given this package's potential to cause problems, this situation is not ideal. It would be really great if an active Gentoo developer would step forward and take care of this one. I am actively maintaining ICU and test many reverse dependencies with new versions of ICU (using a not package.masked version of GCC). Members of proxy-maintainers team or others actively commit fixes/updates, so there is no need to change current situation. Yeah... I don't agree with that. Floppym wouldn't be looking for a new maintainer if that was accurate. The package has been cranky enough in parallel with revdeps breaking everytime it's bumped that this needs a dev watching it, rather than whichever random proxy-maintainer member snags that version. Anyone got the spare cycles for it? ~harring
[gentoo-dev] reworking python-wrapper and wrapper scripts.
If folks haven't looked at python_generate_wrapper_scripts in
python.eclass, I'd suggest doing so. For examples of it's usage, grep
for 'python_generate_wrapper_scripts' in /usr/bin/; any place you see
it, look for that-script-name-${PYTHON_TARGETS} (for example,
/usr/bin/sphinx-build{,-2.7,-3.2}.
Each usage there is a separate custom script for that specific binary;
if there is a bug in the script, well, we're screwed- requires
re-merging the package.
This setup, at least on my hardware, is .04s added to every
invocation; this is ignoring the inode cost for each, and the issue if
a bug ever appears in the script generation code (in which case we're
screwed- would require re-merging the package).
In parallel, we've got python-wrapper (ls /usr/bin/python -l); this is
provided by eselect-python and basically discern what the active
python version is, and use that in the absense of any directives.
This is implemented in C, and is reasonably sane; the overhead for
that is basically non-existant.
Roughly, I'm proposing we do away with python eclass's
generate_python_wrapper_scripts generation of a script, instead having
that just symlink to a binary provided by eselect-python that handles
this. This centralizes the implementation (fix in one spot), and
would allow a c version to be used- basically eliminating the
overhead.
There's a trick to this; currently, those generated scripts hardcode
the allowed/known python versions for that package. We obviously have
to preserve that; I propose we shove it into the symlink path.
Basically, we add a /usr/libexec/python directory; within it, we have
a wrapper binary (explained below), and a set of symlinks pointing at
the root of that directory. To cover our current python versions, the
following would suffice:
for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8}
\2.7-pypy-1.9; do
ln -s ./ /usr/libexec/python/$x
done
While that seems insane, there is a reason; via that, we can encode
the allowed versions into the symlink. Using pkgcore's pquery for
example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3)
instead of a wrapper script at /usr/bin/pquery, we'd have thus:
targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy )
targets=$(IFS=/;echo -n ${targets[*]})
# This results in
# targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy
ln -s /usr/libexec/python/${targets}/wrapper \
/usr/bin/pquery
/usr/libexec/python/wrapper upon invocation, takes a look at argv[0];
sees how it was invoked basically. This will be the /usr/bin/whatever
pathway. It reads the symlink, in the process getting the allowed
versions and preferred order of the versions.
Few notes; vast majority of filesystems will store the symlink target
into the core inode if at all possible- in doing so, this avoids
wasting an inode and is only limited by the length of the target.
That length is capped by PATH_MAX- which can range from 256 to 4k (or
higher).
For the pquery example above, that comes out to ~73 bytes for the
symlink pathway; well under PATH_MAX.
For the scenarios where PATH_MAX caps the symlink pathway, or for
whatever reason we don't want to use that trick, a tree of files
contained within /usr/libexec/python/ holding the allowed versions for
the matching pathway would suffice.
Either proposal here would be far faster than what we've got now; also
will use less space (ancillary benefit).
One subtle aspect here is that if we did this, it makes it possible to
avoid losing the invocation information- currently if you did
`/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things
are implemented now (specifically the two layers of wrappers)- you'll
get python2.7 running that sphinx-build invocation.
This is wrong (it's directly discarding what the invocation
requested), although you're only going to see it for scripts that
do python introspection.
Via doing the restructuring I'm mentioning above, that issue can be
fixed, while making things faster/saner.
On a related note; we currently install multiple versions of the same
script- the only difference being the shebang. If one ignores the
shebang, in some cases this is necessary- where the script is 2to3
translated, and the code for py2k vs py3k differs. For most, the only
difference is in the shebang however.
While it's minor in space savings, it's possible to eliminate that
redundancy via a shebang target that looks at the pathway it was
invoked via. Fairly easy actually, and basically zero overhead if
done.
Either way, thoughts?
What I'm proposing isn't perfect, but I'm of the view it's a step up
from what's in place now- and via centralizing this crap, makes it
easier to change/maintain this going forward as necessary.
~harring
[gentoo-dev] Re: reworking python-wrapper and wrapper scripts.
On Thu, Oct 18, 2012 at 12:00:43PM +0200, Dirkjan Ochtman wrote: On Thu, Oct 18, 2012 at 11:15 AM, Brian Harring ferri...@gmail.com wrote: Either way, thoughts? It looks like you haven't looked at the python-r1 effort. That means you probably also aren't subscribed to the gentoo-python mailing list. I suggest you do both first. :) Yeah, I'm not on the ML- as for the eclass, I read the source; nothing in there in re: to wrappers currently. Only thread I see that is relevant is http://article.gmane.org/gmane.linux.gentoo.python/11 ; roughly executable wrapper, just what I'm spelling out here is the exact mechanism's of how to do it. Re: python-r1; frankly... if folks are fine w/ the wrapper, I don't want it delayed waiting on python-r1 (or only used there). As I laid out, what I'm proposing can be dropped in *now* for existing python eclass- basically moves that logic out of it and into eselect-python. It'll be faster/saner to maintain, less code in python eclass, and reusable going forward for the new attempts; plus I'm willing to do it (meaning it'll get done in the next week or two). That said; +1 or -1, or some where in between? ~harring
Re: [gentoo-dev] reworking python-wrapper and wrapper scripts.
On Thu, Oct 18, 2012 at 08:02:42AM -0700, Brian Dolbec wrote: On Thu, 2012-10-18 at 02:15 -0700, Brian Harring wrote: On a related note; we currently install multiple versions of the same script- the only difference being the shebang. If one ignores the shebang, in some cases this is necessary- where the script is 2to3 translated, and the code for py2k vs py3k differs. For most, the only difference is in the shebang however. What if the invoking script is not needed to be 2to3 translated (super minimal python code) but the remaining python libs need to be? Then we dedup 'em, using a shebang that knows to look at argv[0]'s last few chars to find the appropriate python binary to invoke via. As for the installation- either hardlinked'd scripts, or symlinked; doesn't really matter, although hardlink gets my vote (so sphinx-build-2.7 would be a hardlink to the same inode as sphinx-build-3.2). While it's minor in space savings, it's possible to eliminate that redundancy via a shebang target that looks at the pathway it was invoked via. Fairly easy actually, and basically zero overhead if done. Either way, thoughts? What I'm proposing isn't perfect, but I'm of the view it's a step up from what's in place now- and via centralizing this crap, makes it easier to change/maintain this going forward as necessary. ~harring +1 from me. Eclean has been checking the name it was invoked with long before I did the major re-write. From that it cleans either distfiles or packages if invoked by either the eclean-dist or eclean-pkg symlinks. If invoked by eclean itself then it looks for the target in the arguments. So Brian's proposal is not something totally new, never been done before... While this proposes something a little different. It is still very much along the same line and in my opinion a much better solution. Regarding having the script check on it's own... yeah, this has been on pkgcore's todo for a while, but basically got blocked by the current wrapper bits. I expect once this wrapper is in place, I may screw with the setup a bit to find a way so that scripts that can handle this on their own, are left to handle this on their own. Any other +1's/-1's? Else I may do this on the weekend for destress-hacking time... ~harring
[gentoo-dev] Re: reworking python-wrapper and wrapper scripts.
On Thu, Oct 18, 2012 at 11:28:59AM -0400, Mike Gilbert wrote:
On Thu, Oct 18, 2012 at 5:15 AM, Brian Harring ferri...@gmail.com wrote:
If we are somehow going to eliminate the installation of a separate
script for each python version, then the symlink idea sounds like a
good solution for expressing the supported python versions. However,
I'm really not sure how you would eliminate the separate scripts, in
light of the 2to3 issue.
The elimination is just a dedup run; track the PYTHON_ABI script's
being installed- check the md5 (minus the shebang). If it's the same,
they can be hardlinked together with the appropriate shebang wrapper.
It's doable, is the short version. Matter of fact, the dedupe run
there could be extended to do deduplication in general across python
versions, but that's an optimization for further down the line.
Regarding your /usr/bin/python3.2 /usr/bin/sphinx-build example:
invoking python on a binary (or a symlink to a binary) isn't going to
work at all. So I don't see how you've solved that issue.
Kindly provide reasoning/justification for such a claim in the future-
saves a round of email ping/ponging, and saves me getting pissy
towards people if they persist in it. :)
Lecture aside, nuh uh, it does work! is appropriate. I already
hacked up a prototype locally that works fine and does this via an
env passing hack- w/ what was proposed, this can be done w/out those
hacks, and would be faster.
Back to the discussion djc pointed out: mgorny has implemented a
binary solution already, called python-exec, in conjunction with
python-r1.eclass.
https://bitbucket.org/mgorny/python-exec
Another one I didn't see in the ML emails... joy. This is partial.
Finishing this to what I'm proposing above isn't too hard to do
however- that said, I strongly believe this doesn't belong in a
separate repo/package, this should be shoved into eselect-python.
eselect-python already gives us python-wrapper after all; no point
in having two layers of wrappers, and content in two repos.
python-exec assumes there will be a separate script for each version,
and just calls exec several times. In shell code, it would look like
this:
scriptname=$0 # argv[0]
exec scriptname-${EPYTHON}
exec scriptname-$( /etc/env.d/python/config)
for x in python2.7 python2.6 python3.2 python3.1 ... ; do
exec scriptname-${x}
done
The list of implementations for the loop at the end is hard-coded when
python-exec is built. This is the weak point in the solution. It could
be improved with your symlink idea.
What are your thoughts on mgorny's python-exec solution?
Code's a bit raw, but as I mentioned in my other email, it's on a
similar path to what I'm proposing- same core idea, just specifics in
mine + the symlink trick.
We could adapt it for python.eclass if you really want to support
that. Probably by modifying python.eclass to install scripts like
sphinx-python2.7 instead of sphinx-2.7.
Honestly, w/ what I'm intending, we don't need to change the python
eclass much- we can preserve the same naming in use if desired. If a
rename is desired, sure, do it when this is phased in.
That said if there isn't a reason for the rename, then don't do it,
imo.
~harring
[gentoo-dev] Re: reworking python-wrapper and wrapper scripts.
On Thu, Oct 18, 2012 at 04:50:04PM -0400, Mike Gilbert wrote:
On Thu, Oct 18, 2012 at 3:29 PM, Brian Harring ferri...@gmail.com wrote:
On Thu, Oct 18, 2012 at 11:28:59AM -0400, Mike Gilbert wrote:
Regarding your /usr/bin/python3.2 /usr/bin/sphinx-build example:
invoking python on a binary (or a symlink to a binary) isn't going to
work at all. So I don't see how you've solved that issue.
Kindly provide reasoning/justification for such a claim in the future-
saves a round of email ping/ponging, and saves me getting pissy
towards people if they persist in it. :)
Lecture aside, nuh uh, it does work! is appropriate. I already
hacked up a prototype locally that works fine and does this via an
env passing hack- w/ what was proposed, this can be done w/out those
hacks, and would be faster.
When you invoke a command like:
/usr/bin/python3.2 /usr/bin/somebinary
The python3.2 program assumes somebinary is a python script and tries
to parse it. Since somebinary is actually an ELF object, python bombs
out immediately.
Am I missing something fundamental here?
Nope, you've got me embarassingly, absolutely fricking dead to rights
on that one; I overlooked at the prototype I did was a python script
for 'somebinary', which worked via the env passing.
Would've spotted it upon implementing it in c to say the least, but
still, *cough*. ;)
As you mentioned in a separate email; if we stick a wrapper in front
of the python binaries, that would fly; I don't much like that
however.
Offhand... and this makes it a bit more complex, but still tenuable
imo, but we could get around this via shoving the symlink pathway into
the shebang itself.
scenario 1:
Script gets installed for 2.7, 3.1, 3.2; ignoring the shebang, it's
exactly the same md5 (meaning we can dedup it). The shebang instead
of being /usr/bin/python-whatever, instead is the symlink pathway I'd
referenced. Meaning it would be
/usr/libexec/python/2.7/3.1/3.2/wrapper . If that script was
sphinx-build, this means that sphinx-build{,-{2.7,3.1,3.2}} are all
valid python scripts- they can be invoked either via /path/to/script
or ${python} /path/to/script. There's a few details to sort there,
but nothing egregious best I can tell- implicit in this proposal is
that the wrapper would be smart enough to tell in which context it's
being invoked (sphinx-build vs sphinx-build-2.7).
Scenario 2:
2.7, 3.1, 3.2 are in play; they do /not/ dedupe to the same content,
meaning we actually need indirection in 'sphinx-build' (but not
sphinx-build-{2.7,3.1,3.2} although we would dedupe where possible
mind you).
In that case, to support python3.2 sphinx-build, we'd need a python
script that does the searching/re-exec for the wrapper. This can be
standardized- just have it search for sphinx-build-${PYTHON_ABIS} to
identify the installed versions, then do the usual active
version/epython trick to decide which to use. This honestly isn't too
far off from what we have now.
We can however optimize that; same trick, for the wrapper the shebang
isn't /usr/bin/python; it's just a c version of the python script,
done in c for speed.
So... caveats. If we screw w/ the shebang like this, file isn't going
to like it- it'll fail to identify certain scripts as python. That
said, it's worth keeping in mind that if we use the symlink route, a
file /usr/bin/sphinx-build would result in ELF, vs file
/usr/bin/sphinx-build-2.7 being python.
This can be fixed via an additional rule added to the magic db.
Pretty straight forward, and eselect-python (once it gets these
wrappers) could level an appropriate dep to ensure that's sorted.
So... my preference is shebang; it's basically an extension of the
symlink trick I'd originally mentioned, just cleaner imo.
What are folks thoughts on that?
~harring
Re: [gentoo-dev] reworking python-wrapper and wrapper scripts.
On Thu, Oct 18, 2012 at 11:54:21PM +0200, Micha?? G??rny wrote:
On Thu, 18 Oct 2012 02:15:43 -0700
Brian Harring ferri...@gmail.com wrote:
There's a trick to this; currently, those generated scripts hardcode
the allowed/known python versions for that package. We obviously have
to preserve that; I propose we shove it into the symlink path.
Basically, we add a /usr/libexec/python directory; within it, we have
a wrapper binary (explained below), and a set of symlinks pointing at
the root of that directory. To cover our current python versions, the
following would suffice:
for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8}
\2.7-pypy-1.9; do
ln -s ./ /usr/libexec/python/$x
done
While that seems insane, there is a reason; via that, we can encode
the allowed versions into the symlink. Using pkgcore's pquery for
example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3)
instead of a wrapper script at /usr/bin/pquery, we'd have thus:
targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy )
targets=$(IFS=/;echo -n ${targets[*]})
# This results in
# targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy
ln -s /usr/libexec/python/${targets}/wrapper \
/usr/bin/pquery
/usr/libexec/python/wrapper upon invocation, takes a look at argv[0];
sees how it was invoked basically. This will be the /usr/bin/whatever
pathway. It reads the symlink, in the process getting the allowed
versions and preferred order of the versions.
Few notes; vast majority of filesystems will store the symlink target
into the core inode if at all possible- in doing so, this avoids
wasting an inode and is only limited by the length of the target.
That length is capped by PATH_MAX- which can range from 256 to 4k (or
higher).
For the pquery example above, that comes out to ~73 bytes for the
symlink pathway; well under PATH_MAX.
For the scenarios where PATH_MAX caps the symlink pathway, or for
whatever reason we don't want to use that trick, a tree of files
contained within /usr/libexec/python/ holding the allowed versions for
the matching pathway would suffice.
While I agree that it's a clever trick, I doubt it's worth the effort.
Did you got any numbers proving it being superior over, say, trying to
exec() scripts like I do in python-exec?
While I can imagine, that in an worst case that bunch of exec()s is
going to be definitely slower than storing the list anyway, I doubt
such a bad case is often.
The difference in performance there is going to be negligable; I'm not
particularly concerned about that, and it shouldn't be a debate point
between my notion and yours (it's only a debate point when one is
talking about a c binary vs a python script).
One thing you're ignoring here is that the route(s) I mentioned all
allow for control/specifying what the order of preference is for
lookup/fallback (each link/shebang can encode that order how ever the
hell it wants).
Considering that the most common Python version used now is Python 2,
how often doesn't the script support that Python version? That's a very
rare case, and often just executing foo-${EPYTHON} works. In your
case, that common case involves readlink() + parsing + exec().
Even in case of Python 3 being selected, I doubt the overhead
of multiple exec()s on the scripts not supporting Python 3 is really
relevant. Please measure it if you believe so.
To be honest, I don't see any real advantage in this solution. It is
complex; understanding it requires explanation or some thinking.
The code will be fragile, and I'm not even sure if I'm not missing
something important here.
Not particularly sure how you claim this is fragile, but whatever,
your view.
I suggest you read my response to floppym. There's differing ways to
tackle this while getting the required speed, and covering what I want
(python3.2 sphinx-build working).
Thus, unless you've got a good arguments how this solution is superior
to the straightforward one done in python-exec, or numbers proving that
it is more efficient in a way noticeable to our users, -1. Smart hack,
yes, but not really beneficial.
Your -1 is duly noted.
Either proposal here would be far faster than what we've got now; also
will use less space (ancillary benefit).
One subtle aspect here is that if we did this, it makes it possible to
avoid losing the invocation information- currently if you did
`/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things
are implemented now (specifically the two layers of wrappers)- you'll
get python2.7 running that sphinx-build invocation.
This is wrong (it's directly discarding what the invocation
requested), although you're only going to see it for scripts that
do python introspection.
Via doing the restructuring I'm mentioning above, that issue can be
fixed, while making things faster/saner.
I don't
Re: [gentoo-dev] Re: [gentoo-pms] GLEP: gentoo sync based unified deps proposal
On Tue, Oct 02, 2012 at 02:08:02PM -0400, Ian Stakenvicius wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 02/10/12 01:56 PM, Ciaran McCreesh wrote: On Tue, 02 Oct 2012 13:51:01 -0400 Ian Stakenvicius a...@gentoo.org wrote: On 30/09/12 05:53 PM, Ciaran McCreesh wrote: On Sun, 30 Sep 2012 14:42:14 -0700 Brian Harring ferri...@gmail.com wrote: The second is that it starts the conceptual shift from cat/pkg is a build dep, and cat/pkg is a run dep to cat/pkg is a dep that is required for build and run. Fairly weak argument at best; you're claiming that via labels, contextually they know it's these deps in comparison to via dep:build contextually they know it's exposed only in build. Same difference. It's rather a big deal now that we have := dependencies. So you would using your labels syntax, specify an atom with a := dep using certain labels and the same atom without ':=' on other labels? I don't quite follow what you're getting at here as to how this is a big deal.. A := only makes sense for a dependency that is present both at build time and at runtime. Currently, the only place you should be seeing a := is on a spec that is listed in both DEPEND and RDEPEND. Conceptually, the := applies to the spec that is in both DEPEND and RDEPEND. But with the current syntax, there's no such thing as the spec that is in both. There are two specs, which happen to be identical as strings, one in DEPEND and one in RDEPEND, and there's no way for the two to be associated. Current syntax = *DEPEND, yes. Completely agree. In relation to Brian's proposal for DEPENDENCIES, tho, the two specs which happen to be identical strings would be rolled out from the same - -actual- string in the ebuild, and so, I don't see any such 'big deal' between the ability to conceptually express what's going on via his syntax and your labels. Unless i'm missing something, 'same difference' still fits.. Same difference applies; he's making the claim that the resolver can't tell that the python atom should be the same between build/run: dep:build,run? ( dev-lang/python:2.7= ) build: dev-python/snakeoil # vs labels build+run: dev-lang/python:2.7= build: dev-python/snakeoil The argument there is basically predicated on the belief that only labels can 'color' the sections it contains. This is a bullshit claim, and possibly specific to paludis internal failings. A sane implementation can walk that parse tree, and minimally infer that on it's own via the walk- or if it's saner, just track where things came from, and sort it via that way. Realistically a *good* implementation would likely be doing a partial rendering anyways (a good implementation already has the machinery for this for QA analysis reasons)- meaning conditionals beyond dep: would be finalized, leaving just those nodes unrendered, and then doing quick pass rendering of that intermediate form to get each phases specific requirements. Honestly it's a bullshit argument anyways; the unstated, but core argument of such nonsense is that the resolver if it saw dep:build? ( dev-lang/python:2.7= ) dep:run? ( dev-lang/python:2.7= ) would, because it's not one single build/run construct, think it can vary python:2.7 Any/all sane resolver already do collapsing and stabilization of common nodes across dep phases (and if paludis doesn't, well, that's their mess to sort; we're not getting any PROPERTIES=funky-slots hacks to work around their brain dead breakage here). The same situation can occur w/ labels via eclass dep manipulation; this is an artificial example, but anyone who has done deps know this sort of thing can/does occur via eclasses injecting common deps in: encode? ( build: dev-lang/python:2.7= ) build,run: dev-lang/python:2.7= Oh noes. How ever will the resolver know that it shouldn't vary the micro version of dev-lang/python:2.7 between build and run in that case! You just *know* it wants to vary the micro version because, such a completely fucking worthless thing for the resolver, it must do because it can, right? Etc. It's a pure bullshit argument, potentially derived from implementation issues for his own code, or just academic wankery; unsure of which, don't care which since the core argument is a new level of cracked out. ~harring
Re: [gentoo-dev] [RFC] Initial python-r1.eclass distutils-r1.eclass
On Mon, Oct 01, 2012 at 08:36:12AM +0200, Fabian Groffen wrote: On 30-09-2012 14:47:17 -0700, Brian Harring wrote: In the worst case it returns Bad marshalling data. Examples wanted for this. If this occurs, that's a python bug- one exception... portage (figures). They install into a non /usr/lib/python* location, meaning the .pyc/.pyo from py2.6 is exposed/accessed for py2.7 for example. https://bugs.gentoo.org/show_bug.cgi?id=300922 I doubt whether it's a Python bug, we have to mess with the files. But then again, I did some toying, and it seems Python doesn't care about this (any more?). Well, offhand that bug is pre EAPI3 (eapi3 was approved 01/18/10, and adoption was slow- lot of people skipped straight to eapi4) - so the mtime wouldn't have been guaranteed preserved for a long while. Meaning the bugs data I don't trust to be relevant due to timing, and age. As you said, this needs revisiting- minimally, portage is screwing around contents there, and I don't trust the python eclass to /not/ be forcing a compileall after the fact anyways. Suggest backing down the various protections for a full test, and resuming that bug- if you can replicate it, I'm definitely interested (dealt with this when it occurred for 2.3-2.4 for example). ~harring
[gentoo-dev] Re: [gentoo-pms] GLEP: gentoo sync based unified deps proposal
On Mon, Oct 01, 2012 at 08:13:49AM +0100, Ciaran McCreesh wrote: x? ( build: a run: b ) *is* nested conflicting. You're still failing to understand the point of labels parsing rules, though: the point is to make uses like the above well defined and consistent. I understand them just fine; you're just either very fucking daft, which I have a hard time believing, or lieing through your teeth (which fits a decade of behaviour including multiple suspensions for exactly that behaviour). Implicit labels context is build+run. Meaning the following x? ( build: a run: b ) *is* nested conflicting. is actually build+run x? ( build: a run: b ) Which isn't a nested conflict- subset, not conflict. You argue labels are required so people can do nested conflicts; meaning the following extreme example: run x? ( build: a test: b ) And as I nicely pointed out, /not a single fucking exheres/ does that. you've yet to pull out an example contradicting that analysis in addition. So... with that in mind- I'm doing two things; 1) can't force you back under a bridge, instead I'll do the killfile equivalent for a few weeks, 2) my original proposal if you kept being a tool seems appropriate: As said, you come up w/ real world examples, I'll include them; else persist and I'll just fold the academic wankery description of labels into the glep if you'd truly like me to (or you piss me off enough I do so to be a dick). What I truly love about that solution there is that it's both accurate, and if I play my cards right, I may be able to get a glep passed calling your proposal academic wankery; minimally, it'll be fun from my standpoint to try, so at least something came out of the last few emails from you. hugs and kisses- ~harring
[gentoo-dev] CVS - git, list of where non-infra folk can contribute
Cross-posting to scm; responses should go to scm please (and the people who whinge about cross posting should go promptly to hell if I have any say in the matter). On Mon, Oct 01, 2012 at 05:58:43PM -0700, Diego Elio Petten?? wrote: On 01/10/2012 17:51, Gregory M. Turner wrote: Anyhow, I get it: administering the vcs for a huge project such as Gentoo is very hard work. If I somehow gave some other impression, I'm sorry. Perhaps Rich and I insensitively voiced our shared assumption that Gentoo's continued reliance on cvs stems from a lack of motivation and consensus, rather than a shortage of labor and resources. That's definitely not the case. While we do have had some complains (mostly from Prefix last I knew) about git's working, the consensus for going to git is there. The problems are vastly technical. Problems such as how many developers would be fine with having to checkout 2GB of history to be able to commit? git support shallow clones but not if you want to commit to them. Few corrections; 1) You can commit to shallow clones. You can actually push from them too- you just have to know what you're doing (your parent *has* to be known to the other side, else you're trying to push a disconnected history/graph to the other side, which doesn't know how to connect the two). We won't be doing that fortunately, just noting that it is possible if you're careful (and I know what the man page says; what I'm saying is the full version, rather than the short version they list there). 2) graft's are what we'll be doing there; kind of shallow, but now. Basically the same thing the kernel folk did. As for the quit your bitching and contribute already rant angle; Diego's accurate; minimally, it's more productive to contribute and you're less likely to crap on folks motivation, let alone risk the wraith of a pissy person like me yelling at you. Here in is the kicker; certain chunks of this can't be handled by random joe blow off the street- they require core infra access. Bluntly (no disrespect to people, just being brutally direct) I don't care if you have infra friends, I don't care if you maintain a couple of boxes; if you're doing heavy OPs in a production environment, you'll understand the issue of trust/access- thus you'll understand that some of this work, cannot be done by anyone but infra. Like it or not, very few people have access to the core cvs - rsync hosts/machinery- since each/every/one/of/us means it's a security angle that has to be tracked. That's not arguable, so don't even try please. That said, there are non-infra contributions people can make. I suggest people do that; here's the list off the top of my head (these are things worst case, I'll sort- which means it'll be months out till I finish them considering my own time constraints and focus on getting eapi5 support into pkgcore first). 0) First the rules of the road for this discussion; assume that I'll be bitchy if you violate this. 0.a) We're not dropping the existing history. Suggesting this is asking for a killfile entry, it's viable for small or throw-away projects; gentoo-x86 cvs repository is not a throw-away project. 0.b) Lesser offence since it's not obvious; the various suggestions that we just snapshot this, then try to fix history after the fact won't work- look into git's transitive trust via sha1's of the parent's sha1. To do that sort of proposal means forcing a full history rewrite down the line; this doesn't fly. 0.c) For whatever I've missed, assume that if it craps on developers workflow... it's a no go, and needs to be addressed. Does CVS suck? Yes, I hate having to use it. But it *works*; switching to git has to be, minimally, a lateral move for developers in terms of their workflow- we cannot make it worse else what's the point of this whole exercise? There may be an exception or two here- things that aren't sorted immediately upon conversion, but those exceptions will only fly if they're minor, don't require history rewrites, and someone is locked in/guranteed to be working on it now (else we have no gurantee it'll actually be sorted). 1) We need a thin manifest - thick manifest converter. Thin manifests are used for git- they store just DIST entries. Thick (also known as 'full'), are what cvs/rsync users are familiar with- it holds checksums for all content. 1.a) This converter must use portage api's; ultimately, this thin-thick conversion will be signed by an infra key (rather than the current hodgepodge of devs). I suggest nesting it under the emaint command. 1.b) This converter needs to be fast. $VCS - rsync updates occur every 30 minutes. thin/thick sorting should be sub minute, frankly; go parallel (multiprocessing) being my suggestion, threadpool worst case (since most of the work won't be gil bound). 1.c) This absolutely has to be fucking stable. This will be a core part of our infrastructure after
[gentoo-dev] Re: [gentoo-pms] GLEP: gentoo sync based unified deps proposal
On Sat, Sep 29, 2012 at 05:05:09PM +0100, Ciaran McCreesh wrote: On Tue, 25 Sep 2012 15:46:14 -0700 Brian Harring ferri...@gmail.com wrote: Fun fact; peoples usage of labels in exherbo is thus: build+run: set of deps run: set of deps/conditionals/etc That's largely because there are a lot of former Gentoo developers there who all said oh, yeah, I forgot we could do it the other way when this was pointed out... I analyzed *all* exheres on git.exherbo. To be crystal clear, these include your packages. You yourself didn't use nested labels. So either the author of labels 'forgot' he could use it, or just didn't find the nesting actually useful. Considering I've not found any examples where nesting /would/ be useful, I'm inclined to agree w/ y'alls usage- that nesting doesn't matter. So... real world usage removes one of the core arguments of labels, leaving it just as it's a new syntax/aesthetically more pleasing in comparison to dep:build? ( blah ) form. Not expecting you'll agree with that statement based on the facts of y'alls own repo... so if you're going to retort, bust out actual examples from eithe trees, where nesting would be preferable to the form people use now please; else just drop it (-your- own usage of labels disproves your claim; thus why I want actual examples now if that point will be debated any further). Specification in terms of rendering has a huge problem, though. Remembering the crazy rules Gentoo has for || ( flag? ( ) ), what does this do? || ( dep:build? ( a ) dep:run? ( b ) ) Honestly, I was waiting for you to bring this up :) You're conflating two different things here; 1) someone being a dumb ass and writing what's effectively a || ( atom) block, just doing so in a manner w/out any reason to do so. 2) Your ongoing jihad against || (), specifically the occasionally valid complaint that build/rdepend different means the resolver can get stuck in certain pathways when slots are involved, abi, etc. Either way, in my proposal, I'm not going to single that out and try blocking it. The rendered version of it is still stable, albeit if it's build/run it's unlikely to be desired if there is ABI involved (for non ABI, specifically self-bootstrapping codebases, I suspect someone could come up with a valid construct- sed has something similar if memory serves). The rendered version ends up as ( a b ), in effect... It doesn't end up as || ( a (at build time) b (at runtime) ). Er, I assume you left out some chars there. The rendered version there isn't ( a b ); in old form it is: DEPEND=|| ( a ) RDEPEND=|| ( b ) This is why I label it a stupid use of syntax, but not ultimately harmful. Which is stupid, but syntactically correct. Nor is this a new issue, thus I don't particularly agree with your approach of trying to sink the proposal via an orthogonal problem. No, you're introducing a new kind of weirdness for || ( ) here. Na uh, you're the smelly face! As I said, and via correcting your misrendering, this isn't introducing anything truly new here; people can/have done '|| ( a )'; it's a stupid construct, and for paludis it means it /does/ treat that as an OR block (for hte rest, we do the more obvious tree collapses during parsing, folding a ( b ) down into a b, same for a || ( b ) into a b since they're the same thing). Either way, via http://dev.gentoo.org/~ferringb/unified-dependencies/labels/translated-to-use-deps.txt , I think it's pretty clear labels in real world usage aren't bringing anything to the tabel that we wouldn't have via my proposal; that leaves labels as just a different syntax (perhaps aesthetically more pleasing at first glance, but the label stacking bit via exheres analysis is proven to be something people explicitly go out of their way to protect against; meaning the aesthetics have a mental model cost). It's not go out of their way to protect against. It's there's an easy way of making sure everything is composable. Your misappropriation of use flags doesn't have that. Again, you're pulling a na uh, you're the smelly face counter argument. Bluntly, you want something that is academically possible, but pragmatically/realistically unneeded- meaning the gains are basically not there, but the costs most definitely are. Now, for exherbo were y'all lack actual versioned format (exheres-0 has changed heavily since it's inception), and chucked everything and did it from scratch (right? or do I need to do a copyright analysis in addition?)... the situation differs. You can invent whatever syntax you want, since you're starting from scratch, changing the mental mode for parsing is fine. We however are *not* starting from scratch. This shifts what we'll accept for costs/gains ratio; frankly, the fact y'all don't make use of those claimed 'gains' makes me think y'all tried something
Re: [gentoo-dev] Addressing GLEP-62 itself
On Sat, Sep 29, 2012 at 11:55:22AM +0200, Micha?? G??rny wrote: On Wed, 26 Sep 2012 03:29:17 -0700 Brian Harring ferri...@gmail.com wrote: On Wed, Sep 26, 2012 at 08:02:44AM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 12:54:39 -0700 Brian Harring ferri...@gmail.com wrote: On Tue, Sep 25, 2012 at 08:58:07PM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 14:47:33 -0400 Ian Stakenvicius a...@gentoo.org wrote: Based on the above I do expect the reference implementation would also need to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. Yes, unless I'm missing something that's the intent. I will re-read and update the GLEP a bit sometime this week. There's a fairly strong user interaction component here, along w/ potential nastyness for ebuilds (the proposal assume that a flag will be toggable in all cases within an ebuild if IUSE_RUNTIME specified; I guarantee instances where that fails can be found in the tree if a basic audit was done). Additionally, this *is* useless if it's done in a form the UI an't display/handle; Ciaran may bitch about REQUIRED_USE's UI (which I knew going in was going to be problematic, just to be clear), but he's right on that front. ^^^ This point still needs addressing. What kind of addressing? The user interaction works like usual -- portage lists a bunch of flags, some of them may have additional hammers or sickles to mean that they will not trigger the rebuild. Nothing more is required. What is even more important, there is nothing new to learn for the user. In fact, he doesn't need anything new from UI. He will just set the flag like he did 10 years ago. 1) REQUIRED_USE interaction. That's a rats nest, trust me on that one. If your proposal is to just to have people tweak, get yelled at by the pm, tweak, etc, well, on behalf of users, thanks a lot. 2) While hard to comment since your 'updated' glep wasn't fully updated- now is self inconsistent (minimally, trace backwards compatibility; fix it in full next time)... you're not exactly covering how this will go; best I can figure, you just want to shove yet another coloring (great for color blind people) or syntax markup on emerge -pv style output, somehow indicating runtime toggable or not; this is getting picked at because that display already is a crapfest and overloaded. 3) You're ignoring cycles here; specifically suggested dep based cycles that influence the originating node, and how that is represented- this isn't counting representing during --tree mode what is brought in where/when because of it. 4) Finally, and more damningly, you're ignoring apps like porthole. You need to think long/hard about how *exactly* porthole is going to indicate to users what optionals are there- more importantly, what those optionals induce/require (that's where it truly gets ugly and your lack of dep resolver knowledge, and unwillingness to do a patch and learn the basics there become infuriating); || () or blockers w/in suggested alone are going to make things painful. Additionally, this needs to be thought out how it'll interact with eclasses; what stacking, etc. It doesn't cover the basics there to say the least. The proposal didn't cover eclasses at all. Is there a need to do so or are we chasing some kind of perfection based on filling all unused slots? Eclass stacking here matters; if it's stacked, it means ebuilds have to use out of bound (ie, other vars) to tell the eclass when it shouldn't mark a flag as runtime toggable. If it's not stacked by the pm, then they have to manually stack; that differs from the norm and makes it easier to screwup; however; does allow for them to filter, albeit a slight pain in the ass doing so. There's a choice there, and the answer matters, so yes, you should actually have a complete glep before trying to shove it up to the council and extract a vote out of them. Lest the intention is to just have them kick it back to the curb... As others have said, we're not stacking it. Using it in eclasses is whacky and should be avoided. End of the story. It's your proposal there boss. That's a stupid decision, but as said, your proposal to run into the ground if you'd like. However. I suggest you actually document that in your proposal that it breaks from the stacking norm. Also, drop the backwards compatibility claim at the bottom. It was bullshit before, the fact I keep having to picking
[gentoo-dev] Re: [gentoo-pms] GLEP: gentoo sync based unified deps proposal
On Sun, Sep 30, 2012 at 09:30:18PM +0100, Ciaran McCreesh wrote: On Sun, 30 Sep 2012 13:14:53 -0700 Brian Harring ferri...@gmail.com wrote: That's largely because there are a lot of former Gentoo developers there who all said oh, yeah, I forgot we could do it the other way when this was pointed out... I analyzed *all* exheres on git.exherbo. To be crystal clear, these include your packages. You yourself didn't use nested labels. So either the author of labels 'forgot' he could use it, or just didn't find the nesting actually useful. That's a rather disingenuous claim, considering how none of the packages I maintain have any non-trivial dependencies... You could equally well say that every single package I maintain makes use of nested labels in every single place where they're relevant. Admittedly, that was a punch nearing the belt or a bit below; that said it's not disenguous. Reality is, our current form can handle deps generally fine- what you label as trivial is the vast majority- I argue effectively all. And I intentionally gave you a way to disprove that; find real world dependency examples to disprove it. So... real world usage removes one of the core arguments of labels, leaving it just as it's a new syntax/aesthetically more pleasing in comparison to dep:build? ( blah ) form. Not expecting you'll agree with that statement based on the facts of y'alls own repo... so if you're going to retort, bust out actual examples from eithe trees, where nesting would be preferable to the form people use now please; else just drop it (-your- own usage of labels disproves your claim; thus why I want actual examples now if that point will be debated any further). It's not just that it's more aesthetically pleasing. There are two arguments favouring labels over use abuse. The first is that it doesn't have perverse behaviour associated with it like your misappropriation of use conditionals does. If you put labels deep in a tree, it's well defined. If you put your conditionals anywhere except the top level, crazy stuff happens. This statement of yours however is fairly disenguous; label behaviour when nested suffers the same mental parsing oddity (wait, we're in build context, and this is test? Wtf happens there?). With 'use abuse' however, the situation is clear: dep:build,run? ( x? ( dep:test? ( blah ) foon ) monkeys ) Means that 'blah' target doesn't show up. Which is the *same* as what happens if someone did thus in our existing syntax: x? ( !x? ( blah ) ) Worth noting, you didn't ban that from exherbo; you left that to sort itself out, same as I'm doing for dep:blah flags. Were I punching below the belt, the word 'hypocritical' would likely be involved. The second is that it starts the conceptual shift from cat/pkg is a build dep, and cat/pkg is a run dep to cat/pkg is a dep that is required for build and run. Fairly weak argument at best; you're claiming that via labels, contextually they know it's these deps in comparison to via dep:build contextually they know it's exposed only in build. Same difference. Bluntly, you want something that is academically possible, but pragmatically/realistically unneeded- meaning the gains are basically not there, but the costs most definitely are. No, I want something where things that are different look different. Dependency types are nothing like use flags, so they shouldn't look the same. I'll buy that argument, and to some degree- agree. I just view that as academic wankery without real world gain. So like I said, academically possible, but pragmatically/unrealistically unneded. No amount of arguing is going to dissuade me on that view either, although real world tree examples *might*- aka, stop talking, go analyzing. Either way, this is gentoo, we're talking about the ebuild format; just the same as everyone shredded mgorny's ass for proposing we mangle atom syntax w/out gain, and proposal you push for the deptree changing, has to have significant gains for changing the existing form; aesthetics at a cost aren't enough. The gain is that you have a syntax designed for what's being represented, not an existing syntax abused to sort of (but not quite, because it's still defined in terms of rendering down) do new things. Really, all it takes to see that your syntax is bad is one tiny little example: dep:build? ( dep:run? ( cat/pkg ) ) x? ( !x? ( cat/pkg ) ) Outlaw that in paludis/exherbo, and *perhaps* I'd listen to you. You're going into broken record mode; the fact people can do stupid shit if they're willingly trying to abuse the syntax isn't much of an argument- especially considering the PM can handle it either way, rendering it out to a noop. There shouldn't be any need to add in a repoman check to catch that kind of thing. The problem is entirely caused by bad syntax design. Implementing labels
Re: [gentoo-dev] [RFC] Initial python-r1.eclass distutils-r1.eclass
On Sun, Sep 30, 2012 at 10:58:06AM +0200, Fabian Groffen wrote:
On 30-09-2012 10:31:17 +0200, Pacho Ramos wrote:
Personally, I usually run:
- python_clean_py-compile_files - Clean py-compile files to disable
byte-compilation allowing us to drop all various ways of doing this that
were living in the tree some time ago.
Hmm, what's the problem with compiling them? Do you mean some case when
the results of the compilation are different from the way done
by the eclass?
Well, if I don't misremember, we currently prefer to compile them at
postinst phase instead of during src_compile, but maybe this is no
longer needed (no idea :( )
The files are indeed cache, and should be generated on the system that
installs the files, not the system that builds them. They are currently
outside of VDB. pyc files store the path to the original files, so
generating in ${ROOT} yields in wrong paths. Python sometimes
regenerates the files when it runs. It may as well just ignore them
(since they are newer but non-matching, unclear to me).
.pyc should be compatible within slotted python versions;
recompilation occurs if .pyc/.pyo mtime differs from the originating
.py. Via EAPI=3 mtime gurantees, that should be addressed- below
that, compilation via pkg_postinst is necessary due to the lack of
mtime guarantees.
In the worst case it returns Bad marshalling data.
Examples wanted for this. If this occurs, that's a python bug- one
exception... portage (figures). They install into a non
/usr/lib/python* location, meaning the .pyc/.pyo from py2.6 is
exposed/accessed for py2.7 for example.
That said, a .pyc/.pyo from an older python version causing a blow up
in a differing python version hasn't occurred since the 2.3 - 2.4
transition (if memory serves).
Either way, examples desired please.
~harring
Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies
Pardon the belated response; responding to emails that are quick where possible, but lagging on -dev. Missed this one however... On Wed, Sep 19, 2012 at 09:16:02AM -0400, Ian Stakenvicius wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 19/09/12 09:09 AM, Michael Orlitzky wrote: On 09/19/2012 06:59 AM, Duncan wrote: Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as excerpted: On 16 September 2012 21:15, Brian Harring ferri...@gmail.com wrote: So... basically, people are already doing this manually with their own intermediate vars. And this works fine, so it doesn't warrant a cosmetic change. @ferringb: yngwin has a point that I've not seen addressed. What /is/ wrong with the whole CDEPEND intermediate var idea? It seems to work and /I/ don't know of any problems with it (and it would appear, neither does yngwin), yet you talk about it as if there's something wrong with it. And while we're at it, do DEPEND=$RDEPEND ... style solutions have the same problems (or lack thereof)? The problem appears as we introduce more DEPEND variables (which is what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND, CDEPEND, and DDEPEND, and there's only some (i.e. not total) sharing going on then the COMMON_DEPEND pattern starts to fall apart. You potentially need, AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND (COMMON_DEPEND) This obviously gets worse as more DEPEND vars are introduced. Well not really, no -- the additional *DEPENDs that are being proposed (or at least mentioned) for new EAPI will either remove atoms from COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a COMMON_DEPEND or other intermediate variable won't really be necessary for them. It depends on the dep type actually, and how we're viewing those deps- if they must be complete or not. Consider test depends for example. Either we can specify it as in addition to depend- which sucks, because it's entirely possible for a DEPEND target to not be required for TDEPEND. While that's a corner case, it's actually easy to address; just require TDEPEND to be comprehensive for the test phase. Thus DEPEND and TDEPEND suddenly share a lot, and TDEPEND shares part of RDEPEND. Moving on to a real world example... PDEPEND. Currently a full graph resolution requires pulling RDEPEND and PDEPEND, collapsing them, and ensuring they're fullfilled for anything that is already installed (ie, PDEPEND is required after the transaction is completed). In our current separated var setup, this as said, this requires the PM to track/handle it separately. However, were PDEPEND to be complete/full- that would mean the PM could just render for dep:post and know this is what is necessary outside of the transactional block of building/installing it. So RDEPEND and PDEPEND actually share a *shitton*- they're the same contents in 97% of the tree. Meaning dep:run,post? for 97% of the tree, w/ 3% needing to have an addition dep:post? section. Offhand, comprehensive deps make things easier for devs- it gives them the ability to be crystal-freaking-clear as to what's needed at each stage; via them doing so, it means the resolver has a greater space to dig itself out of fucked up situations if necessary. Additionally, it actually makes life easier for PM authors. If we had TDEPEND (test), this is how things would go; note this is written for just an install, not a replacement (replace is similar, just noisier deflecting from my point): 1) ensure DEPEND is satisfied 2) run phases setup - compile 3) ensure TDEPEND is satisfied; Implicitly, DEPEND in the process (if there is a cycle induced by TDEPEND + DEPEND being required, we have no way out of it). 4) run install phase 5) ensure RDEPEND is satisfied. No longer care about TDEPEND/DEPEND. 6) run pkg_preinst, merge, pkg_postinst. 7) a transaction is opened up, that must be completed before the PM exits; before that exit, RDEPEND must still be satisfied, as must PDEPEND. With what I'm suggesting. 1) ensure dep:build is satisfied 2) run phases setup - compile 3) Ensure dep:test is satisfied. If cycle breaking is necessary, anything in dep:build but not dep:test can be pulled. 4) install phase 5) ensure dep:run is satisfied. Again, if cycle breaking is necessary, punt whatever isn't in dep:run as necessary to break that cycle. 6) pkg_preinst, merge, pkg_postinst. 7) transactional block; ensure dep:post is satisfied before exiting said transaction. That's for install; for the PM considering a pre-installed pkg, it converts from; 1) ensure RDEPEND and PDEPEND are satisfied to 1) ensure dep:post is satisfied. Tbh, I'm not sure I'm communicating these gains as well as I could be; at first glance, I'm pretty sure people will say what's the difference? because the gains/difference there are subtle
[gentoo-dev] Re: [gentoo-pms] GLEP: gentoo sync based unified deps proposal
On Sun, Sep 30, 2012 at 10:53:40PM +0100, Ciaran McCreesh wrote: But here's the thing: when you sell something as pragmatic, what you're really saying is it's wrong, I know it's wrong, and I'm going to pretend that wrong is a good thing. Getting it wrong should be something you do only after you're sure you can't afford get it right; it shouldn't be something you're proud of. No, when I say pragmatic, what I'm actually saying is that people who can't focus on cost/gain, by large, haven't had real jobs (else they would've had that perfectionism/decreasing gains ground out of them sooner or later), and are spending their time whacking off chasing a mythical 'perfect' solution. Academic wankery, is the short version. You're good at technical, but you frequently do the academic wanking crap which leads to things dead-ending... plus wasted time because to you, 'pragmatic' is a dirty word (compromise? Heaven forbid!). In my proposal, I am addressing labels; will fold in your claims, but those claims basically are shit- however, if you *did* find a conflicting nested example that wasn't contrived, preferablly multiple, I'd like those examples so I can include them into the proposal (give labels a fair hand, basically). You already have an example in your proposal, in the form of mplayer's X? ( ) dependencies. I said nested conflicting labels. Meaning build: x? ( dar run: blah ) which isn't the case for any of mplayer deps. Actual examples from the tree where a conflicting nested label is preferable. That isn't one of 'em, and you're unwillingness/inability to point out real world examples is just digging a deeper ditch for your argument. But that's missing the point. Even if you pretend complicated dependencies don't exist, labels are still by far the better proposal. Your argument boils down to it's more pragmatic to do a quick and dirty implementation in Portage and thus force the technical debt onto every single ebuild than it is to do it cleanly. My argument boils down to thus: We are not exherbo- we do not have the luxury of chucking out syntax and pulling NIH renaming of things for shits and giggles. Especially if the new syntax is directly translatable into a tweak of our existing syntax (a tweak that we should do anyways- recall I built this off of fixing USE_EXPAND). Your argument boils down to it's not labels, ignore that it's aesthetic knob polishing (you can do the same w/ our existent syntax, thus the analogy of waxing it I truly mean), use labels because I'll berate you incessently till you accede. Beauty of open source, you want it, go do it. If in, what, 4 years? 3? You've not been able to get off your ass, write a proposal, hell, do a portage patch (you've *never* done portage patches of any worth, bluntly- I know this since in the past I used to fix shit you requested), you lose your voice in the matter. Considering your points boil down to aesthetic academic wanking at this point... put up, or shut up, really is that simple. As said, you come up w/ real world examples, I'll include them; else persist and I'll just fold the academic wankery description of labels into the glep if you'd truly like me to (or you piss me off enough I do so to be a dick). ~harring
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposal
On Wed, Sep 26, 2012 at 08:58:54AM +0200, Micha?? G??rny wrote: On Sun, 16 Sep 2012 06:52:11 -0700 Brian Harring ferri...@gmail.com wrote: Keeping it short and quick, a basic glep has been written for what I'm proposing for DEPENDENCIES enhancement. The live version of the doc is available at http://dev.gentoo.org/~ferringb/unified-dependencies/extensible_dependencies.html One more question -- are we going to keep 'foo,bar?' syntax as a special case applying only to dependency atoms or are we going to extend it to USE flags? Note that's dep:foo,bar; not a bare allow any use flags to be OR'd together. In light of the fact it *is* just an expansion hack, the usage is semi limited although there are scenarios for it; arches, namely (if amd64 or x86, use this, if mips, that, etc). I have no preference either way; extending it outside of dep isn't necessary if people hate it, although as said, there are some potential uses for it. That said, if we were to start using it, the ',' as an 'or' operator mildly sucks; dep:build|run also sucks (hard to read), and dep:build+run, to me at least, implies 'and'. And yep, bikeshedding potential there. ~harrin
Re: [gentoo-dev] patch eutils.eclass for EAPI 5
On Thu, Sep 27, 2012 at 10:30:21AM -0700, Zac Medico wrote:
On 09/27/2012 10:16 AM, Ian Stakenvicius wrote:
On 27/09/12 01:07 PM, Zac Medico wrote:
On 09/27/2012 09:49 AM, Ulrich Mueller wrote:
As far as I can see, only the definition of the usex function
must be disabled. Please review the patch included below.
Ulrich
--- eutils.eclass 15 Sep 2012 16:16:53 - 1.403 +++
eutils.eclass 27 Sep 2012 16:45:14 - @@ -1373,7 +1373,9 @@ #
@DESCRIPTION: # If USE flag is set, echo [true output][true
suffix] (defaults to yes), # otherwise echo [false
output][false suffix] (defaults to no). +if has ${EAPI:-0} 0
1 2 3 4; then usex() { use $1 echo ${2-yes}$4 || echo
${3-no}$5 ; } #382963 +fi
# @FUNCTION: prune_libtool_files # @USAGE: [--all]
Looks good to me.
It may not work for unofficial EAPIs that don't include usex, but
I guess there's nothing we can do for those, and they can just be
replaced with newer EAPIs that include usex.
i actually just committed the fix discussed in #gentoo-dev , using
'declare -F' instead (similar to the eqawarn conditional declaration
already in eutils.eclass)
Sorry..
It's fine with me, but some of the other package manager devs might
object, since it makes assumptions about implementation details.
PMS isn't particular clear on the metadata sourcing env for EAPIs-
especially after the requirement that we pull the EAPI out of the file
on the fly was aded.
Point is, if it's EAPI5 the env should already export usex- even if
it's disabled- so case/esac is better imo.
~harring
Re: [gentoo-dev] Addressing GLEP-62 itself
On Wed, Sep 26, 2012 at 07:25:11PM -0300, Alexis Ballier wrote: On Wed, 26 Sep 2012 14:02:57 -0700 Brian Harring ferri...@gmail.com wrote: On Wed, Sep 26, 2012 at 02:38:02PM -0300, Alexis Ballier wrote: IUSE_RUNTIME is optional for PMs, why does the UI matter at all ? Also, the proposal doesn't assume flags are toggable at will, it assumes they are useflags and obey the same rules. I truly hate claims like this; it's optional, so who cares if it's whacky. Think through the proposal; for it to work reliably, it's not optional. Same issue I've been y'all over the head with, rendered/finalized vs raw/unfinalized deps being stored in the VDB. All managers have to write unfinalized if that proposal goes through, even if they don't support the optional toggling after the fact. Why? _Current_ PMs will rebuild the package. The point of this is just to give them a hint that they do not need to rebuild it. We already have an implementation actually: one that ignores the hint :) Bullshit. This is optional in the sense of embrace/extend 'optional'; if one PM takes up the new functionality, all have to switch to writing unfinalized deps to the VDB, and all have to switch to transfering that IUSE_RUNTIME crap to the VDB. If they don't, whatever sole/crappy PM that runs w/ this proposal will wind up forcing rebuilds on any packages merged by the PM's that don't do this optional glep. Thus rendering it nonoptional since it implicitly is reliant on all switching to the degrade *DEPEND writing that this proposal is reliant on. The blame game becomes well, you shouldn't use the PMs that don't do this *optional* thing. There in is the implicit lie of that 'optional' crap. Claiming other wise is ignoring reality; I called it embrace/extend because this is exactly how that shit goes- sure it's optional, 'cept you're forced to support it (even partially) else the whole degrades and that PM winds up getting blackballed or fragmentation occuring. As far as I'm concerned, any PMS intended proposal must not pull the 'should' or 'optional' crap; it has no place in a spec (spec's are supposed to be assertions after all). As for the UI... arguing but it's optional! doesn't give a blank check for the UI angle. What the plan, more colorization and a new char for emerge -vp? Because we're kind of running out of chars there... How is this relevant ? Um... dude... This proposal is about adding suggested/optional deps so people can inspect/select/enable them per package. You're asking how is the UI relevant in light of that. Just saying; it's kind of core to this whole damn thing, else we're just trying to add an optimization hack; either one runs a strong risk of my next response including a joke about elderberries and hamsters. ;) It's a simple enough request; one that wouldn't even need to be made if there was code backing this proposal; on a related note, hell yes I'm wary of having this dumped on manager authors heads and having to be told sort out the mess/make it so. So I'm asking y'all to at least put in an equivalent time investment doing a quick prototype. This isn't an unreasonable request, and has been the norm for most gleps for a long while. I guess people do not want to invest time in writing code for something doomed. This is one of the cases where tough fucking luck really/truly fits. If it's doomed, consider why it's doomed. I'm not requiring a prototype just because I'm a dick; I'm requiring a prototype because I fully expect since y'all won't listen to what people are telling you, trying to write the code will educate y'all to what we've been saying. This is ignoring that prototypes are bsaically the norm for proposals of this sort (both PMS and gleps)... meaning it's the standard, and y'all are trying to get this proposal special cased. Does it suck you can't just get what you want via writing a quick doc and arguing on an ml? At times, yes. If you believe it's worth it, you do the legwork. If the folks backing this can't be bothered to write a freaking patch, well, I think that's a pretty strong vote of no-confidence on the backers part. The original request was just to have it 'accepted' so that an implementation can start. If the implementation is good then make it final, otherwise amend or reject the glep. This isn't unreasonable either. Also known as rubber stamp it. And if it sucks, of course it's easy to roll that bad idea back? Right? If the idea was universally accepted and lacked issues, that may fly; that's not been the case. It cannot be stacked because y'all are trying to shove this in as an optional; unlike it's brother IUSE, which stacks. As for ons of others that don't stack; very few actually influence the package manager; ~14 roughly, minimally 5 of those stack (those that don't, basically aren't lists). So it's not stacked, nothing
Re: [gentoo-dev] Addressing GLEP-62 itself
On Wed, Sep 26, 2012 at 08:02:44AM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 12:54:39 -0700 Brian Harring ferri...@gmail.com wrote: On Tue, Sep 25, 2012 at 08:58:07PM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 14:47:33 -0400 Ian Stakenvicius a...@gentoo.org wrote: Based on the above I do expect the reference implementation would also need to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. Yes, unless I'm missing something that's the intent. I will re-read and update the GLEP a bit sometime this week. There's a fairly strong user interaction component here, along w/ potential nastyness for ebuilds (the proposal assume that a flag will be toggable in all cases within an ebuild if IUSE_RUNTIME specified; I guarantee instances where that fails can be found in the tree if a basic audit was done). Additionally, this *is* useless if it's done in a form the UI an't display/handle; Ciaran may bitch about REQUIRED_USE's UI (which I knew going in was going to be problematic, just to be clear), but he's right on that front. ^^^ This point still needs addressing. Additionally, this needs to be thought out how it'll interact with eclasses; what stacking, etc. It doesn't cover the basics there to say the least. The proposal didn't cover eclasses at all. Is there a need to do so or are we chasing some kind of perfection based on filling all unused slots? Eclass stacking here matters; if it's stacked, it means ebuilds have to use out of bound (ie, other vars) to tell the eclass when it shouldn't mark a flag as runtime toggable. If it's not stacked by the pm, then they have to manually stack; that differs from the norm and makes it easier to screwup; however; does allow for them to filter, albeit a slight pain in the ass doing so. There's a choice there, and the answer matters, so yes, you should actually have a complete glep before trying to shove it up to the council and extract a vote out of them. Lest the intention is to just have them kick it back to the curb... Pretty much, this needs an implementation, partial conversion of the tree to demonstrate it. Just to prove that fricking point; if you had tried implementing this, a full investigation of what's involved alone, you'd have spotted that the core of the proposal is based on a wrong assumption. Portage doesn't write unfinalized DEPEND/RDEPEND/PDEPEND to the VDB. There's a footnote there, saying: The package manager has to ensure that all relevant information is stored in the installed package metadata. Frankly I don't fully buy that you were aware of this issue from the start of the proposal; the wording partially covers it however. Ddoesn't call it out, but via tha req it dumps it on the package manager developers heads to sort it- which already is the case. Binpkgs minimally weren't addressed which is why I still don't think this was actually spotted up front. Were it, the performance impact should've been mentioned, and quantified; con's are part of a normal glep. Meaning.. wait for it... writing a prototype is required to get those stats. ;) Based on the above I do expect the reference implementation would alsoneed to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. I expect, as a corollary to this, that a rebuild would be necessary if (on-disk-IUSE_RUNTIME xor in-ebuild-IUSE_RUNTIME) was non-empty (--newuse) or resulted in any flags that are in USE (--reinstall=changed-use). IMO this would be necessary to ensure the local ebuild copy and all related metadata for it gets updated in vdb. I think that's a common package manager logic and it's out of scope of the GLEP. Portage doesn't do physical updates last I saw- if it did, well, that's fairly dangerous. Only thing stored in the VDB is the ebuild and the environment dump, and the environment dump is what's ran from. You cannot sanely pick part the dump and try to intermix current ebuilds; rephrasing, I'll kick in the head anyone who tries that. What are you talking about? As far as I can see, we are talking here about *full rebuild*. Added the full quote
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposal
On Wed, Sep 26, 2012 at 08:58:54AM +0200, Micha?? G??rny wrote: On Sun, 16 Sep 2012 06:52:11 -0700 Brian Harring ferri...@gmail.com wrote: Keeping it short and quick, a basic glep has been written for what I'm proposing for DEPENDENCIES enhancement. The live version of the doc is available at http://dev.gentoo.org/~ferringb/unified-dependencies/extensible_dependencies.html One more question -- are we going to keep 'foo,bar?' syntax as a special case applying only to dependency atoms or are we going to extend it to USE flags? It's fairly dep specific; that's effectivelly foo|bar when you think about it; it multiplies out to dep:foo? ( that block ) dep:bar? ( that block ) in a dumb PM (smarter one just leaves the tree collapsed and filters as it goes). Phrased another way, I'm not sure we really need shorthand for the following: x? ( blah ) y? ( blah ) z? ( blah ) into x,y,z? ( blah ) It's a rare case; I could only foresee that potentially being of use for arch flags; ie, amd64,x86? ( blah ); which I'd write as arch:amd64,x86? ( blah ) personally since I don't like the notion of introducing ',' into raw, non use group flags. That said, I don't hugely care; people think it's useful, then have at it. ~harring
Re: [gentoo-dev] [RFC] Multiple ABI support through package appending/partial removal
On Wed, Sep 26, 2012 at 08:35:37AM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 13:12:56 -0700 Brian Harring ferri...@gmail.com wrote: On Mon, Sep 24, 2012 at 12:09:49AM +0200, Micha?? G??rny wrote: Hello, Since my previous idea of DYNAMIC_SLOTS proved too complex to design and implement, I would like to offer an another idea, based partially on what Ciaran mentioned. Before I start getting into details, I'd like to know your opinions, and what possible problems am I missing. To keep it clean, I will focus on Python ABIs but other languages and multilib could be handled in a similar manner. The problem === Right now, building packages for multiple Python ABIs is done using USE_EXPAND-based useflags. This is a working solution but it requires rebuilding the package for all ABIs whenever the chosen ABI list changes. While it may be not that important for most of the Python packages, it becomes such when it comes to things like boost or -- if we'd extend that to multilib -- say, llvm. In that case, whenever a newly-installed package requests a specific ABI, user has to spend twice as much time to rebuild the same version. The general idea While not getting too deep into ebuild syntax, the core part of the idea is to mark some of the USE_EXPAND variables 'special'. In this particular example, such a special flag group would be 'PYTHON_TARGETS'. Now, let's consider user installs a new package with one python_targets_python2_7 enabled. The package is built and installed like usual but aside to regular vdb files an additional file is introduced, listing all the installed files as 'belonging' to python_targets_python2_7. If user enables python_targets_python3_2 on the same package, the PM doesn't trigger a full rebuild. Instead, it builds the package with the new flag being the only flag in PYTHON_TARGETS. The new files are installed over the installed package (and added to CONTENTS in vdb), and the files in install image are listed in vdb as 'belonging' to python_targets_python3_2. What you're proposing would liter the ebuild/eclass with has_version checks; in brain dead simple cases, you can replace parts of the pkg as you're proposing there. However if it installs scripts, things start getting more complex; needs to vary how it installs if it's overlaying part of itself. That's the idea. You're given a tool, now thinking twice before using it. This statement doesn't make sense. Clarify it. This proposal also doesn't work in the phase of := slot deps either, not unless you've got a way to ensure, potentially weeks/months after the first build, that the node locks to the same slotting. A rebuild then? More logic to dump on the PM. Yes, you can hack around it- I pointed it out because the existence of that corner case says something about the proposal. Whenever files from two ABIs collide, package manager either replaces the installed files if the 'new' ABI is considered 'better' than the old one or preserves it. This follows the current behavior when multiple ABIs are built, and later builds overwrite files from earlier ones. This is handwavey and kind of crackadled; the PM has no way of knowing which USE_EXPAND target is considered 'best', so in the case of multilib (say 64b and 32b subversion) there isn't any way to which svn binary should be there- 64b or 32b. Best I can tell, your proposal winds up just being last one to merge wins which isn't acceptable. This is just an early idea. Details like precedence weren't converted into any syntax yet. Precedence is a core requirement of anything that's more than just a non lib; python would've already been required to be addressed if we didn't have that damn wrapper in place (when a multislotting proposal goes through, that wrapper should die in the process). At the point, the additional file contains something like (ugly pseudo-syntax): /usr/lib64/python2.7/foo.py python_targets_python2_7 /usr/lib64/python3.2/foo.py python_targets_python3_2 /usr/share/doc/foo-1.2.3/README.bz2 python_targets_python2_7 \ python_targets_python3_2 Now, if user requests disabling python_targets_python2_7 on the package, the package manager may not rebuild it as well. Instead, it removes python_targets_python2_7 from the above list, and unmerges the files which don't belong into any other ABI. If we're going to do sub-packaging... which is what you're attempting here... the VDB backend for it minimally cannot be a one off USE_EXPAND hack. That'll just back us into a corner- which the vdb already does quite heavily. Any subpackaging content tracking needs to be generic and usable. Really is that simple. Give a better
Re: [gentoo-dev] Addressing GLEP-62 itself
On Wed, Sep 26, 2012 at 02:38:02PM -0300, Alexis Ballier wrote: On Wed, 26 Sep 2012 03:29:17 -0700 Brian Harring ferri...@gmail.com wrote: On Wed, Sep 26, 2012 at 08:02:44AM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 12:54:39 -0700 Brian Harring ferri...@gmail.com wrote: On Tue, Sep 25, 2012 at 08:58:07PM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 14:47:33 -0400 Ian Stakenvicius a...@gentoo.org wrote: Based on the above I do expect the reference implementation would also need to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. Yes, unless I'm missing something that's the intent. I will re-read and update the GLEP a bit sometime this week. There's a fairly strong user interaction component here, along w/ potential nastyness for ebuilds (the proposal assume that a flag will be toggable in all cases within an ebuild if IUSE_RUNTIME specified; I guarantee instances where that fails can be found in the tree if a basic audit was done). Additionally, this *is* useless if it's done in a form the UI an't display/handle; Ciaran may bitch about REQUIRED_USE's UI (which I knew going in was going to be problematic, just to be clear), but he's right on that front. ^^^ This point still needs addressing. IUSE_RUNTIME is optional for PMs, why does the UI matter at all ? Also, the proposal doesn't assume flags are toggable at will, it assumes they are useflags and obey the same rules. I truly hate claims like this; it's optional, so who cares if it's whacky. Think through the proposal; for it to work reliably, it's not optional. Same issue I've been y'all over the head with, rendered/finalized vs raw/unfinalized deps being stored in the VDB. All managers have to write unfinalized if that proposal goes through, even if they don't support the optional toggling after the fact. As for the UI... arguing but it's optional! doesn't give a blank check for the UI angle. What the plan, more colorization and a new char for emerge -vp? Because we're kind of running out of chars there... It's a simple enough request; one that wouldn't even need to be made if there was code backing this proposal; on a related note, hell yes I'm wary of having this dumped on manager authors heads and having to be told sort out the mess/make it so. So I'm asking y'all to at least put in an equivalent time investment doing a quick prototype. This isn't an unreasonable request, and has been the norm for most gleps for a long while. Additionally, this needs to be thought out how it'll interact with eclasses; what stacking, etc. It doesn't cover the basics there to say the least. The proposal didn't cover eclasses at all. Is there a need to do so or are we chasing some kind of perfection based on filling all unused slots? Eclass stacking here matters; if it's stacked, it means ebuilds have to use out of bound (ie, other vars) to tell the eclass when it shouldn't mark a flag as runtime toggable. If it's not stacked by the pm, then they have to manually stack; that differs from the norm and makes it easier to screwup; however; does allow for them to filter, albeit a slight pain in the ass doing so. There's a choice there, and the answer matters, so yes, you should actually have a complete glep before trying to shove it up to the council and extract a vote out of them. Lest the intention is to just have them kick it back to the curb... It can't be stacked and it's not wise to do so; it is a simple bash variable like tons of others in eclasses: It cannot be stacked because y'all are trying to shove this in as an optional; unlike it's brother IUSE, which stacks. As for ons of others that don't stack; very few actually influence the package manager; ~14 roughly, minimally 5 of those stack (those that don't, basically aren't lists). Package managers not implementing this GLEP will consider the ``IUSE_RUNTIME`` variable as an irrelevant bash variable 2. introduce additional ``IUSE_RUNTIME`` variable listing names of USE flags related to optional runtime dependencies (without prefixes related to IUSE defaults). Treating bash variables as bash variables is rather the norm, stacking is the exception. As I understand it, your only objection here is that you want to see written 'IUSE_RUNTIME gets no special treatment' in the proposal ? My objection is punting it to the council till it's actually nailed down/sane; having them mark it accepted
Re: [gentoo-dev] Addressing GLEP-62 itself
On Tue, Sep 25, 2012 at 08:58:07PM +0200, Micha?? G??rny wrote: On Tue, 25 Sep 2012 14:47:33 -0400 Ian Stakenvicius a...@gentoo.org wrote: Based on the above I do expect the reference implementation would also need to change. I expect, for instance, that the PM's metadata-handling would need to occur as normal even though none of the package's phase functions would run, that is, *DEPEND (realistically RDEPEND as that should be the only one affected here, maybe PDEPEND too) and USE/PKGUSE would get updated. Since portage would not be re-emerging the package from the tree the original ebuild would remain. Yes, unless I'm missing something that's the intent. I will re-read and update the GLEP a bit sometime this week. There's a fairly strong user interaction component here, along w/ potential nastyness for ebuilds (the proposal assume that a flag will be toggable in all cases within an ebuild if IUSE_RUNTIME specified; I guarantee instances where that fails can be found in the tree if a basic audit was done). Additionally, this *is* useless if it's done in a form the UI an't display/handle; Ciaran may bitch about REQUIRED_USE's UI (which I knew going in was going to be problematic, just to be clear), but he's right on that front. Additionally, this needs to be thought out how it'll interact with eclasses; what stacking, etc. It doesn't cover the basics there to say the least. Pretty much, this needs an implementation, partial conversion of the tree to demonstrate it. Just to prove that fricking point; if you had tried implementing this, a full investigation of what's involved alone, you'd have spotted that the core of the proposal is based on a wrong assumption. Portage doesn't write unfinalized DEPEND/RDEPEND/PDEPEND to the VDB. Literally, USE=-x RDEPEND=x? ( dev-util/diffball ), the RDEPEND in the vdb is going to be . The retort is meh, that's minor; the fact this was missed doesn't inspire confidence; further, aside from the fact that writing finalized deps to the VDB ie, an accurate form of the deps- they're locked at that point, a smart PM can take advantage of that to avoid reading USE/IUSE till it's needed. This has measurable speed impact; if in doubt, rebuild a tree without it- that little trick I came up with first in pkgcore and it works on two fronts; 1) parsing is quicker, less nodes/smaller tree, 2) a smart PM can lazy load the USE/IUSE for performance gain. This is why portage/pkgcore were intentionally modified to do this; paludis may do it, haven't looked. My views, if the council is sane they'll slap it back down stating prove it's a viable solution; regardless of my views of the proposal (that it's the wrong way to solve it), it's not ready for an actual vote since approving it without actually verifying it could work makes things fucktons worse. I expect, as a corollary to this, that a rebuild would be necessary if (on-disk-IUSE_RUNTIME xor in-ebuild-IUSE_RUNTIME) was non-empty (--newuse) or resulted in any flags that are in USE (--reinstall=changed-use). IMO this would be necessary to ensure the local ebuild copy and all related metadata for it gets updated in vdb. I think that's a common package manager logic and it's out of scope of the GLEP. Portage doesn't do physical updates last I saw- if it did, well, that's fairly dangerous. Only thing stored in the VDB is the ebuild and the environment dump, and the environment dump is what's ran from. You cannot sanely pick part the dump and try to intermix current ebuilds; rephrasing, I'll kick in the head anyone who tries that. We used to do that shit; it broke, frequently. Env saving took a long time to get landed across the board- doing what you're proposing the PM should do breaks those guarantess and is a step back on that front alone. Basically, not happening, if it is, a series of bugs need filing. What portage does is that if it sees an ebuild in the tree that matches, it uses the deps from the tree; that's wholy different from what you think is happening in the vdb. Additionally, portage is the only PM that does this; the rest of the PMs treat the vdb ebuild as a constant- without extreme care, or some form of insane deptree comparison, you can't just swap in what a new ebuild has and expect it to work in all cases. ~harring
Re: [gentoo-dev] [RFC] Multiple ABI support through package appending/partial removal
On Mon, Sep 24, 2012 at 12:09:49AM +0200, Micha?? G??rny wrote: Hello, Since my previous idea of DYNAMIC_SLOTS proved too complex to design and implement, I would like to offer an another idea, based partially on what Ciaran mentioned. Before I start getting into details, I'd like to know your opinions, and what possible problems am I missing. To keep it clean, I will focus on Python ABIs but other languages and multilib could be handled in a similar manner. The problem === Right now, building packages for multiple Python ABIs is done using USE_EXPAND-based useflags. This is a working solution but it requires rebuilding the package for all ABIs whenever the chosen ABI list changes. While it may be not that important for most of the Python packages, it becomes such when it comes to things like boost or -- if we'd extend that to multilib -- say, llvm. In that case, whenever a newly-installed package requests a specific ABI, user has to spend twice as much time to rebuild the same version. The general idea While not getting too deep into ebuild syntax, the core part of the idea is to mark some of the USE_EXPAND variables 'special'. In this particular example, such a special flag group would be 'PYTHON_TARGETS'. Now, let's consider user installs a new package with one python_targets_python2_7 enabled. The package is built and installed like usual but aside to regular vdb files an additional file is introduced, listing all the installed files as 'belonging' to python_targets_python2_7. If user enables python_targets_python3_2 on the same package, the PM doesn't trigger a full rebuild. Instead, it builds the package with the new flag being the only flag in PYTHON_TARGETS. The new files are installed over the installed package (and added to CONTENTS in vdb), and the files in install image are listed in vdb as 'belonging' to python_targets_python3_2. What you're proposing would liter the ebuild/eclass with has_version checks; in brain dead simple cases, you can replace parts of the pkg as you're proposing there. However if it installs scripts, things start getting more complex; needs to vary how it installs if it's overlaying part of itself. This proposal also doesn't work in the phase of := slot deps either, not unless you've got a way to ensure, potentially weeks/months after the first build, that the node locks to the same slotting. Whenever files from two ABIs collide, package manager either replaces the installed files if the 'new' ABI is considered 'better' than the old one or preserves it. This follows the current behavior when multiple ABIs are built, and later builds overwrite files from earlier ones. This is handwavey and kind of crackadled; the PM has no way of knowing which USE_EXPAND target is considered 'best', so in the case of multilib (say 64b and 32b subversion) there isn't any way to which svn binary should be there- 64b or 32b. Best I can tell, your proposal winds up just being last one to merge wins which isn't acceptable. At the point, the additional file contains something like (ugly pseudo-syntax): /usr/lib64/python2.7/foo.py python_targets_python2_7 /usr/lib64/python3.2/foo.py python_targets_python3_2 /usr/share/doc/foo-1.2.3/README.bz2 python_targets_python2_7 \ python_targets_python3_2 Now, if user requests disabling python_targets_python2_7 on the package, the package manager may not rebuild it as well. Instead, it removes python_targets_python2_7 from the above list, and unmerges the files which don't belong into any other ABI. If we're going to do sub-packaging... which is what you're attempting here... the VDB backend for it minimally cannot be a one off USE_EXPAND hack. That'll just back us into a corner- which the vdb already does quite heavily. Any subpackaging content tracking needs to be generic and usable. Really is that simple. Sadly, this will not 'downgrade' common files to another ABI but I believe that it is not really a killer-feature. Installing new packages and upgrading existing == Whenever a new package is to be built and multiple ABIs are requested, the package manager should split the build process between particular ABIs. Preferably, it should build all of them one-by-one, recording the 'belongs' entries from the image and then install them as a single package. And how does the package know that it's being targetted at multiple ABIs? Your proposal is built on the assumption ebuilds will happy overlay themselves in differing configurations w/out ever fucking up. That's not the case frankly, and worse... for the cases where it doesn't fly, your proposal basically requires the PM to hide the we're building/installing your ass multiple times information from the ebuild, further compounding the issue. Whenever a package is to be
Re: [gentoo-dev] Suggest to specify a way to query for USEs in next council
On Sat, Sep 22, 2012 at 10:37:57PM +0200, Micha?? G??rny wrote:
On Sat, 22 Sep 2012 21:41:24 +0200
Pacho Ramos pa...@gentoo.org wrote:
Hello
This comes from:
http://www.gossamer-threads.com/lists/gentoo/dev/260536
In that one, we try to use the following:
has vala ${IUSE//+/} ! use vala return 0
Just please stop repeating the random broken snippet and use `in_iuse`
from eutils.eclass. That one is correct at least.
Inline that function into EAPI. We shouldn't have basic metadata
lookup issues like that reliant on an eclass to resolve it.
Exporting IUSE_FLATTENED or whatever the hell would be more than
enough.
~harring
[gentoo-dev] Re: [gentoo-pms] GLEP: gentoo sync based unified deps proposal
Pardon the delay; got busy with work, plus to actually address your claims re: labels (or refute, as I intend to do)... data was necessary. So I went and got the data. :) Analysis was done roughly 09/17 or so; just looping back and commenting now however. On Sun, Sep 16, 2012 at 05:59:21PM +0100, Ciaran McCreesh wrote: Your syntax also prevents the following: DEPENDENCIES=foo? ( $(make_foo_deps blah) ) Err, no it doesn't. I think you're reading too literally into the example mplayer translation I put in the doc- again, that was just a quicky, automated form, you can push dep:blah down beneath conditionals as necessary/desired. If you see something claiming otherwise, or implying otherwise in the glep, please tell me exactly where so I can fix the wording. The point is that nesting prevents composition. Labels are context insensitive, which allows groups of dependencies to be added anywhere, whereas dep: blocks can only be added if the surrounding groups are specified in a particular way. Fun fact; peoples usage of labels in exherbo is thus: build+run: set of deps run: set of deps/conditionals/etc You get the idea. Technically, each block is usually wrapped in ()... which frankly is a sign that the context switch third party code can introduce is problematic. Basically, y'all are already using labels in exactly the fashion I propose, just with different synax. I couldn't find a *single* instance of the following in usage in any git.exherbo repo: build+run: dep1 x? ( dep2 test: dep3 ) Ie, a nested override. Exherbo deps themselves basically dispute the claim that nesting somehow blocks people from doing composition; fact is, y'all can do nesting/context switching of a label w/in a block, but y'all don't at all. This strongly makes me think you're either blowing smoke up folks asses or that you're chasing perfection ignoring the realities of how this is used on the ground. Now, if there are actually examples of it in use in y'alls trees- (ones that aren't added after I send this email mind you ;) ) please point them out. Offhand, there's actually 4 pkgs that do stacking/nesting of deps, although it's accidental and doesn't actually do it for gain. They are: dev-haskell/language-c[=0.3.2]::haskell dev-haskell/language-javascript[=0.5.2]::haskell kde/kdemultimedia[~scm]::kde kde/kdemultimedia[=4.8.5]::kde basically DEPENDENCIES= ( build: some deps # dev forgot to close the block $(some dep_generator that forces a label immediately, per the norm) ) While the norm is DEPENDENCIES= ( build: some deps ) ( $(some dep_generator that forces a label immediately) ) Interesting sidenote btw; every usage I've found is directly translatable to my proposal, w/out any loss of expression in use. Now, it's possible I fucked up. I strongly doubt it however. Please provide examples either way- else y'alls own dependencies disprove your claims about nesting being evil. Either way, the data for that is at http://dev.gentoo.org/~ferringb/unified-dependencies/labels/translated-to-use-deps.txt 1) first, collapse dependencies down, than render the *DEPEND views, thus enabling easy and quick initial integration; effectively no impact on the api/functionality of the PM at this phase. Specification in terms of rendering has a huge problem, though. Remembering the crazy rules Gentoo has for || ( flag? ( ) ), what does this do? || ( dep:build? ( a ) dep:run? ( b ) ) Honestly, I was waiting for you to bring this up :) You're conflating two different things here; 1) someone being a dumb ass and writing what's effectively a || ( atom) block, just doing so in a manner w/out any reason to do so. 2) Your ongoing jihad against || (), specifically the occasionally valid complaint that build/rdepend different means the resolver can get stuck in certain pathways when slots are involved, abi, etc. Either way, in my proposal, I'm not going to single that out and try blocking it. The rendered version of it is still stable, albeit if it's build/run it's unlikely to be desired if there is ABI involved (for non ABI, specifically self-bootstrapping codebases, I suspect someone could come up with a valid construct- sed has something similar if memory serves). Worth noting, the following idiocy is valid: x? ( dev-util/diffball ) !dev-util/diffball Which is stupid, but syntactically correct. Nor is this a new issue, thus I don't particularly agree with your approach of trying to sink the proposal via an orthogonal problem. This is why we have QA tools. Ultimately, it comes down to the observation that the flag? ( ) syntax is strongly nested and hierarchical, but dependency roles aren't. There is a bit of truth in that views on flag? ( ) vs the random-ass context labeling (which is hierarchical- keep in mind your stack pushing/popping confusion). There's not any
Re: [gentoo-dev] A more natural (human-friendly) syntax for dependencies
On Sat, Sep 22, 2012 at 07:19:09PM +0200, Micha?? G??rny wrote: On Sat, 22 Sep 2012 10:05:41 -0700 Brian Dolbec dol...@gentoo.org wrote: On Sat, 2012-09-22 at 09:55 +0200, Micha?? G??rny wrote: Hello, The current dependency syntax: [VERSION-OP] PACKAGE-NAME [- PACKAGE-VERSION] suffers a few problems: 1. It is not really human-friendly. People don't say things like: I need newer than monkey-1.2. They say instead: I need monkey, newer than version 1.2. [snip :/ ] 4. It follows the syntax used by bash (for conditionals), pkg-config -- it is more natural in the environment. The BIG problem with that is bash has nothing to do with evaluating dependencies. All bash does is source the *DEPEND and pass the value to the package manager which does all the processing. And all 3 current package managers are set up to parse those dep strings with a set syntax and whitespace. None of the PM's dependency resolvers are written in bash, two are python based, one C++. This proposal would throw a big monkey wrench into parsing those strings. Introducing lots of bugs, both in the PM and the ebuilds. It has all to do with people writing ebuilds. Also, I don't really see a problem with parsing it. Bash is not really relevant here; Python and C++ doesn't have a problem with either syntax. It's just about correct tokenizer design. And this after all the fuss about the unified DEPENDENCIES proposal, which is a small syntax change for the current processing code, easily incorporated into the PM's. Err, no, it isn't. It requires redesigning ebuilds, cache, and probably a lot of code paths in the dependency parser unless the new syntax is going to be converted back to old one. It requires none of those things you claim. I know this since I actually wrote patches for it already. You should know this since you responded to that exact email, and have been very active in that thread. I laid out exactly how this plugged into our existing design and posted patches proving it. Phrased as nicely as I can, you're trolling, poorly at that. Please desist. Reiterating it again for others who weren't as active in that thread, patches are here: http://dev.gentoo.org/~ferringb/unified-dependencies/patches/ Mine is easily incorporated into the PM; it is just a change in a single place splitting and parsing the tokens. You really need to do a walk through portage internals. All ebuild based parsers are whitespace based parsing; the sole exception is for src_uri's - file rename, and by large that code is a bit icky for each; in portages case, doubly so frankly (pkgcore/paludis use a reversible iterator design in that case; less icky, but it throws off the normal flow ). This isn't a simple change for the PM; worse, it'll require two parse tree codebases since trying to inline your notion in would fuck the code up something fierce in terms of maintenence/flow. If you'd like to argue that, I'd suggest you take pkgcore and modify it for your intent (the parse code is pkgcore/ebuild/conditionals.py; it's the simplest but full implementation out there imo, thus the suggestion), while preserving parsing support for the old format; just the python side, you don't need to do the c extension. Post that patch and prove it; else... bluntly, you're arguing w/ authors about their own code. That usually isn't a scenario people win. Some authors may be retards, hell, I may be, but I'm not wrong on that section of code across the managers. AND has definite, measurable advantages. We still didn't get a single one. So, I think you just don't like it and are inventing disadvantages without even caring enough to consider them before writing. Just kicking the corpse to make damn sure it stays down, here's the flaws I saw pretty much immediately in reading the first email; if an hour was spent, I'm sure worse issues could be found. *) cocks up emerge invocation. Previously `emerge =dev-util/diffball-1`; your proposal, `emerge 'dev-util/diffball = 1'`. You argue well you already have to quote for or ; thing is, we don't have to quote for equals, and you now require it- globbed versions, same thing. *) If you argue emerge should just collapse the args if given `emerge dev-util/diffball = 1`, then I suggest *you* write that code, and maintain it. It'd take a strong council action before I did that in pkgcore. I think it's that level of wrong. *) The proposal completely overlooked profiles. We'd have to use the old form through out the entirety of the profiles. Woot, maintaining two dep syntax/parse trees indefinitely... * *) This requires changing the parsing rules of every single atom based file in the profile tree. Anything use related (package.use.mask) is whitespace split; not anymore it ain't. Code duplication baby, lots of it. *) The proposal ignores
Re: [gentoo-portage-dev] Try to specify how to get that a USE flag is present in current ebuild
On Fri, Sep 21, 2012 at 12:45:30PM -0700, Zac Medico wrote:
On 09/21/2012 12:08 PM, Pacho Ramos wrote:
Hello
This comes from this gentoo-dev thread:
http://www.gossamer-threads.com/lists/gentoo/dev/260536
In that one, we try to use the following:
has vala ${IUSE//+/} ! use vala return 0
as already done in many eclasses/ebuilds. The problem is that Ciaran
wants to forbid it because he says it's not specified in PMS. My
suggestion was to simply specify it as it's currently implemented in
portage because that functionality is (apart of needed) being used for a
long time in the tree by numerous eclasses/ebuilds, then, from my point
of view, wouldn't be any sense on lose time for moving them to current
functionality to a worse one, wait for the next eapi and, finally,
revert them back to current behavior.
The problem is that I cannot find any doc about how this is currently
handled in portage. Could you help me on it please?
That `has vala ${IUSE//+/}` thing should work for all versions of
*cough* negated defaults; you need a - in addition.
~harring
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposal
On Tue, Sep 18, 2012 at 10:25:51AM +0200, Micha?? G??rny wrote:
test depends: to specifically mark those dependencies that are only
needed for when the pkg is being tested; effectively ephemeral
build/run time depends that go away once testing is completed.
Does that mean that USE=test is going away somehow?
If you think it through, a test use flag still is needed in the cases
where the rdep itself would change if test was enabled; such a source
is fairy rare, but not always just someone being moronic- certain
cases to do testing, the tests need to reach in fairly deeply and
recompilation for compile vs test isn't exposed.
Also, could you please stop spreading FUD with your examples?
It's not FUD; it's rendered deps, and a demonstration of how they
collapse down naturally on their own regardless of how you generate
them.
Quite frankly, it's a fairly effective demonstration in my views, but
so it goes.
A quick
glance shows that what you have expanded there, a fairly reasonable
Gentoo dev will solve using:
RDEPEND=[common depends]
DEPEND=${RDEPEND}
[build only depends]
from diffball (under current EAPIs)
RDEPEND==sys-libs/zlib-1.1.4
=app-arch/bzip2-1.0.2
app-arch/xz-utils
DEPEND=${RDEPEND}
virtual/pkgconfig
becomes the following under the proposal:
DEPENDENCIES==sys-libs/zlib-1.1.4
=app-arch/bzip2-1.0.2
app-arch/xz-utils
dep:build? ( virtual/pkgconfig )
Suspect I may add that to the doc; it's a good example of the ground
level simple gains for devs inherent in the proposal- thanks for
helping improve it.
So if you really want to show some advantages, please compare it with
*real* code.
I think I'll take the risk, and assume people capable of discussing
DEPENDENCIES and vaguely knowledgable in the ebuild format will be
able to understand how their ebuilds will change; thus I'll skip that
request of yours.
A productive suggestion for you; you should go looking through the
tree finding cases where DEPENENCIES is a regression in form at the
shell level, or rendered deps level.
Should you manage to find something that's not contrived or
intentionally cracktastic, I expect people would be interested.
~harring
Re: [gentoo-dev] Re: [PATCH] eutils: Warn on built_with_use usage
On Tue, Sep 18, 2012 at 02:35:42PM -0600, Ryan Hill wrote: On Mon, Sep 17, 2012 at 08:45:22AM +0200, Ralph Sennhauser wrote: Almost all affected packages can be bumped straight to 4 anyway and so use the improved syntax. toolchain_src_compile: EAPI=0: count: 38 I'm not sure this can change any time soon. :/ Honestly, if it was knocked down to 50 left in the tree, I'd be fine with it. Once reaching that level, the following could be done: 1) mangling built_with_use adding a an explicit whitelist of callers, making everything else a die. 2) keep the existing /var/db/pkg format aware code in place. 3) Expand the function w/ manager aware code as necessary to deal w/ said managers potentially shifting to new vdb forms. I doubt ciaran will like #3, but if we can get it down to .15% of the tree at that point pragmatism should rule the day; as long as the PM has alternate ways to query the info without accessing the VDB directly (even if those ways are outside PMS), I'm fine with it, and I expect most folk would be. ~harring
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposal
On Tue, Sep 18, 2012 at 08:48:16AM +0200, hasufell wrote: I am unsure if that does or could solve the problem why GLEP 62 was created, meaning... would enabling the foo useflag after the package has been emerged trigger a remerge in the following example? DEPENDENCIES= dep:run? ( foo? ( dev-libs/foobar ) ) Just transfering over the discussion from IRC, tbh hadn't thought about it till you mentioned it since it has some potential flaws that aren't necessarily recoverable. Specifically, what happens if to enable dev-libs/foobar support, something has to be done at build time? Think about a systemd use flag, where the script just installs some configuration for systemd; that's not toggable. It's not obvious till you trace the implications through, but w/ those issues what you wind up with at that point is trying to classify use flags, ala glep62; see the past complete-ass-ripping of that proposal for why it doesn't fly. Just adding another; ebuild devs are completely up shit creek if the flag induces a build time effect in one spot, and controls optional deps in another section of the dep tree. If someone sees a way to make that work, have at it, although to be clear any such notion I'm intentionally leaving out of my proposal since I don't see a way to do it without an explicit dep labeling. ~harring
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposal
On Tue, Sep 18, 2012 at 06:04:51AM +0200, Arfrever Frehtes Taifersar Arahesis wrote: A potential dev-libs/dep package I assume this is a hypothetical package; if this is something out of your personal eapi/repo, please state so. might have valid use case for USE flags related to USE_EXPAND=DEP. Your suggested syntax for types of dependencies in DEPENDENCIES would conflict with these USE flags after implementing : delimiter for USE_EXPAND-related USE flags. Actually, that was both the intent, and I thought explicitly clear/documented; 'dep' would be a PM controlled namespace- as I'm pretty sure I stated in the doc, else in that email thread on the subject. Thus, yep, you got me, you can't create a USE_EXPAND/USE_GROUP named 'dep'. I very, very strongly doubt that anyone ever would come up with a scenario where this is required, and the alternative name is somehow worse. Please give examples. Also, you should keep in mind that w/ what I ultimately want for USE_EXPAND, we'd have a couple other namespace that couldn't be used by ebuilds/profiles. Top of the head, * arch; kind of a given, alternate addressing of x86 via arch:x86. Would be added purely for consistency, although iteration of the potential values would warrant the group existing. * use; same reasoning as arch, added for consistency so the consuming code doesn't have to special case things. * phase; intentionally reserved should we ever decide to do per phase restrict control (aka, turning userpriv off just for the test phase). * license; Now, this one I *am* spitballing a bit- I'm not proposing it, just frankly thinking out loud. If we had a license namespace there, we could potentially mask out certain deps if the user requested say pure bsd, or as a potential way to properly integrate in our existing bindist support; keep in mind if the group existed, we could use it in REQUIRED_USE also. Either way, you get the idea; it was explicit that in fixing use_expand, a few namespaces would be offlimits. I vote for a separate syntax for types of dependencies. A separate syntax, or keeping dep:build? from conflicting w/ someone wanting to use USE_EXPAND=DEP ? If you've got other critiques state them, else, while your opinion is yours, I doubt anyone is going to agree with you that it's a deal breaker. ~harring
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposas
On Tue, Sep 18, 2012 at 11:47:42AM +0200, Micha?? G??rny wrote:
On Tue, 18 Sep 2012 02:24:26 -0700
Brian Harring ferri...@gmail.com wrote:
On Tue, Sep 18, 2012 at 10:25:51AM +0200, Micha?? G??rny wrote:
test depends: to specifically mark those dependencies that are
only needed for when the pkg is being tested; effectively
ephemeral build/run time depends that go away once testing is
completed.
Does that mean that USE=test is going away somehow?
If you think it through, a test use flag still is needed in the cases
where the rdep itself would change if test was enabled; such a source
is fairy rare, but not always just someone being moronic- certain
cases to do testing, the tests need to reach in fairly deeply and
recompilation for compile vs test isn't exposed.
Yes, and sometimes we're doing 'use test'. I simply don't see how
adding a separate group of dependencies just for 'test' phase is going
to help us.
They fit just fine into build-time dependencies right now.
I'm going to assume you typo'd build-time into run-time; on the
offchance you've never written actual test code, to test the code you
have to *run* the results.
Simple example, portage doesn't need eselect nor logrotate, nor afaik
selinux or paxutils, till runtime since it doesn't test those
pathways.
A non-crap resolver can exploit that gap when it comes to
parallelization.
Just heading off an email from you, no, you cannot just stick it into
RDEPEND then.
If you did so, the test deps would be locked into the required runtime
graph for as long as the pkg was installed.
If in doubt of how that matters; trace the usage of gtest, nose, etc.
Nose is a good example additionally since a properly setup setup.py,
the pkg doesn't need nose for build- just strictly for test.
A quick
glance shows that what you have expanded there, a fairly reasonable
Gentoo dev will solve using:
RDEPEND=[common depends]
DEPEND=${RDEPEND}
[build only depends]
from diffball (under current EAPIs)
RDEPEND==sys-libs/zlib-1.1.4
=app-arch/bzip2-1.0.2
app-arch/xz-utils
DEPEND=${RDEPEND}
virtual/pkgconfig
becomes the following under the proposal:
DEPENDENCIES==sys-libs/zlib-1.1.4
=app-arch/bzip2-1.0.2
app-arch/xz-utils
dep:build? ( virtual/pkgconfig )
Err, shouldn't the first three deps be namespaced?
No.
Please read the glep, specifically the section basic rules.
Also, you come up with a valid criticism, valid point, etc, something
*worthwhile*, I'll respond. If it doesn't meet that criteria, assume
I won't respond (feel free to bitch to the council during whatever
vote occurs for this GLEP that I ignored your noise; it's a risk I'll
willingly take).
~harring
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposal
On Tue, Sep 18, 2012 at 11:38:50AM +0200, Ulrich Mueller wrote:
On Tue, 18 Sep 2012, Brian Harring wrote:
On Tue, Sep 18, 2012 at 10:25:51AM +0200, Micha?? G??rny wrote:
Also, could you please stop spreading FUD with your examples?
It's not FUD; it's rendered deps, and a demonstration of how they
collapse down naturally on their own regardless of how you generate
them.
Quite frankly, it's a fairly effective demonstration in my views, but
so it goes.
A quick
glance shows that what you have expanded there, a fairly reasonable
Gentoo dev will solve using:
RDEPEND=[common depends]
DEPEND=${RDEPEND}
[build only depends]
from diffball (under current EAPIs)
RDEPEND==sys-libs/zlib-1.1.4
=app-arch/bzip2-1.0.2
app-arch/xz-utils
DEPEND=${RDEPEND}
virtual/pkgconfig
becomes the following under the proposal:
DEPENDENCIES==sys-libs/zlib-1.1.4
=app-arch/bzip2-1.0.2
app-arch/xz-utils
dep:build? ( virtual/pkgconfig )
Which is longer than the original. ;-)
I see 5 lines in the first version, and 4 in the second. I also see
either someone who counted wrong, or basing that statement purely on
byte count (which is frankly arguing to argue on your part).
Either way, pretty sure your view is -1; I'll add it into the glep
along mgorny, skipping sniping like the above.
~harring
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposal
On Wed, Sep 19, 2012 at 12:53:09AM +0200, Micha?? G??rny wrote:
On Tue, 18 Sep 2012 23:06:19 +0100
Ciaran McCreesh ciaran.mccre...@googlemail.com wrote:
On Wed, 19 Sep 2012 00:01:21 +0200
Micha?? G??rny mgo...@gentoo.org wrote:
On Tue, 18 Sep 2012 22:37:19 +0100
Ciaran McCreesh ciaran.mccre...@googlemail.com wrote:
On Tue, 18 Sep 2012 23:34:29 +0200
Micha?? G??rny mgo...@gentoo.org wrote:
On Tue, 18 Sep 2012 22:08:43 +0100
Ciaran McCreesh ciaran.mccre...@googlemail.com wrote:
On Tue, 18 Sep 2012 23:06:06 +0200
Micha?? G??rny mgo...@gentoo.org wrote:
But didn't we already point out that we can't have them in
RDEPEND since they introduce conflicts?
You are missing a basic and important part of how dependency
resolution works: currently, cycles consisting purely of
RDEPENDs are ignorable.
So, what do we lose? If PDEP comes 'ASAP' officially, I believe
that we actually gain RDEPs which can be actually trusted.
ASAP is a weaker guarantee that RDEPENDs currently have --
RDEPENDs currently have the weakest guarantee necessary to ensure
that they can be trusted. It's also a useless guarantee, since
ASAP can be arbitrarily late.
And can't RDEPENDs be arbitrarily late if there is a cycle?
No. RDEPENDs have to be available when a package is used to satisfy a
dependency. That's the difference between an RDEPEND and a PDEPEND.
So, if a particular cycle prohibits RDEPENDs being fulfilled when
RDEPEND is needed to satisfy a dependency, we have a failure now,
correct?
Depends on the cycle, but yes.
Order of depdencies for this converation; depends is strongest,
rdepends is second, pdepend is weak. If a pkg both deps and rdeps on
something, for building (since that's the first op), dep obviously
trumps all other dep forms for that pkg.
pkg1 depends - pkg2 depends cycle, we're boned, unsolvable; use dep
toggling at best.
pkg1 rdepends - pkg2, pkg2 depends on pkg1, strictly speaking, we're
boned; pkg1 isn't considered 'usable' until pkg2 is considered
'usable'. This is the common case where use pdepend instead of rdep.
pkg1 rdepends - pkg2 rdepends; this is a contained cycle, and is
mergable.
Now if for pkg1 rdep-pkg2 rdep, pkg2 rdeps on pkg3 (which neds to
be built), which deps on pkg1 this too, is an unsolvable chain.
you can build pkg1 and pkg2, and even install them. But pkg3 cannot
be built until pkg1 is considered 'usable', which can't be be
considered usable till pkg2 is considered usable, which (take a guess
where I'm going with this) can't be considered usable until pkg3 is
considered usable.
Do we have that guarantee somewhere in the PMS?
It's not too hard to check in the doc yourself, ya know. ;)
relevant latex chunk:
\begin{compactitem}
\item Build dependencies (\t{DEPEND}). These must be installed and
usable before any of the ebuild \t{src\_*} phase functions is
executed. These may not be installed at all if a binary package is
being merged.
\item Runtime dependencies (\t{RDEPEND}). These must be installed and
usable befor the results of an ebuild merging are treated as usable.
\item Post dependencies (\t{PDEPEND}). These must be installed at some
point before the package manager finishes the batch of installs.
\end{compactitem}
keyword there is 'usable'. Wording could be expanded, but the core
notion is there- it just skips going over graph theory/resolver
guts/cycles since they're not explicitly a property of dependecy
types.
Feel free to write a patch expanding the wording htere...
~harring
Re: [gentoo-dev] Re: [PATCH] eutils: Warn on built_with_use usage
On Mon, Sep 17, 2012 at 08:45:22AM +0200, Ralph Sennhauser wrote:
On Sun, 16 Sep 2012 19:41:14 -0700
Brian Harring ferri...@gmail.com wrote:
On Sun, Sep 16, 2012 at 10:10:47PM -0400, Mike Frysinger wrote:
On Sunday 16 September 2012 03:51:04 Brian Harring wrote:
+ if ! has $EAPI 0 1 2 3; then
+ eqawarn built_with_use should not be used in
$EAPI; use USE deps.
+ elif has $EAPI 2 3; then
+ if [[ $hidden == yes ]] || $missing_was_set; then
+ eqawarn built_with_use in EAPI=$EAPI
without --missing or --
hidden
usage, should use USE deps instead. + else
+ eqawarn built_with_use should not be
used; upgrade to EAPI=4
instead
+ fi
+ fi
i'd do:
case ${EAPI:-0} in
# No support in these EAPIs, so don't warn.
0|1) ;;
# Maybe warn as some functionality exist.
2|3) [[...]] eqawarn ... ;;
# Assume EAPI=4 or newer where all functionality exists.
*) eqawarn ... ;;
esac
I'd be fine w/ it; worth noting, that was a 4am patch, so I'm not
claiming perfect implementatoin there. :)
My main focus here is switching built_with_use to actively nagging
people to stop using it; this includes nagging EAPI0/1 users of it.
Sans the implementation details, anyone got complaints with the
intent?
How about raising the EAPI baseline from 0 to 2 - ie. every package may
use EAPI 2; not the same as deprecating 0 1 - and do:
case ${EAPI:-0} in
0|1|2|3|4) eqawarn From date onwards this will die ;;
*) die ... ;;
esac
as EAPI 2 supports the --missing case via constructs as:
|| (
=foo/bar-1
foo/bar-1[baz]
)
I'd rather be more aggressive on this one, actually; either way, for
herds/devs, a full scan of the tree was done to identify what invokes
bulit_with_use whether directly, or indirectly via invoking a function
that does.
Resultant logs/tree is at
http://dev.gentoo.org/~ferringb/built_with_use-cleanup/ .
For the 'env' files that are nestled away in there, that's a pkgcore
dump of the environment of the ebuild post sourcing for ease of
tracing how built_with_use was actually invoked.
Sidenote, if in looking at the env dumps you see something that looks
like it shouldn't be saved, let me know- I'm generally pretty anal
about trying to ensure nothing pkgcore related is accessible by
ebuilds/eclasses, nor saved to the env.
Almost all affected packages can be bumped straight to 4 anyway and
so use the improved syntax.
~11% already are EAPI4, just triggered via eclass pathways.
Either way, herds, please take a look- the views should make it easy
for y'all to trace down the offenses and deal with them.
~harring
[gentoo-dev] [PATCH] eutils: Warn on built_with_use usage
At this point, the functionality built_with_use provides should
be covered near or more likely, in full, but USE deps in EAPI2 and
EAPI4; thus warn on usage.
While this may be a bit annoying, this is the only major consumer
left at this point that knows about /var/db/pkg layout; once that's
gone, alternative VDB formats can occur (meaning faster package
manager operations, or at least the potential if implemented sanely).
This patch is a rough first stab at the wording (english isn't
exactly my forte) to use for warning; better/clearer warning text
would be appreciated.
---
eutils.eclass | 13 +
1 file changed, 13 insertions(+)
diff --git a/eutils.eclass b/eutils.eclass
index 8bfc2bc..7f5a616 100644
--- a/eutils.eclass
+++ b/eutils.eclass
@@ -1196,7 +1196,9 @@ built_with_use() {
fi
local missing_action=die
+ local missing_was_set=false
if [[ $1 == --missing ]] ; then
+ missing_was_set=true
missing_action=$2
shift ; shift
case ${missing_action} in
@@ -1205,6 +1207,17 @@ built_with_use() {
esac
fi
+ if ! has $EAPI 0 1 2 3; then
+ eqawarn built_with_use should not be used in $EAPI; use USE
deps.
+ elif has $EAPI 2 3; then
+ if [[ $hidden == yes ]] || $missing_was_set; then
+ eqawarn built_with_use in EAPI=$EAPI without --missing
or --hidden usage, should use USE deps instead.
+ else
+ eqawarn built_with_use should not be used; upgrade to
EAPI=4 instead
+ fi
+ fi
+
+
local opt=$1
[[ ${opt:0:1} = - ]] shift || opt=-a
--
1.7.12
Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies
On Sun, Sep 16, 2012 at 09:56:27AM +0200, Micha?? G??rny wrote: But consider that for example Zac AxS (correct me if I recall it correctly) considered making changing the meaning of RDEPEND to install them before the build, thus effectively making 'build,run' useless. I really am not trying to be a blatant dick to you, but this has /zero/ relevance. RDEPEND means required for runtime. That ain't changing. If they were discussing changing what RDEPEND meant, then they were high, period. If zac/axs want to try and make the resolver install RDEPEND before DEPEND... well, they're free to. That doesn't change the fact that the deps still must be specified correctly; in short, build,run is very much relevant. What I suspect they were intending on doing is letting the resolver work on RDEPENDS of a pkg in parallel to that pkg being built; this is a parallelization scheduling optimization, still requires accurate deps. I'm trying to be nice here, but you're very confused on this matter. Total cache savings from doing this for a full tree conversion, for our existing md5-cache format is 2.73MB (90 byes per cache entry). Calculating the savings from the ebuild/eclass standpoint is dependent on how the deps are built up, so I skipped that. You're storing the cache in a tarball? Going to assume you're not trolling, and instead use this as a way to point out that this actually *does* matter, although it's admittedly not obvious if you don't know much about the guts of package managers, or don't spend your saturday nights doing fun things like optimizing ebuild package manager performance. First, the figure is 3.204MB if default context is used; ~9.5% of the content footprint for md5-cache specifically. Little known fact; rsync transfers for gentoo are required to be --whole-file; meaning no intra-file delta compression, it transfers the whole file itself. This is done to keep cpu load on rsync nodes low (else they'd be calculating minimally 97k md4's for every sync, not counting the rolling adl32 chksum for all content dependent on the window cut off threshold- sounds minor, but it's death by a thousand cuts). For obvious reasons, the cache is the hottest part of the tree due to cascading updates due to eclass changes. In other words, that ~9.5% reduction targest the core data actually transferered in a sync. In terms of the total tree footprint, it's a 1% reduction; mostly lost in blocksize overhead unless you're using squashfs (which a decent number of folks do for speed reasons), or use tail packing FS for the tree (again, more than you'd think- known primarily due to reiserfs corruption bugs causing some hell on PM caches). There's also the fact doing this means best case, 2 less inodes per VDB entry (more once we start adding dependency types). For my vdb, I have 15523 across 798 pkgs. 1331 of that is *DEPEND, converted to DEPENDENCIES the file count is 748. Note that's preserving DEPEND, although it's worthless at this stage of the vdb. So 5% reduction in files in there. Whoopy-de-doo, right? This one I can't test as well since the only rotational media I've got these days is a hardware raid w/ a beefy cache; the closest I can manage is local network nfs to an ssd FS, so it'll have to serve as a stand in for cold cache/hot cache, and for a demonstration of why having a backend that is a 101 small individual files is bad. Best of 5 is displayed below: Iterating over the vdb, and parsing and rendering all depends for our current layout, w/ the vdb stored on nfs: cold cache: real0m30.405s user0m1.046s sys 0m0.390s hot cache: real0m16.483s user0m0.883s sys 0m0.168s non-optimized, hacked to work (known slower for parsing in comparison to the non quicky hack), iterating over the vdb, parsing all depends and rendering said depends when it's stored as DEPENDENCIES; literally, rendering DEPEND from it, RDEPEND, PDEPEND. cold cache: real0m18.329s user0m0.908s sys 0m0.280s hot cache real0m12.185s user0m0.860s sys 0m0.128s You get the idea. See the various infamous cold cache/hot cache performance tests in doubt; I can tell you that a similar trick, done in '07, literally just skipping loading USE till it was needed for provides parsing was enough to bring a 5400RPM drive's run time down from 15s to 12s for cold cache- for parsing provides *alone*, nothing else. Either way, do your own investigation, it's a good education on performance. Hopefully for the others listening, that last section was a random but useful tidbit of info; if not, pardon, just being through to make sure this point is not raised again. ~harring
Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies
On Sun, Sep 16, 2012 at 01:21:26PM +0200, Micha?? G??rny wrote: On Sun, 16 Sep 2012 04:10:01 -0700 Brian Harring ferri...@gmail.com wrote: On Sun, Sep 16, 2012 at 09:56:27AM +0200, Micha?? G??rny wrote: But consider that for example Zac AxS (correct me if I recall it correctly) considered making changing the meaning of RDEPEND to install them before the build, thus effectively making 'build,run' useless. I really am not trying to be a blatant dick to you, but this has /zero/ relevance. RDEPEND means required for runtime. That ain't changing. If they were discussing changing what RDEPEND meant, then they were high, period. If zac/axs want to try and make the resolver install RDEPEND before DEPEND... well, they're free to. That doesn't change the fact that the deps still must be specified correctly; in short, build,run is very much relevant. I don't think we have made up our mind what *exactly* we want from deps. Are we now expecting deps to give us ponies or something? We know *exactly* what we want from deps, and their current definition- the problem isn't the definition, it's that we don't have the forms we need. Just because we have something semi-correct right now, doesn't mean that we don't want to change that. This is a no-op argument against the proposal: we can't change the deps because we might want to change the deps. It's also irrelevant due to the core basis of it being broken as fuck (described above). There's also the fact doing this means best case, 2 less inodes per VDB entry (more once we start adding dependency types). For my vdb, I have 15523 across 798 pkgs. 1331 of that is *DEPEND, converted to DEPENDENCIES the file count is 748. Note that's preserving DEPEND, although it's worthless at this stage of the vdb. So 5% reduction in files in there. Whoopy-de-doo, right? So we can modify vdb now? What about all those applications which obviously are broken due to that? You're misunderstanding the problem of the VDB. We cannot change the core structure of it- certain ebuilds currently expect to be able to find USE/IUSE at specific pathways. That's where we're blocked. Adding a new metadata key can, and has been done (defined_phases, properties, required_use, etc). Removing keys can be done. For efficiency, not writing a key if it's empty, can be and is being done. Basically, your retort again, is wildly off mark and misunderstanding the problem. I strongly suggest you go do some real research into the existing PMs, and the problems being discussed. Go read some code; you've minimally got 3 different codebases to work from, and that's not counting ML archives, tools, PMS, or the devmanul. If per you're statements, you're actually serious about writing your own PM, you're going to need to know this sort of shit /anyways/ to actually do the basic legwork of a PM, so please go do the necessary legwork if you want to persist in arguing about internals. ~harring
Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies
On Sun, Sep 16, 2012 at 03:39:22PM +0800, Ben de Groot wrote:
On 16 September 2012 09:20, Brian Harring ferri...@gmail.com wrote:
Dumps are at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/ .
Herds, if you want to see what your pkgs would look like, look at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/herds/ .
If you'd like to see an *example effect* it has on what gets displayed
to the user (aka, after all major use conditionals are stripped), look
at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/user-visible.txt
; warning, that's a 55MB file. The syntax in use there isn't great,
but as said, it's an example.
...
Additionally, the form used here makes *no assumption about default
context*; in any final solution we use, a default context would be
wise- say build,run. Again, an example of what I mean.
The dumps were regenerated; a default context of 'build,run' was
added. Basically in the absense of an explicit dep: targetting,
dep:build,run is assumed.
Essentially it makes the deps cleaner to read for the common case,
while also reducing the footprint at the cache, and vdb level; see
http://dev.gentoo.org/~ferringb/unified-dependencies-example/vdb-effect.txt
Thanks. I have given it a quick overview for the qt herd. I still
don't see what using DEPENDENCIES adds to what we do now with separate
*DEPEND variables. I see no convincing reason to change what we do.
Ok, so here's some stats: in the tree, we have 31360 ebuilds, and 194
eclasses; grand total of 31554 sources of metadata content (I say
metadata since vapier has eblits in use which are just phase
functions- I'm not scanning those intentionally).
Doing some simple scans of the tree, here's some stats; note these
stats are duplicated in the glep (they're nice selling points, thus
might as well):
1) 746 hits in the tree for COMMON_DEPEND; that's 2%, and the usages
I'm aware of have been for literally, what it sounds like- depends
that are both DEPEND and RDEPEND.
2) scanning for assignments of RDEPEND=.*|$\{?DEPEND\}? gets a hit on
5343 unique sources. Searching for the inverse gets a hit on 10008
unique sources. Meaning that right there, ~48.6% of the tree is
duplicating deps between the two forms. This puts us to 16083 unique
sources in the tree that would benefit in some form (~51%).
3) What's interesting about that 51% is the eapi groupings; in EAPI4,
the autosetting of RDEPEND=${RDEPEND:-${DEPEND}} was discontinued.
Roughly 50% of the initial 51% match is EAPI4; the rest are eapi0-3.
4) Again, keep in mind that the grep's in use above are single line
matches- I'm definitely missing some ebuilds, and for complex
dependencies that are appended/set by the eclass, likely missing a lot
of that too.
So... basically, people are already doing this manually with their own
intermediate vars.
Just a rough but mildly entertaining stat there's basically 8.38MB
worth of normalized, literal fricking dependency; using a crappy algo
(rather than a human doing the deps who can do it better), 37.4%
(3.1MB) of that is removed via going to dependencies. It goes
without saying that this would be a helluva lot less torturous on a
proper PM implementation that parses the tree once, and renders- let
alone repoman being able to avoid repeat scans of things it already
has examined.
Mind you, portage doesn't do that, but this would be good incentive to
do a proper tree. :)
As I've said before on IRC, we need a good costs/benefits overview.
Right now I only see costs (migrating ebuilds and eclasses) and no
benefits.
Offhand... I wouldn't be pushing for this if I didn't think this would
be a boon over all- both for devs, and PMs.
I think the actual 'cost' probably got lost in the noise of the
various flames; what I'm proposing is basically zero cost for devs, it
shoves the work to the PM, leaving the option for devs to do a more
fine grained form if they can.
I'm starting a seperate thread w/ a glep for this; I think you should
take a look at the exact details of the specification including how
DEPEND/RDEPEND/PDEPEND are handled in parallel to DEPENDENCIES (short
version: if existent, they're automatically folded into DEPENDENCIES
in my proposal); the end result is basically zero pain transition,
while enabling us to start getting gains.
Cheers-
~brian
Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies
On Sun, Sep 16, 2012 at 02:02:24PM +0200, Micha?? G??rny wrote: On Sun, 16 Sep 2012 04:49:21 -0700 Brian Harring ferri...@gmail.com wrote: On Sun, Sep 16, 2012 at 01:21:26PM +0200, Micha?? G??rny wrote: On Sun, 16 Sep 2012 04:10:01 -0700 Brian Harring ferri...@gmail.com wrote: On Sun, Sep 16, 2012 at 09:56:27AM +0200, Micha?? G??rny wrote: But consider that for example Zac AxS (correct me if I recall it correctly) considered making changing the meaning of RDEPEND to install them before the build, thus effectively making 'build,run' useless. I really am not trying to be a blatant dick to you, but this has /zero/ relevance. RDEPEND means required for runtime. That ain't changing. If they were discussing changing what RDEPEND meant, then they were high, period. If zac/axs want to try and make the resolver install RDEPEND before DEPEND... well, they're free to. That doesn't change the fact that the deps still must be specified correctly; in short, build,run is very much relevant. I don't think we have made up our mind what *exactly* we want from deps. Are we now expecting deps to give us ponies or something? We know *exactly* what we want from deps, and their current definition- the problem isn't the definition, it's that we don't have the forms we need. No, the problem is that we think we need more than we have now. Read what I wrote. we don't have the forms we need; a more proper statement is we don't have all of the forms we need. Please read what I write, rather than just responding blindly. You may have time to waste, but I don't, nor do the people on this ml need to see you respond 13 minutes after I send an email when you can't even be bothered to read the fucking content properly. Unless you're considering the whole point of this thread is cosmetics... then please leave that to Fedora or other people who are paid to change stuff just because they can. This isn't productive; frankly it's childish. Take it to the forums if you want to continue on tangents like this. Just because we have something semi-correct right now, doesn't mean that we don't want to change that. This is a no-op argument against the proposal: we can't change the deps because we might want to change the deps. It's also irrelevant due to the core basis of it being broken as fuck (described above). What I'm trying to say is that you're making a lot of noise about cosmetics while we haven't even agreed on what's supposed to be inside. So, are we introducing this obtuse syntax for three DEPEND variables, of which the third is almost never used? Reiterating the points again, and for the final time for you since you seem intent on riding the short bus for this particular subject: 1) This unifies the existing syntax down into a collapsed form. In doing so, there are measurable gains across the board for PM efficiency and rsync alone. 2) In unifying the syntax via reusing our /existing fucking syntax/, we formalize the adhoc common dependency assignments devs already are doing in the tree. 3) In moving to a unified syntax, it positions us to easily introduce new dependency types without introducing more redundancy. Easier to add new dep types, faster to add new dep types, more efficient in doing so in comparison to existing approaches, and done in a fashion that devs can reuse existing conditionals. 4) It is not exherbo's DEPENDENCIES. Meaning it is not label based. Meaning you do not need to knee-jerk attack it because of some notion it's ciaran based/related. Honestly, stop wasting my (and others time) and please read this email full and through, including the /full thread you're blindly responding to/ before responding again. There is no prive for having the fastest turn around time in responding to an email; not unless you consider a permenant /ignore and killfile addition to be a prize. ~harring
[gentoo-dev] GLEP: gentoo sync based unified deps proposal
Folks- Keeping it short and quick, a basic glep has been written for what I'm proposing for DEPENDENCIES enhancement. The live version of the doc is available at http://dev.gentoo.org/~ferringb/unified-dependencies/extensible_dependencies.html Wording fixes will occur, but the core concept shouldn't change (and if it does, per PEP standards, the changes will be noted/tracked). Please have a read through; it should be clear/concise as to why PM authors are pushing for a unified dependencies to be done, what we can get from it now, what it enables moving forward, and the direct cost to devs should this idea be implemented. Just to be absolutely clear, what I'm proposing is basically zero cost for devs- they can (and should when it's in their benefit) switch to the new syntax when it makes their job easier; it can be used in parallel to existing DEPEND/RDEPEND/PDEPEND both to ease transition/compatibility. In addition, a prototype portage patch has been cut for dependencies support (needs work/full validation), and a matching pkgcore one has been cut. I realize it's a complex subject; if you have questions, please feel to ask. Thanks, and pardon the ancillary/daft noise that has gone with this subject. ~harring
[gentoo-dev] Re: [gentoo-pms] GLEP: gentoo sync based unified deps proposal
On Sun, Sep 16, 2012 at 03:39:49PM +0100, Ciaran McCreesh wrote:
On Sun, 16 Sep 2012 06:52:11 -0700
Brian Harring ferri...@gmail.com wrote:
The live version of the doc is available at
http://dev.gentoo.org/~ferringb/unified-dependencies/extensible_dependencies.html
I think you're being a bit glib with your dismissal of the labels
parsing scheme. You've got the following problems, that labels don't
have:
You dismiss the dep:build? ( dep:run? ( ) ) problem as don't do that.
x? ( !x? ( dep ) ) already exists syntactically. dep:build? (
dep:run? ( blah ) ) is no different, so yes, I dismiss it- frankly
with prejudice since you keep bringing that example up but ignoring
that it's not a new issue (plus ignoring that it never occurs in the
wild because it's a self-solving issue).
Labels doesn't have this problem: it doesn't try to reuse an existing
syntax precisely because the existing syntax is extremely awkward for
this kind of thing.
Labels have a human comprehension problem, and require a fair amount
more work for the various parsers.
You may not agree on that view, but there seems to be some consensus
on that (as much as one ever gets in gentoo at least).
You say there's an implict build,run. This can't be an implicit
dep:build,run? ( ) using that syntax, however, since then you
wouldn't be able to put a dep:post? inside it by the above.
You're misunderstanding; you're thinking about it as to how exheres
labels are done which is basically
build+run: ${DEPENDENCIES}
One thing to keep in mind here; I don't mind making the parser do some
extra work if it's easier on devs to deal with (including
aesthetically; like it or not, dependencies is needed, and screaming
labels are the one true religion isn't going to get us to where we
need to go- pragmatic compromise will).
Either way, in the absense of an explicit dep context, dep:build,run?
is assumed. The parser/visitation implementation is admittedly more
complex, but it's not in the realm of 'hard'. This is assuming a
non-shit implementation mind you.
Either way, my focus here is on the human, as I've stated.
So again
you've got a special case to deal with simply because of the syntax --
your syntax doesn't correspond directly to the meaning of dependencies.
doesn't correspond directly to the meaning of dependencies is a
vague assertion; back that one up. I'm assuming you're complaining
that conditionals are being used (the but they're not USE flags! bit
which is a subjective view of the conditional namespace).
If that's not what you're referencing, then frankly I'll just go with
explain to me how this same critique doesn't apply to labels.
There's also the issue of what negations do at the top level...
Yeah, I did skimp on that one; technically speaking, negations aren't
required if they prove too much of a pain in the ass. Negation at the
top level could be interpretted two ways:
1) negating against all possible dep types; thus a !dep:build? would
be dep:post,run? . Too slick in my view, but who knows, othes may
think it straight forward.
2) Treat it as a negation of the implicit dep:build,run; meaning
!dep:build? would be dep:run?.
Unsure of which is preferably at this juncture.
You discourage grouping of dependencies by the use flag that selects
them (see xscreensaver in your examples). Logically, xscreensaver
related things belong together.
I do not discourage grouping. The example is an automated
conversion, as explicitly mentioned in the doc.
A secondary condensing of those deps I'll add to make clear it's
supported.
Your negation example also tries to
bring related dependencies together. But it's a mess. What if one of
the dependencies was a build dependency, and one a run dependency?
You'd end up with something silly like this:
dep:build,run? (
!dep:run? ( dev-util/diffball )
!dep:build? ( dev-util/bsdiff )
)
Ciaran, we use a fucking bash format. We lost the ability to block
silly idiocy long ago via that choice.
As said, 'x? ( !x? ( dep ) )' already exists, despite being dumb. The
beauty of it however is that the syntax rules themselves make it such
that it doesn't ever actually come up in real world usage- the dep
wouldn't be used fundamentally.
My intention is a syntax/format that is natural to the dev, and
doesn't force them to do silly shit. If they choose to do silly shit,
that's on their head; trying to explicitly ban all possible dumb uses
just makes the rules worse via increased complexity.
Also, just to be clear, this critique applies to exheres labels just
the same. That example, rewritten to drop the negations is thus:
dep:build,run? (
dep:build? ( dev-util/diffball )
dep:run? ( dev-util/bsdiff )
)
Yep. That's pretty dumb. But no less retarded than someone doing the
following in labels:
build+run:
build: dev-util/diffball
run: dev-util/bsdiff
Which I'll note isn't banned in your usage
[gentoo-dev] Re: [PATCH] eutils: Warn on built_with_use usage
On Sun, Sep 16, 2012 at 10:10:47PM -0400, Mike Frysinger wrote:
On Sunday 16 September 2012 03:51:04 Brian Harring wrote:
+ if ! has $EAPI 0 1 2 3; then
+ eqawarn built_with_use should not be used in $EAPI; use USE
deps.
+ elif has $EAPI 2 3; then
+ if [[ $hidden == yes ]] || $missing_was_set; then
+ eqawarn built_with_use in EAPI=$EAPI without --missing
or --
hidden
usage, should use USE deps instead. + else
+ eqawarn built_with_use should not be used; upgrade to
EAPI=4
instead
+ fi
+ fi
i'd do:
case ${EAPI:-0} in
# No support in these EAPIs, so don't warn.
0|1) ;;
# Maybe warn as some functionality exist.
2|3) [[...]] eqawarn ... ;;
# Assume EAPI=4 or newer where all functionality exists.
*) eqawarn ... ;;
esac
I'd be fine w/ it; worth noting, that was a 4am patch, so I'm not
claiming perfect implementatoin there. :)
My main focus here is switching built_with_use to actively nagging
people to stop using it; this includes nagging EAPI0/1 users of it.
Sans the implementation details, anyone got complaints with the
intent?
~brian
Re: [gentoo-dev] GLEP: gentoo sync based unified deps proposal
On Sun, Sep 16, 2012 at 07:32:39PM +0300, Alex Alexander wrote:
On Sep 16, 2012 4:55 PM, Brian Harring [1]ferri...@gmail.com wrote:
Folks-
Keeping it short and quick, a basic glep has been written for what
I'm
proposing for DEPENDENCIES enhancement.
The live version of the doc is available at
[2]http://dev.gentoo.org/~ferringb/unified-dependencies/extensible_depe
ndencies.html
Am I the only one who thinks that this dep:{build,...} thing looks
really ugly and is hard to read?
IMO simply removing the dep part would greatly improve things:
That 'dep' part isn't great, but it's added for a reason; to unify
with USE_EXPAND/use group intended syntax. There's a reference in
there to
http://www.gossamer-threads.com/lists/gentoo/dev/260069#260069 which
I'll formalize soon enough.
DEPENDENCIES=
:build,run? ( ... )
:run? ( ... )
For your suggestion, consider it if we *do* fxi USE expand- via using
the same namespace:setting form.
Using app-admin/mcollective ad an example, it's deps are thus:
DEPEND=ruby_targets_ruby18? ( dev-lang/ruby:1.8 )
ruby_targets_ree18? ( dev-lang/ruby-enterprise:1.8 )
RDEPEND=dev-ruby/stomp
ruby_targets_ruby18? ( dev-lang/ruby:1.8 )
ruby_targets_ree18? ( dev-lang/ruby-enterprise:1.8 )
Which, if USE_EXPAND targets were groupped, would go from this
ruby_targets_ruby18? ( dev-lang/ruby:1.8 )
ruby_targets_ree18? ( dev-lang/ruby-enterprise:1.8 )
dep:run? ( dev-ruby/stomp )
to this:
ruby:targets_ruby18? ( dev-lang/ruby:1.8 )
ruby:targets_ree18? ( dev-lang/ruby-enterprise:1.8 )
:run? ( dev-ruby/stomp )
s/:/@/ would also be interesting
Just a note; the character choosen was *intentionally* one that isn't
a valid use character. @ is a valid character due to linguas. See
the thread I referenced (ciaran's response, then my response).
Short version; to use @, we need use subgroups; thus
linguas@ca@valencia .
DEPENDENCIES=
@build,run? ( ... )
@run? ( ... )
DEPENDENCIES=
ruby@targets_ruby18? ( dev-lang/ruby:1.8 )
ruby@targets_ree18? ( dev-lang/ruby-enterprise:1.8 )
@run? ( dev-ruby/stomp )
Using equivalent syntax for mcollective.
I'm not a huge fan of dep:, and I'm a bit wary of a bare
@{run,test,whatever} since it carries with it an implicit this is
targetting the dep namespace.
That said, I'm not opposed to it- just as I said, I'm a bit wary at
first glance.
Comments?
~harring
Re: [gentoo-dev] Unified DEPENDENCIES concept
On Sat, Sep 15, 2012 at 11:06:01PM +1200, Kent Fredric wrote:
On 14 September 2012 10:17, Brian Harring ferri...@gmail.com wrote:
All you need is something in bash that can parse DEPENDENCIES and
populate *DEPEND , and the underlying guts could be done in
practically any language without requiring PM specific
implementations.
You've got it inverted; if any autopopulation is occuring, *DEPEND -
DEPENDENCIES is the sane form.
While it definitely *is* possible to render DEPENDENCIES down into
depend/rdepend (after all, the PM has to do exactly this for
resolution), that does /not/ mean doing it in bash is a good idea.
I'd really not want to try that using labels; using use conditionals
('dep:run,build? ( targets )') is frankly a bit easier imo, but still;
why do so unless one likes pain? It doesn't actually gain us
anything via missing the point of DEPENDENCIES.
The point of unified DEPENDENCIES var (regardless of the form) is
thus:
1) ability to specify common deps once, w/out having to use
intermediate vars/copy-pasting/etc. Think COMMON_DEPEND, and this
should make sense.
2) To shift to a form where adding new dependency targets is easy-
whether it be sdepend, fdepend, tdepend, or hdepend (or
ONE-RING-DEPEND to rule them all). This actually is rather important;
for the average 95% case, devs won't actually have to pay much
attention to those vars; but for those of us a bit further out (cross
compilation, heavy parallelization, etc) those depend forms are
becoming increasingly painful in their absense.
Basically, having devs specify DEPENDENCIES in ebuilds, which then an
eclass chunks out into DEPEND/RDEPEND misses the point of this; it's
doable, it's just not particularly sane imo.
The other way around, having *DEPEND automatically be collapsed into
DEPENDENCIES, however is very sane- it makes transition/compatibilty
for devs bloody simple, while structuring it so we can do further
enhancements.
~harring
Sure, but at least this makes it a viable proof-of-concept without
needing all the different PM's to implement the new spec first, and
due to not being EAPI bound when done this way, means you can just do
it and have it work both now and in the future.
And because of this experimental nature, you don't have to do *ALL*
the parsing in bash, you could make the eclass use some external code
to parse it and spit it out, and simply have the eclass depend on that
external program regardless.
I agree that long term, a Unified DEPENDENCIES implementation is the
way forward, but if you want to convince people, having something
which has been demonstrated and tested in a real world setting goes a
long way.
Honestly, QA would be well within their rights to kick anyone who did
this, *hard* in the shins.
I understand your notion- specifically proof of concept, show the
data, etc; I just think you've still got inverted, too focused on
trying to do it in bash.
To demonstrate the gain of this, we basically take the existing tree's
deps, and re-render it into a unified DEPENDENCIES form.
As for adding support to a PM, if we use the use conditional proposal
of mine, it's bloody simple- the PM already supports it, we just need
some minor USE_EXPAND adjustments.
~harring
[gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies
On Sun, Sep 16, 2012 at 12:03:36AM +0200, Micha?? G??rny wrote: On Sat, 15 Sep 2012 13:33:18 -0700 Brian Harring ferri...@gmail.com wrote: To demonstrate the gain of this, we basically take the existing tree's deps, and re-render it into a unified DEPENDENCIES form. But in order to do this, we first have to decide exactly what kind of dependencies do we want to have. Then convert the tree to a separate-variable form with new dependencies. Then we can compare it with the DEPENDENCIES form and decide which one is better. Funny you mentioned that, I just finished tweaking pquery to generate real world example unified dependencies; these *are* accurate, just to be clear. Dumps are at http://dev.gentoo.org/~ferringb/unified-dependencies-example/ . Herds, if you want to see what your pkgs would look like, look at http://dev.gentoo.org/~ferringb/unified-dependencies-example/herds/ . If you'd like to see an *example effect* it has on what gets displayed to the user (aka, after all major use conditionals are stripped), look at http://dev.gentoo.org/~ferringb/unified-dependencies-example/user-visible.txt ; warning, that's a 55MB file. The syntax in use there isn't great, but as said, it's an example. Total cache savings from doing this for a full tree conversion, for our existing md5-cache format is 2.73MB (90 byes per cache entry). Calculating the savings from the ebuild/eclass standpoint is dependent on how the deps are built up, so I skipped that. The algorithim used is fairly stupid, but reasonably effectively; essentially it intersects the top level of each individual type of dep, breaking out common groupings. In other words, it won't pick up this: DEPEND=x? ( dev-util/diffball dev-util/bsdiff ) RDEPEND=x? ( dev-util/diffball ) and convert it into thus DEPENDENCIES= dep:build,run? ( x? ( dev-util/diffball dep:run? ( dev-util/diffball ) ) ) Additionally, the form used here makes *no assumption about default context*; in any final solution we use, a default context would be wise- say build,run. Again, an example of what I mean. If we said in the absense of a context, the default is dep:build,run the following: DEPEND=dev-util/diffball dev-util/bsdiff RDEPEND=dev-util/diffball de-vutil/bsdiff x? ( sys-apps/pkgcore ) PDEPEND=dev-python/snakeoil would be: DEPENDENCIES= dev-util/diffball dev-util/bsdiff dep:run? ( x? ( sys-apps/pkgcore ) ) dep:post? ( dev-python/snakeoil ) The quicky algo I used assumes no default context, thus it writes this: DEPENDENCIES= dep:build,run? ( dev-util/diffball dev-util/bsdiff ) dep:run? ( x? ( sys-apps/pkgcore ) ) dep:post? ( dev-python/snakeoil ) Etc. ~harring
Re: [gentoo-dev] EAPI5: require ebuilds/eclasses to not use any vars/funcs prefixed with __
On Thu, Sep 13, 2012 at 01:53:21AM -0400, Mike Frysinger wrote:
On Wed, Sep 12, 2012 at 4:36 PM, Brian Harring wrote:
Currently, there is a minor amount of ebuild/eclass usage of things
named __*; ~90% of it is 'import once' eclass code like the following:
if [[ ${___ECLASS_ONCE_LIBTOOL} != recur -_+^+_- spank ]] ; then
___ECLASS_ONCE_LIBTOOL=recur -_+^+_- spank
Converting that is easy enough, and I'll be doing that work for
gentoo-x86 if folks don't have an issue.
pick a funner prefix and i approve
_LIBTOOL_ECLASS_GOT_SPANKED_ALREADY
Either that or one could just inject futurama quotes:
_LIBTOOL_ECLASS_DOESNT_WANT_TO_LIVE_ON_THIS_PLANET_ANYMORE
For instances where the sandbox needs to be turned off for a command-
we do the same thing we did w/ nonfatal;
sandboxless the command and args
fine idea
withoutsandbox. ;)
Joke aside, 'nosandbox' is more consistent; also, there's basically
one usage in the tree (virtualx) that would need it, although the
level of ebuild usage for that, I do not know- donnie might, that was
his code originally iirc.
~harring
[gentoo-dev] USE_EXPAND / USE 'configuration space' refactoring.
Hola.
CC'ing pms since obviously they should comment, although the
discussion should be on -dev (aka, public, not an alias).
Sorry, this is a long email; condensing it down into a glep is viable,
just my time is limited (it's telling I started the gentoo stuff at
2am, wrapping up likely at 4am).
The length of this email basically relates to the subtlety of this
proposal; at first glance it may appear as if I'm proposing a syntax
shuffle just for aesthetic reasons, but as elaborated in the tail end,
that subtle syntax change opens up a lot of possibilities.
Either way, what I propose, is pretty simple. Currently USE_EXPAND'd
groups are pushed into USE in the form ${use_group}_${setting}.
Thus VIDEO_CARDS=intel nvidia becomes video_cards_intel
video_cards_nvidia. This should be reasonably well known and
understood from the dev standpoint; from the PM side of things, this
actually seriously sucks ass; if we see video_cards_intel in an IUSE
somewhere, we don't know if it's a USE flag, or a use_expanded group;
to figure that out, we basically have to do a longest prefix matching
(or suffix tree).
There in is one of the subtle design flaws of it; two USE_EXPAND'd
targets, VIDEO_CARDS and VIDEO, can intersect in their namespace.
While we (devs) intentionally avoid this potential via careful name
choice, the fact the possibility exists is a bit of a sign that
USE_EXPAND was a bit of a hack.
What I propose is simple enough, and is for =EAPI5:
1) We disallow '@' in USE flags (yes, a use flag can actually have
'@' in it's name according to PMS; someone was hitting the crack
pipe pretty damn hard when they allowed that one). This doesn't
impact anything in gentoo-x86, nor any tree I've ever seen.
2) If EAPI5, we continue exporting video_cards_nvidia
(${use_group}_${setting}).
3) If =EAPI5, we export ${use_group}_${setting}, *and*
${use_group}@${setting}. While there may be complaints about having
it exported in two fashions, the reason for this is simple- it makaes
transition way the fuck easier, meaning we don't make devs wrie pain
in the ass if eapi x, else blah code. Basically, for eclass code,
the dev would do thus:
3.a) If the eclass supports EAPI5, they check for the old form;
${use_group}_${setting}.
3.b) If however that eclass is =EAPI5, they should check for the new
form; ${use_group}@{setting}.
4) At some point in the future, when either code involved has migrated
all to EAPI5, or we do something in a later EAPI that renders
supporting ${use_group}_${setting} untenuable, we stop exporting
${use_group}_${setting} for those EAPIs. Preferably with one EAPI
cycle w/ deprecation warnings.
Basically, this approach is zero impact for devs for the most part;
you don't have to do multiple paths for checking use wise if
supporting EAPI5 in parallel to others. Once your code is EAPI5 (and
above), you use the new syntax- or worst case, someone interested goes
and converts your =EAPI5 ebuild/eclass to use the new syntax on your
behalf.
Thus far, I've not particularly sold this notion in this email- I
focused on making damn clear this cleanup is basically no cost for
everyone but PM authors (and it's easy enough for us to handle).
Aside from the cleanup/sanity improvements to our algos for
backmatching USE_EXPAND groups, this syntax gets us actually some
rathre nifty shit; it basically allows us to introduce a generation
configuration space.
So... onto examples of why we should do this beyond the basic it's a
cleaner syntax/design argument:
--Per phase RESTRICT usage--
Exactly as it sounds. Consider userpriv- most of the time that's
turned off, that's purely because one phase- literally just one
phase- needs to do something via root. Thus we weaken the entirety
of the protection. Now, we could do some bastardly shit w/ existing
use_expand for this (which has some conflict potentials for the PM),
or via this syntax, we could do:
RESTRICT=phase@src_install? ( userpriv )
Now, not all restricts are phase level- fetch, strip, installsources,
etc, operate on a global level. userpriv, usersandbox, sandbox
(latter two aren't used in gentoo-x86, although PM's support it still)
are phase specific. That may seem like a small list, but consider
that
1) pkgcore has fakeroot code; it's had it for a while, I'm intending
on resurrecting that; this requires a RESTRICT='fakeroot' however.
2) I intend to start abusing the 'unshare' utility (man 1 unshare) to
contain ebuilds build process; basically gives us a way to level some
protections for the build host. I'd be very surprised if other PM's
don't follow suit (or do it before I do even); it's a simple
protection that's worth doing.
3) There's been repeated talk of using containers/overlays to
constrain the build process; sooner or later someones going to do it,
and you can be damn sure a restrict will be necessary.
Etc, you get the idea; the point of the new syntax is general
flexibility and
Re: [gentoo-dev] USE_EXPAND / USE 'configuration space' refactoring: bikeshedding the separator
On Thu, Sep 13, 2012 at 04:24:27PM +0100, Ciaran McCreesh wrote:
On Thu, 13 Sep 2012 03:39:19 -0700
Brian Harring ferri...@gmail.com wrote:
1) We disallow '@' in USE flags (yes, a use flag can actually have
'@' in it's name according to PMS; someone was hitting the crack
pipe pretty damn hard when they allowed that one). This doesn't
impact anything in gentoo-x86, nor any tree I've ever seen.
No crack pipe was involved in that decision! It's because of LINGUAS.
(Incidentally, we used : rather than @ for separation for exheres-0 for
that reason.)
Who says Linguas definition didn't involve a bit 'o crack? ;) On that
subject, not entirely sure how the hell a grepp'ing of profiles on my
part missed that file; worse, I distinctly recall this coming up in
the past. Suspect it's time we add a footnote to that section of
names since it's non-obvious. Sigh...
Two angles;
1) Mind you, the woken up/caffeine ratio currently blows so this
could be quite off kilter- but at first glance the '@' linguas usage
actually seems to map to sub use groups (both in intent and
grouping), at least for the quick scan I did of what we use. Might
not actually be an issue, iow if we allow that, although that's
assuming the '@' subgroup approach translates reasonably cleanly.
The potential failure I'd see with that approach is it's a bit reliant
on the assumption that the rules:
`language[_territory[.codeset]][@modifier]`
have been abided by- that the modifier is just that, a modifier.
Were we to do this, which, at least at first sight, seems like a nifty
solution that addresses it, we'd *likely* want sub groups to have a
rule such that if you try to expand the subgroup of a setting, and
that setting isn't a subgroup, it is considered 'on'. Basically, for
the linguas case, it kind of sucks if we get into the situation where
the consuming ebuilds/eclasses has to be sensitive to which modifiers
were exposed. Essentially:
linguas@de_DE@, if not a subgroup to expand to, is treated as scalar,
rather than list. Impliciations of that I've not yet fully thought
out- note that tweak isn't necessary for the basic notion of #1 to be
usable also.
2) bikeshedding potentials: just spitballing a couple of potentials
if '@' subgroups there doesn't fly for folks reading (mean it nicely,
we shouldn't diverge arbitarily, but in the same instant we shouldn't
import syntax/notions from exherbo unless they explicitly make sense
in the gentoo/PMS context; the formats diverge enough the reuse for
compatibility argument doesn't hold much water):
ruby_targets@ruby_18
ruby_targets:ruby_18
ruby_targets|ruby_18
ruby_targets(ruby_18)
Potentially fun thought, although the syntax is kind of ugly;
basically we treat 'ruby_target' as a matching target (restriction in
pkgcore vernacular, something that matches something else); the nice
thing about this is it naturally plugins into the notion of multiple
settings:
ruby_targets[ruby18]
ruby_targets[ruby18,ruby19]
In the same angle, while it's partially valid bash (and not in the
single setting case):
ruby_targets{ruby_18}
ruby_targets{ruby_18,ruby_19}
That said, I consider the [] and {} syntax absolutely freaking ugly to
the eye. I mention it should others think it's not butt-fugly.
If approach #1 doesn't fly, using ':' as the delimiter gets my vote.
~harring
Re: [gentoo-dev] Unified DEPENDENCIES concept
On Fri, Sep 14, 2012 at 07:18:54AM +1200, Kent Fredric wrote:
On 11 September 2012 14:16, Brian Harring ferri...@gmail.com wrote:
On Fri, Sep 07, 2012 at 04:14:17PM -0400, Ian Stakenvicius wrote:
Is there anything in particular in the spec/proposal for DEPENDENCIES
that would exclude the addition of individual build: app-cat/myatom
run: app-cat/myatom deps by an eclass or eclasses? I know the
goal here is to make things atom-centric, but I can't see an
implementation ever working of this that wouldn't permit the pile-on
of additional entries of different (or even the same) roles on
identical or near-identical atoms.
They could be piled on; it would require each eclass to reset the
label for safety reasons though; same goes for ebuilds frankly (or the
PM would have to reset the context to build+run: each time through).
Pardon if addressed elsewhere; this thread is a fucking mess...
~harring
Correct me if I'm wrong, but couldn't the entire proposition could be
implemented in an eclass, not needing the EAPI development cycle to be
tied up with it.
All you need is something in bash that can parse DEPENDENCIES and
populate *DEPEND , and the underlying guts could be done in
practically any language without requiring PM specific
implementations.
You've got it inverted; if any autopopulation is occuring, *DEPEND -
DEPENDENCIES is the sane form.
While it definitely *is* possible to render DEPENDENCIES down into
depend/rdepend (after all, the PM has to do exactly this for
resolution), that does /not/ mean doing it in bash is a good idea.
I'd really not want to try that using labels; using use conditionals
('dep:run,build? ( targets )') is frankly a bit easier imo, but still;
why do so unless one likes pain? It doesn't actually gain us
anything via missing the point of DEPENDENCIES.
The point of unified DEPENDENCIES var (regardless of the form) is
thus:
1) ability to specify common deps once, w/out having to use
intermediate vars/copy-pasting/etc. Think COMMON_DEPEND, and this
should make sense.
2) To shift to a form where adding new dependency targets is easy-
whether it be sdepend, fdepend, tdepend, or hdepend (or
ONE-RING-DEPEND to rule them all). This actually is rather important;
for the average 95% case, devs won't actually have to pay much
attention to those vars; but for those of us a bit further out (cross
compilation, heavy parallelization, etc) those depend forms are
becoming increasingly painful in their absense.
Basically, having devs specify DEPENDENCIES in ebuilds, which then an
eclass chunks out into DEPEND/RDEPEND misses the point of this; it's
doable, it's just not particularly sane imo.
The other way around, having *DEPEND automatically be collapsed into
DEPENDENCIES, however is very sane- it makes transition/compatibilty
for devs bloody simple, while structuring it so we can do further
enhancements.
~harring
[gentoo-dev] EAPI5: require ebuilds/eclasses to not use any vars/funcs prefixed with __
Hola folks.
Currently portage exposes a fair amount of it's internal
implementation via vars/funcs into the ebulid env; this frankly makes
it easier for ebuilds/eclasses to localize themselves to portage
(rather than PMS), leading to breakage.
Thus a proposal for EAPI5 has been made, banning ebuilds/eclasses from
using/accessing __, and requiring the PM to store it's internals in
that namespace.
Basically, instead of portage doing thus:
is_function dyn_pkg_preinst dyn_pkg_preinst
It does thus:
__is_function __dyn_pkg_preinst __dyn_pkg_preinst.
Aside from avoiding accidental conflicts/usage, the standardized
namespacing makes it a helluva lot easier to have repoman/qa tools
look for bad usage.
Currently, there is a minor amount of ebuild/eclass usage of things
named __*; ~90% of it is 'import once' eclass code like the following:
if [[ ${___ECLASS_ONCE_LIBTOOL} != recur -_+^+_- spank ]] ; then
___ECLASS_ONCE_LIBTOOL=recur -_+^+_- spank
Converting that is easy enough, and I'll be doing that work for
gentoo-x86 if folks don't have an issue.
Note there is a few vars we need to exempt; that list is currently
SANDBOX_* and FEATURES. FEATURES is fine to exempt from this rule.
For SANDBOX_*, while that's a PM internal, that's a bit of a grey
zone; regardless, we can actually address that via extending the
sandbox functions a bit:
addwrite [-r|--remove] pathway # for example, to do a removal.
For instances where the sandbox needs to be turned off for a command-
we do the same thing we did w/ nonfatal;
sandboxless the command and args
which is just
sandboxless() {
SANDBOX_ON=0 $@
}
or SYDBOX_ON=0 (or their equivalent var) for sydbox usage.
Comments?
~harring
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Tue, Sep 11, 2012 at 06:36:46PM -0400, Rick Zero_Chaos Farina wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 09/06/2012 02:50 PM, Micha?? G??rny wrote:
On Thu, 6 Sep 2012 09:49:13 -0700
Brian Harring ferri...@gmail.com wrote:
One additional thought- re: the scenarios where we don't fetch to an
intermediate location, then transfer that contents into ${WORKDIR},
while a better name is needed, something along the lines of
RESTRICT=fetches-into-workdir comes to mind.
Realizing this is a late response I would like to add Um, what?
src_fetch should only be putting things into /usr/portage/distfiles,
never into ${WORKDIR}, that's for src_unpack to handle.
Am I missing something here?
WORKDIR was an example; punting it directly into the pkgs distfiles is
also fine.
Basically, you're assuming that the content *is* cachable- cause if it
isn't, then dumping in ${DISTDIR} is wasteful (both since it'll
require a copy into the ebuild's workdir, and since it means
increasing crap accumulating in the workdir).
Further, there are cases where content *is* available, but is
fundamentally outside ${DISTDIR}- in cros, they have the
chrome/chromium source available in certain cases.
Now, either we can store 13GB int ${DISTDIR} that's a copy of that
external developer checkout, then copy that into ${WORKDIR}, or w/ a
restrict marker, copy that sucker into ${WORKDIR} directly.
Is it a corner case? Yep, but it's easy enough to deal w/- the only
constraint there is that the src_fetch's target dir isn't ${DISTDIR},
it's ${WORKDIR}, and the PM is required to preserve ${WORKDIR} from
src_fetch to the time of that pkg actually running.
~harring
Re: [gentoo-dev] HDEPEND (host dependencies for cross-compilation) for EAPI 5?
On Tue, Sep 11, 2012 at 7:13 AM, Michał Górny mgo...@gentoo.org wrote:
On Thu, 6 Sep 2012 01:11:45 -0700
Brian Harring ferri...@gmail.com wrote:
A compatibility hack that stacks them is strongly advisable;
something akin to the following:
Literally, we do the following:
inherit() {
if eapi blah; then
local DEPEND PDEPEND RDEPEND
usual saving/protection of DEPENDENCIES var
else
usual saving/protection of DEPEND/PDEPEND/RDEPEND vars
fi
normal sourcing machinery
if eapi blah; then
local _deps=( ) _x
for _x in DEPENDENCIES DEPEND RDEPEND PDEPEND; do
[ -n ${!_x} ] deps+=( ${!_x} )
done
[ ${#deps} -ne 0 ] DEPENDENCIES=${deps[*]}
unset DEPEND RDEPEND PDEPEND _x _deps
normal stacking/restoration of DEPENDENCIES rules
else
normal stacking/restoration of RDEPEND/PDEPEND/DEPEND
fi
}
Via that, we eclasses that are pure DEPENDENCIES eapi wise, just set
the DEPENDENCIES directly; those that have to support multiple eapi's
(aka, every fricking eclass that exists right now) can just use the
old form, shifting into the new form as things progress.
If we decide to go with a such a hack, then we either have to support
it indefinitely, or to decide to drop the support in some further EAPI.
It's a transitional hack; we support it as long as we need the
transition coverage. Meaning if by EAPI7, everyone is using EAPI5 and
up (or by EAPI7 it's basicaly insane to try and support
0,1,2,3,4,5,6,7), we drop the transition code in EAPI8. This is no
different to how RDEPEND/DEPEND autosetting was phased out.
If we go for the latter, then it's just delaying the ugly conditional
eclasses will have to suffer at some random point in the future.
Hate to say na uh, but... na uh. The point is as long as an eclass
has to support older EAPI's, they can use set DEPEND/RDEPEND and it
would work fine.
If the eclass needs the newer depends types (hdep, fdep, whatever) for
something, it levels an EAPI check, and then sets DEPENDENCIES.
Meanwhile, until it moves it's minimal EAPI to one requiring
DEPENDENCIES, they can use the old syntax and have it stacked.
Basically, either we can have git.eclass (for example) doing:
if has $EAPI 0 1 2 3 4; then
DEPEND==dev-util/git-1.6
else
DEPENDENCIES=dep:build? ( =dev-util/git-1.6 )
fi
Or, as long as it's suppporting EAPI's 0 1 2 3 4 and the transition is
still enabled for the actual EAPI it's being used in...
DEPEND==dev-util/git-1.6
Doing this means no eclass code has to maintain EAPI5 dependencies
code and EAPI5 depends/rdepends in parallel; they can transition to
pure DEPENDENCIES form at their own pace, dependent on their code, and
the EAPIs they support. At some point we state the transition hack
will be removed in the next EAPI- giving decent forewarning- then
remove it if it's not an undue dev burden. As for eclasses/ebuilds
that are purely =EAPI5, they convert to DEPENDENCIES syntax as they
go/have spare cycles.
Bluntly, this approach makes transition pretty painless for the vast
majority of the tree. In the grand scheme of things, this is actually
one of the simplest/cleanest migration EAPI will see I suspect.
There really isn't a sane argument to be made against this beyond
screw it, just make the devs convert immediately and damn the costs
(which I don't view as sane).
Either way, strongly get the feeling you're arguing purely because it
had the term DEPENDENCIES in it...
~harring
Re: [gentoo-dev] Unified DEPENDENCIES concept
On Fri, Sep 07, 2012 at 04:14:17PM -0400, Ian Stakenvicius wrote: Is there anything in particular in the spec/proposal for DEPENDENCIES that would exclude the addition of individual build: app-cat/myatom run: app-cat/myatom deps by an eclass or eclasses? I know the goal here is to make things atom-centric, but I can't see an implementation ever working of this that wouldn't permit the pile-on of additional entries of different (or even the same) roles on identical or near-identical atoms. They could be piled on; it would require each eclass to reset the label for safety reasons though; same goes for ebuilds frankly (or the PM would have to reset the context to build+run: each time through). Pardon if addressed elsewhere; this thread is a fucking mess... ~harring
[gentoo-dev] unifying use.mask/package.use.mask, use.force, package.use.force, etc
Hola folks. Currently, our if you needed to mask the use flag introspection globally, but allow it to be used for say app-crypt/gcr, the profile configuration would be the following: use.mask: introspection package.use.mask: app-crypt/gcr -introspection Frankly, this is a bit of a pain in the ass- it creates a shiteton of intermediate files, and it requires the dev to recall use.mask before package.use.mask among other things. I'd suggest instead just killing off the package.* content, and folding them into use.*; the proceeding content would thus be: use.mask: * introspection app-cryt/gcr -introspection Mind you, this isn't a new notion; actually pretty damn old (been in user configuration in one form or another for a long time). For whatever reason, it's yet to be proposed/pushed for in PMS, thus bringing it up now. Any major complaints? Specifically, collapsing: package.use.mask, use.mask - use.mask package.use.force, use.force - use.force package.use.stable.mask, use.stable.mask - use.stable.mask package.use.stable.force, use.stable.force - use.stable.force ~harring
Re: [gentoo-dev] Re: unifying use.mask/package.use.mask, use.force, package.use.force, etc
On Mon, Sep 10, 2012 at 01:28:23AM +, Duncan wrote:
Brian Harring posted on Sun, 09 Sep 2012 15:10:27 -0700 as excerpted:
[Current profile config to to mask the USE=introspection
globally, but unmask it for app-crypt/gcr]:
use.mask:
introspection
package.use.mask:
app-crypt/gcr -introspection
[Suggest killing package.* content, folding it into use.*]
use.mask:
* introspection
app-cryt/gcr -introspection
Specifically, collapsing:
package.use.mask, use.mask - use.mask
package.use.force, use.force - use.force
package.use.stable.mask, use.stable.mask - use.stable.mask
package.use.stable.force, use.stable.force - use.stable.force
You mention doing this for the profile.
Not 'mention' the proposal *is a profiles modification* in entirety,
nothing else.
Reorganizing the questions into user configuration (aka, the PM
authors choice of what they want to do in /etc/portage/profiles), and
PMS profiles (aka, what can be done in gentoo-x86).
user config first:
?? Would user's package.* and general make.conf settings remain the same?
That's the PM's decision. Knowing portage source, I'd expect it'll
wind up enabling that in parallel to the existing for it's user
configuration.
?? What about user's existing /etc/portage/profile overrides, if any?
Same response; that's the PMs decision. What Zac does, is what Zac
does.
?? And if they change, are you proposing a script that a user can run to
automate the process, or simply a news item pointing to the appropriate
gentoo profile upgrade documentation page, or ??
{ cat use.mask | sed -e 's:^:* :'; cat package.use.mask; } t \
mv t use.mask rm package.use.mask
Actual PMS profiles question:
?? Are you proposing the change be only for new profiles, eventually
deprecating the old ones, thus having the PMs (and devs maintaining
profiles) support both methods for awhile much as the cascading profiles
migration was handled?
Profile nodes are EAPI versioned; what I want, is effectively =EAPI-5
(or 6 if people get cranky about minor mods), is this to be the
default in new EAPI profile nodes.
Meaning if someone uses a EAPI5 profile node in gentoo-x86, they don't
use package.use.mask, they use the syntax I mentioned, and do it
within use.mask.
There will not be maintaining in parallel; there isn't a point.
Basically, this is a ~70 file reduction of profiles; 253 - 184
roughly. Aside from less general IO (not much, but some), it also
kills of the question for devs as to which comes first; use.mask or
package.use.mask. Ordering of the file would rule in the new scheme.
Frankly if we wanted to, we could push this further; use.force and
use.mask basically operate in a ternary space: -1, 0, +1; meaning
Forced off, configurable, and enforced on; respecitively such a
syntax would be -use, use, +use.
Thus the following:
use.mask:
blah
-monkeys # Note this is a negation of the parents masking of 'monkeys'
-foon # Negation of the parents masking
package.use.mask:
dev-util/bsdiff foon
use.force:
x
-y # negation of the parents forcing
monkeys # Note we're explicitly forcing monkeys on after reversing
# masking
package.use.force
dev-util/diffball strip
that translates out to basically thus:
use.enforced:
# This globally disables 'blah', globally forces 'monkeys',
# and resets 'foon' back to user controllable.
* -blah +monkeys foon
# Disable blah for dev-util/bsdiff.
dev-util/bsdiff -foon
# Force strip for dev-util/diffball.
dev-util/diffball +strip
The benefit of such an aproach is that
1) I'd argue it's easier on the dev for processing it- just proceed
top down, for the pkg in question considering if each leading
restriction (* or the atom) matches, then applying the resultant
settings.
2) This actually is basically what the PM does now, although it does
so via grabbing the content from multiple files.
In terms of performance gain, couldn't say frankly; pkgcore already
does a variant of this, re-rendering each cat/pkg restriction set so
as to avoid having to do the resolution multiple times (this helps
repoman/pcheck speed for example).
That said I'm not sure if folks would go for the ternary usage there.
Where that scheme to be applied to our profiles, it'd knock the inode
count for these files down to 127. Not the driving reason for it, but
a nice benefit.
~harring
Re: [gentoo-dev] HDEPEND (host dependencies for cross-compilation) for EAPI 5?
On Wed, Sep 05, 2012 at 12:06:45AM +, Jorge Manuel B. S. Vicetto wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 31-08-2012 20:46, Ciaran McCreesh wrote:
snip
Also, we're getting rather a lot of *DEPEND variables here... If
we're making people make major changes to their deps, which for
HDEPEND we definitely would be, then the it's expensive since
people would have to redo their deps argument against a combined
DEPENDENCIES variable goes out of the window, so we should rethink
that too.
I have to agree with Ciaran, instead of multiplying DEPEND variables,
it's probably time we move to a single DEPENDENCIES variable.
Personally, my complaints re: it are that 1) while minor, the
labels in some cases are overly verbose; recommendations instead of
recommends, suggestions instead of suggests, etc. 2) An actual
flaw in their design (imo): it tries to intermix two different
forms of parsing, without any real justification for *why* beyond
*hey look kids, I can!*; The two can intersect in slightly fucked
up ways, case in point:
DEPENDENCIES=
run+build:
cat/the
x? ( cat/cow
test:
y? ( cat/says
z? ( cat/moo
)))
Now, there may be some unstated rules that disallow that, but if that
*is* allowed, that's frankly dumb. As to if it's disallowed, it's
kind of a design flaw that the situation can occur requiring an
explicit suppression.
Rather than invent and try intermixing a secondary form, just using
the existing strikes me as saner; either we can have a specific
use_expand group like thus:
DEPENDENCIES=
dep_run? ( cat/monkeys )
dep_run+build? ( cat/foo )
Or, preferable imo, do away w/ the +, use a more natural ',' for phase
separation, and use ':';
DEPENDENCIES=
dep:run? ( cat/monkeys )
dep:run,build? ( cat/foo )
Doing it that way reuses the existing parsing infrastructure (good)
via just requiring a change to the use validation machinery (easy if
the PM is implemented sanely).
It also is able to express things that exheres variation can't do as
cleanly; considering build/fetch/post/run/test as the viable dep
targets:
DEPENDENCIES=
build+fetch+post+test:
some-dep
vs
DEPENDENCIES=!dep:run? ( some-dep )
I don't much expect that to occur, but the potential exists, thus
mentioning it.
One unstated fault re: DEPENDENCIES btw, is it will not play nice w/
exactly one of blocks. Treating '^^' as exactly one of, consider:
DEPENDENCIES=
^^ (
run:
cat/blah
build:
cat/dar
cat/foon
)
Is that a stupid dep? You bet your ass it is.. But it would have to
be explicitly suppressed by the parser for any such construct- moreso,
repoman would have to spot it which is slightly unfun.
Finally, one note; while certain folk have been making lots of noise
about DEPENDENCIES being the best thing since sliced bread, their
isn't much comment about how one actually transitions to it without
making eclass authors who have to support both pre-DEPENDENCIES, and
post-DEPENDENCIES eapis happy; kind of swiss cheeses the hell out of
the code frankly.
A compatibility hack that stacks them is strongly advisable; something
akin to the following:
Literally, we do the following:
inherit() {
if eapi blah; then
local DEPEND PDEPEND RDEPEND
usual saving/protection of DEPENDENCIES var
else
usual saving/protection of DEPEND/PDEPEND/RDEPEND vars
fi
normal sourcing machinery
if eapi blah; then
local _deps=( ) _x
for _x in DEPENDENCIES DEPEND RDEPEND PDEPEND; do
[ -n ${!_x} ] deps+=( ${!_x} )
done
[ ${#deps} -ne 0 ] DEPENDENCIES=${deps[*]}
unset DEPEND RDEPEND PDEPEND _x _deps
normal stacking/restoration of DEPENDENCIES rules
else
normal stacking/restoration of RDEPEND/PDEPEND/DEPEND
fi
}
Via that, we eclasses that are pure DEPENDENCIES eapi wise, just set
the DEPENDENCIES directly; those that have to support multiple eapi's
(aka, every fricking eclass that exists right now) can just use the
old form, shifting into the new form as things progress.
Either way, the dual parsing required for exheres version I'm -1 on;
I'm generally wary of NIH modifications, but the form I've mentioned
that reuses our existing machinery is +.5...ish... from my standpoint
(+1 on the form, just kind of 'meh' on the single var angle despite
mostly agreeing w/ the reasoning).
Either way, on w/ the flaming/insinuations of
idiocy/counter-insinuations of being a wee bit too friendly w/
sheep...
~harring
Re: [gentoo-dev] supporting static-libs
On Mon, Sep 03, 2012 at 10:54:15PM +0200, Maciej Mrozowski wrote: On Tuesday 28 of August 2012 02:15:40 hasufell wrote: Is there a reason not to support static-libs in an ebuild if the package supports it? It seems some developers don't care about this option. What's the gentoo policy on this? Isn't this actually a bug? A little remark. For CMake controlled buildsystem (as you're coming here from latest dev- games/simgear), there's no automatic way of building both static and shared libs (simgear allows to choose just one). This is why I removed static libs support that you proposed for dev- games/simgear (similar to ruby eclass abi handling - manually calling phases twice to build package 1st as shared, 2nd time as static). This is what I called not worth the effort in private discussion with you, not quite I don't care for static libs :) Guessing in the worst case, you can do a double compile to get around this, no? And yes, that's freaking horrible, just verifying cmake isn't doing something special that blocks it. Is upstream doing anything about this, or is it not on their radar/list-of-things-they-care-about ? ~harring
Re: [gentoo-dev] Re: RFC: PROPERTIES=funky-slots
On Mon, Sep 03, 2012 at 02:08:58PM +, Mark Bateman wrote: Patrick Lauer patrick at gentoo.org writes: On 06/23/12 21:21, Ciaran McCreesh wrote: There's been a move towards using slots for clever things that don't fit the traditional way of how slots worked. Examples include the new gtk2 / gtk3 handling and Ruby gems virtuals. Aside from being abusive, No, it solves a real problem. this screws things up for Paludis users. -EDONTCARE, use a supported package manager So if the packagemanager is struggling to resolve whether a package belongs in a slot or not, would this be a case for encoding such metadata in the ebuild filename. foo-slot2-3.2.1.ebuild This way the PM would be able to determine exactly what it has todo before it starts to parse the ebuild No; the problem isn't getting the slot out of the metadata (moving it to the file name doesn't really do anything beyond make it slightly faster at the cost of being backwards incompatible for any existent PM that sees it); the problem is in the PMs resolver, and how it chooses to search the space. Basically, paludis does x, the rest do y; funky-slots was intended to make 'x' behave better at the cost of ebuild devs having to go mark shit up (leading to the retort 'do y instead'). ~harring
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
On Wed, Sep 05, 2012 at 12:07:22PM +0100, Ciaran McCreesh wrote:
On Wed, 5 Sep 2012 13:00:05 +0200
Micha?? G??rny mgo...@gentoo.org wrote:
I guess that's a pretty comprehensive we need to do this properly
then.
Did I say we don't need to? We have the two eclasses which need to do
this properly, right? So what's your problem?
The problem is that we need to work out how to deal with this at least
for Subversion, and probably for CVS too... As much as we'd like to, we
can't just roll out a src_fetch without eclass changes. This doesn't
appear to be a trivial feature to provide.
You're conflating the required phase/hook w/ existing bad
implementations; they're separate.
The bad implementations can't use the hook till they sort out their
mess; straight forward enough.
Worth noting, last I looked, git eclass actually didn't do this right
either; basically merges the namespace of each remote/fetch source
into the local namespace, no prefixing. Fixable, but the issue mostly
comes about because of the fact we do *not* have a src_fetch in the
first place.
Either way, if the hook was in place, I expect the eclass level issues
to be sorted shortly after; right now there isn't a reason to do that
work (chicken/egg).
Consider that my +1 for src_fetch also, although FDEPEND is needed or
dependencies; I don't care which, getting a src_fetch has been on my
todo for chrome-os for a while now.
One additional thought- re: the scenarios where we don't fetch to an
intermediate location, then transfer that contents into ${WORKDIR},
while a better name is needed, something along the lines of
RESTRICT=fetches-into-workdir comes to mind.
Basically that restriction would be interpretted as $WORKDIR must be
setup/preserved from invocation of src_fetch to actual building.
Via that restrict, both scenarios should be addressed in full.
~harring
Re: [gentoo-dev] [Future EAPI] src_fetch() phase function to support VCS fetching
Yes. The manager can still parallelize prefetching, only consuming a build
job slot post fetch.
On Sep 6, 2012 11:49 AM, Michał Górny mgo...@gentoo.org wrote:
On Thu, 6 Sep 2012 09:49:13 -0700
Brian Harring ferri...@gmail.com wrote:
One additional thought- re: the scenarios where we don't fetch to an
intermediate location, then transfer that contents into ${WORKDIR},
while a better name is needed, something along the lines of
RESTRICT=fetches-into-workdir comes to mind.
Basically that restriction would be interpretted as $WORKDIR must be
setup/preserved from invocation of src_fetch to actual building.
Via that restrict, both scenarios should be addressed in full.
Does separate src_fetch() provide any benefit in that scenario?
--
Best regards,
Michał Górny
Re: [gentoo-dev] EAPI usage
On Sep 6, 2012 10:18 AM, Michael Orlitzky mich...@orlitzky.com wrote: On 09/05/2012 05:29 PM, Brian Harring wrote: Yes, I stated it because I view it as useful/sane. and isn't a compromise at all. I think you're mistaken in assuming a compromise is the required outcome of this. Given the choice between something productive, and something not productive, you don't choose the quasi-productive solution. From a developer's perspective, it's obviously better to be able to do whatever you want. But for users it'd be nice to be able to request a bump to EAPI5 and not get told to buzz off. Some people are unhappy with the current situation or this thread wouldn't exist. A good compromise should make everyone just a little bit unhappy =) Open source is built on scratching your own itch. As I said, you want eapi5 for user patching, either you're on the devs prioritization, or you do it yourself. You may not like that fact, but that *is* reality- filing nagging tickets isn't really going to help (more likely to piss people off in the same way zero-day tickets do). Suggest you put effort towards eapi5 rather than this thread; the thread isn't productive any longer (arguing the point when people have said no in full force is pointless). ~harring
Re: [gentoo-dev] python-distutils-ng.eclass: allow useflag dependencies for python
On Thu, Sep 06, 2012 at 10:44:34PM +0200, hasufell wrote:
I am missing a replacement for PYTHON_USE_WITH.
Would the attached patch help in any way? Review? Other ideas?
--- python-distutils-ng.eclass
+++ python-distutils-ng.eclass
@@ -59,6 +59,25 @@
# Set to any value to disable automatic reinstallation of scripts in bin
# directories. See python-distutils-ng_src_install function.
+# @ECLASS-VARIABLE: PYTHON_USE
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# comma seperated list of useflags needed for python
+# this is directly substituted into dev-lang/python[${PYTHON_USE}] thus
allowing
+# all valid useflag dependencies
+# example 1: PYTHON_USE=xml,sqlite
+# example 2: PYTHON_USE=xml?,threads?,-foo
+
+# @ECLASS-VARIABLE: JYTHON_USE
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# same as PYTHON_USE just for JYTHON
+
+# @ECLASS-VARIABLE: PYPY_USE
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# same as PYTHON_USE just for PYPY
+
EXPORT_FUNCTIONS src_prepare src_configure src_compile src_test src_install
case ${EAPI} in
@@ -107,16 +126,22 @@
fi
unset required_use_str
+# avoid empty use deps
+[[ -n ${PYTHON_USE} ]] _PYTHON_USE=[${PYTHON_USE}] || _PYTHON_USE=
+[[ -n ${JYTHON_USE} ]] _JYTHON_USE=[${JYTHON_USE}] || _JYTHON_USE=
+[[ -n ${PYPY_USE} ]] _PYPY_USE=[${PYPY_USE}] || _PYPY_USE=
Ick.
_PYTHON_USE=${PYTHON_USE:+[${PYTHON_USE}]}
_JYTHON_USE=${JYTHON_USE:+[${JYTHON_USE}]}
_PYPY_USE=${PYPY_USE:+[${PYPY_USE}]}
+
+# set python DEPEND and RDEPEND
for impl in ${PYTHON_COMPAT}; do
IUSE+= python_targets_${impl}
dep_str=${impl/_/.}
case ${dep_str} in
python?.?)
- dep_str=dev-lang/python:${dep_str: -3} ;;
+ dep_str=dev-lang/python:${dep_str: -3}${_PYTHON_USE}
;;
should I ask wth the space is for?
jython?.?)
- dep_str=dev-java/jython:${dep_str: -3} ;;
+ dep_str=dev-java/jython:${dep_str: -3}${_JYTHON_USE}
;;
pypy?.?)
- dep_str=dev-python/pypy:${dep_str: -3} ;;
+ dep_str=dev-python/pypy:${dep_str: -3}${_PYPY_USE} ;;
*)
die Unsupported implementation: ${impl} ;;
esac
Re: [gentoo-dev] EAPI usage
On Tue, Sep 04, 2012 at 09:03:55PM -0400, Michael Orlitzky wrote: On 09/04/2012 05:06 PM, Brian Harring wrote: As a compromise, it could be made policy that bump to EAPI=foo bugs are valid. If someone would benefit from such a bump, he can file a bug and know that it won't be closed WONTFIX. On the other hand, the dev is under no more pressure than usual to do the bump. If you attach a patch and have done the legwork, sure. If you're just opening bugs w/ bump to EAPI=monkeys, bluntly, it's noise and it's annoying. EAPI bump requests for pkgs that need to move forward so an eclass can be cleaned up/moved forward, sure, but arbitrary please go bump xyz without a specific reason (and/or legwork done if not) isn't helpful. Kind of equivalent to zero-day bump requests in my view in terms of usefulness. Except this is what we have now, Yes, I stated it because I view it as useful/sane. and isn't a compromise at all. I think you're mistaken in assuming a compromise is the required outcome of this. Given the choice between something productive, and something not productive, you don't choose the quasi-productive solution. Bluntly, chasing EAPI versions w/out gain is a waste of time; others may think but it should be EAPI4- the latest!- and they'd be wrong. You bump when there is a reason to do so, or when from a maintenance standoint you've got time (now) to do so and can push it forward- getting ahead of future work. Keep in mind the rule every change carries a risk- while the risk is generally stupidly low, it's something I don't think you're being cognizant of in this notion of trying to get everything at EAPI whatever. Filing a bunch of please bump this to EAPI-whatever is just annoying nagging, it doesn't accomplish anything nor is the ticket particularly useful on it's own. A Please bump to EAPI4 due to issue xyz is useful- there is a core reason beyond hey, EAPI4 is the latest AND EVERYTHING MUST BE THE LATEST GREATEST!!! :) Same angle for EAPI5 and user patching... yes, devs will have a reason to move it forward, but user patching is going to be used by a *small* fraction of our userbase. Meaning if you want it, you're likely going to need to do the legwork bumping things forward, else you're on the devs time/prioritizations. Not saying it's perfect, but the comments above are realistic rather than trying to compromise against the realities of the situation. ;) ~harring
Re: [gentoo-dev] EAPI usage
On Sun, Sep 02, 2012 at 10:36:13AM -0400, Michael Orlitzky wrote: On 09/02/2012 09:46 AM, Rich Freeman wrote: On Sun, Sep 2, 2012 at 9:10 AM, Andreas K. Huettel dilfri...@gentoo.org wrote: What I dont actually understand at all is why bumping the EAPI should be so complicated or involved that it even deserves so much resistance... rantOk, it REALLY annoys me when people pull out this kind of a line in an argument... If it isn't all that complicated or involved and it just makes so much sense, then why do we bother to waste time asking for it to be made policy, since obviously everybody will just do it anyway... Believe it or not, people who take up an opposing side in a debate don't ALWAYS do it because they're simply dumber than you. That is, unless they're arguing with me... :) /rant I think everyone would be happier if all ebuilds in the tree were EAPI4. On the other hand, Rich is right that making this a policy will have the opposite of the intended effect: developers just won't fix bugs in EAPI4 ebuilds when they don't have time to do the EAPI bump (one could easily spend a few hours on this). As a compromise, it could be made policy that bump to EAPI=foo bugs are valid. If someone would benefit from such a bump, he can file a bug and know that it won't be closed WONTFIX. On the other hand, the dev is under no more pressure than usual to do the bump. If you attach a patch and have done the legwork, sure. If you're just opening bugs w/ bump to EAPI=monkeys, bluntly, it's noise and it's annoying. EAPI bump requests for pkgs that need to move forward so an eclass can be cleaned up/moved forward, sure, but arbitrary please go bump xyz without a specific reason (and/or legwork done if not) isn't helpful. Kind of equivalent to zero-day bump requests in my view in terms of usefulness. ~harring
Re: [gentoo-dev] Re: EAPI usage
On Fri, Aug 31, 2012 at 03:49:43PM +0100, Ciaran McCreesh wrote: On Thu, 30 Aug 2012 23:58:00 + (UTC) Duncan 1i5t5.dun...@cox.net wrote: Of course an individual PM could choose to keep support for as long as they want, but unless I'm missing something, that'd let PMs drop support for old EAPIs if desired, with at least a reasonably sane upgrade path for both PM devs and users. It's irrelevant: the amount of package mangler code to be saved by removing old EAPIs is so small it's not worth discussing. Most EAPI changes so far have either been additions or very simple behaviour tweaks, not removals of annoying things. Just seconding this statement; no PM author has stated maintaining EAPIs is an undue burden- it's come everytime from folks who don't /actually do any PM code/. So please stop telling us what is, and isn't a burden in our code. :) There are things we might change in future EAPIs that will in the very long term make this discussion worthwhile. If we get rid of VDB access or unconstrained env saving, *then* it might be worth having this discussion. Realistically even then, that's just swivelling vars/functions exposed to the ebuild env- it would require the vast majority of ebuilds migrating to EAPI versions that hide VDB access for this to be worth discussing (else due to backwards compat, it's a pointless discussion). Either way, there's no reason to require devs use the latest EAPI; they migrate at their own pace as they need to, which is fine. Case in point, check gentoo-x86 eapi usage: repository '/var/db/repos/gentoo': eapi: '4' 13523 pkgs found, 42.58% of the repository eapi: '0' 8171 pkgs found, 25.73% of the repository eapi: '2' 5246 pkgs found, 16.52% of the repository eapi: '3' 4297 pkgs found, 13.53% of the repository eapi: '1' 520 pkgs found, 1.64% of the repository 0 is still in heavy usage since a lot of ebuilds don't need the newer EAPI functionality; 1 is mostly dead since the only gain of it (in comparison to 0) was slot deps, 2 had used use deps thus those same folk migrated to 2 (since if you need slot deps, it's semi likely you need use deps). As for 3... that was prefix and xz support. No reason to use it in comparison to 4 frankly. Personally, I don't have any problems if gentoo were to mandate that EAPI1 shouldn't be used for new ebuilds in gentoo-x86, eapi2 instead. That sort of standard would make sense. ~harring
Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)
On Fri, Aug 31, 2012 at 11:12:44AM -0400, Alexis Ballier wrote: On Fri, 31 Aug 2012 15:45:21 +0100 Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller u...@gentoo.org wrote: Coming back to this old topic [1]. Is there still consensus that we should have such an EJOBS variable? (It shouldn't be called JOBS because this name is too generic, see the old discussion.) Then we could add it to EAPI 5. Ulrich [1] http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml If we're doing this, do we tell users to stop setting MAKEOPTS for EAPIs 5 and greater? How can this work ? I cant think of any simple solution. Do we change the name of MAKEOPTS for EAPIs 5 and greater instead? Do we put fancy code in the package mangler to deal with it? IMHO EAPI-5 compliant PMs should do MAKEOPTS=$MAKEOPTS -j$EJOBS for every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5 and greater. This is retroactive but could be classified 'PM internals' so its fine imho. This approach is fine imo, although I'd *potentially* look at adding a magic $PROC_COUNT var that is the # of cpu threads on the system; either that or defaulting jobs to it. I rather dislike requiring users to go jam a 2/4/8 in there when it's easy to compute. That said, it's minor. Either way, yes, I think EJOBS should be in EAPI5. ~harring
Re: [gentoo-dev] Re: news item: changes to stages (make.conf and make.profile)
On Tue, Jul 24, 2012 at 04:32:00PM -0400, Michael Mol wrote: On Tue, Jul 24, 2012 at 4:24 PM, Ian Stakenvicius a...@gentoo.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 24/07/12 02:52 PM, Rick Zero_Chaos Farina wrote: On 07/24/2012 09:33 AM, Fabian Groffen wrote: On 24-07-2012 09:24:03 -0400, Rich Freeman wrote: I guess this is a matter of opinion, but on Gentoo I don't think we're really at much risk of driving people away by OVER-communicating. Our users are used to things changing and a certain level of fix-it-yourself, but if we know something is going to cause no end of questions it only makes sense to throw the users a bone once in a while. The way in which news items aggressively request your attention, makes them something that should only be used if it's obvious it's important for the user (e.g. postfix thing for postfix users). This particular change seems more something for -announce, note in the handbook, and something like the suggestion of a file giving a nice hint. My impression is that the message is absolutely useless to the majority of users on their *already installed* system, so don't make everyone have to see the news item notice a couple of times and run `eselect news read` just for this. While I completely understand where Fabian is coming from on all this I respectfully disagree. Long term gentoo users do NOT read the handbook, ever. I still install new systems with odd hacks that I picked up when gentoo was versioned 1.x and it pleases me, I don't care if those steps are not in the docs anymore or discouraged or whatever. I've not even glanced at the handbook for years, yet I've installed gentoo on dozens of systems since the last time I did. Right, but would a news item now (regarding Catalyst) for something you do next month be particularily helpful, compared to a 'make.conf.moved' reminder file in /etc ? Or maybe a make.conf synlink to profiles/make.conf ? Or something else within the stage itself that makes it obvious that it's changed? I've often seen cases like these handled by keeping a referenced file where it's traditionally expected to be found, but leaving a comment in that file explaining that the content of that file had been moved to a new location, and the old location is deprecated. Would that work for a circumstance like this? Not really, no- it would mean the PM would have to parse/merge both locations, rather than just looking for the file in one of two spots. ~brian
Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)
On Tue, Jun 19, 2012 at 08:54:07PM +0200, Thomas Sachau wrote: Ciaran McCreesh schrieb: On Tue, 19 Jun 2012 20:16:39 +0200 Thomas Sachau to...@gentoo.org wrote: Since there is again no response at all, it seems like everyone is ok with this, so i will propose to add this to the next council agenda for EAPI-5 addition. Got a diff for PMS? Last time you only requested enough details for implementation, you did not require a PMS diff, so i wrote more details for the implementation. If you, Brian (for pkgcore) and zmedico accept this for EAPI-5, i might look into creating a diff against PMS but until then, i dont want to waste my time, especially since noone commented on the implementation details or the technical details and any change would require even more work to rework/adjust the PMS diff. You need a glep here frankly; per the norm, if you want things to move faster, then put in time- aka, generate a patch against PMS, write a patch for portage, etc, you get the idea. The bit re: a PMS patch is mostly that in looking at your proposal... well, I personally don't want to write that patch (nor do I suspect ulm/ciaran do either). One thing to note; this has been posted for all of 2-3 days; that's not exactly much time for 1) people to comment, 2) people to frankly comprehend the quite dense description you wrote. Please write a glep covering details of the implementation, background, preferablly why this route over others. Bluntly... clue everyone else in rather than hoping they'll just sign off on a fairly opaque list of things. :) It'll be useful for dev education also- which is a bit of a requirement for stuff of this sort considering it's not going to be a magic deploy/shit works everywhere situation I suspect. Would also be useful getting commentary from crossdev folk considering your solution is intended to be (best I can tell) full cross compilation support, and they've been leading that front for many, many years. Cheers- ~brian
Re: [gentoo-dev] [RFC] Dynamic SLOTs
On Mon, Jun 18, 2012 at 10:34:44AM +0200, Micha?? G??rny wrote:
On Sun, 17 Jun 2012 09:26:55 +0200
Micha?? G??rny mgo...@gentoo.org wrote:
I'm attaching a reStructuredText version of the spec. You can view it
rendered as a gist[1]. But please keep the replies on the list, rather
than forking the gist.
[1]:https://gist.github.com/2943774
Updated version. I've introduced dynamic SLOT groups and updated all
relevant information.
I didn't break DYNAMIC_SLOTS in ebuild to multiple variables yet but it
will be easy to change that if necessary.
This fixes problem previously described as 9a (dependencies). However,
I've added a new issue, 10c which marienz pointed out.
Bleh; wish your attachment had been text/plain for inline
commenting; pardon any mangling...
3. Defining dynamic SLOT groups
---
The list of supported dynamic SLOT groups should be declared
in profile `make.defaults` as ``DYNAMIC_SLOT_GROUPS`` variable. That
variable should contain whitespace-separated list of group names.
For each group, the same file should have a variable named
``DYNAMIC_SLOTS_`` followed by the group name. That variable should list
all possible dynamic SLOTs belonging to that group.
This is USE_EXPAND machinery; you should clarify why it's not being
reused here, nor treat explicitly as such.
Also, this gets fugly when one starts talking about individual pkgs
that have their own slotting, rather than global patterns like
python/multilib.
Should consider that and address it...
4. Defining supported SLOTs in an ebuild
An ebuild supporting building for multiple dynamic SLOTs has to declare
the supported slots using ``DYNAMIC_SLOTS`` variable. The variable can
be inherited from eclasses.
For example, an ebuild supporting building for multiple Python ABIs
would declare (either explicitly or implicitly through an eclass)::
DYNAMIC_SLOTS='py2.6 py2.7 py3.1 py3.2'
which would mean that when the package is built, one of the ``pyX.Y``
SLOTs must be used. As all of the listed SLOTs belong to the same group,
only of them may be used at a time.
An ebuild may also declare dynamic SLOTs from multiple groups::
DYNAMIC_SLOTS='py2.6 py2.7 lib32 lib64'
In this case, both one of ``pyX.Y`` and ``libX`` SLOTs need to be
declared.
The words I'll implement that when hell freezes over come to
mind. That's a mess requiring the PM to know all potential values
and do interpolation on the fly; you're expecting the PM to be far,
far more intelligent then it can be, and the results won't be
pleasant (not counting the joys of writing such a resolver mind you).
Additionally, your notion breaks down if py3.3 supports lib64, but not
lib32.
If the groupings are treated as USE_EXPAND (even if a segregated group
of it), you can abuse the same REQUIRED_USE machinery to specify the
allowed pigeon holes/slots, including arbitrary group combinations.
Still will be a bit harsh for the resolver I expect, but that's at
least descriptive enough to handle the py3.3/lib64 scenario I
mentioned, while being explicit data the PM can operate on (local to
that ebuild) without having to do nastyness.
Note also that if one just dropped your notion of reinventing the
wheel, and reused REQUIRED_USE logic for slots... well, strikes me
that gives the flexibility you desire while folding it into existing
slot machinery. Haven't prototyped it, so may be cracktastic, but
seems a bit more integrated than what you're proposing.
5. Building the ebuild against chosen SLOTs
---
In ``pkg_*`` and ``src_*`` phases, the build environment is provided
with currently enabled dynamic SLOTs via variables named
``DYNAMIC_SLOT_`` followed by dynamic SLOT group name. The ebuild must
use this variable to adjust the build process accordingly which can be
done either directly or via an eclass.
For example, in an ebuild using dynamic SLOTs for Python ABIs, the check
may look like::
case ${DYNAMIC_SLOT_PYTHON} in
py2.6)
# ...
;;
py2.7)
# ...
;;
# ...
esac
For ebuilds where one can't reuse the same source (moreso, can't do
the build of two different targets w/in that source, meaning you need
two work trees), this breaks down- which I expect is common. That
workflow needs to be non sucky for ebuild devs- so... sort that angle
please.
6. Relevance to binary and installed packages
-
It is necessary to store the dynamic SLOTs for which a package was built
in the binary package and installed package metadata. The exact
semantics are left to be implementation-specific. The implementation
must ensure, however, that a multiple dynamic SLOT variants of the same
package can be
Re: [gentoo-dev] [pre-GLEP] Optional runtime dependencies via runtime-switchable USE flags
On Sun, Jun 17, 2012 at 10:31:59PM +0200, Micha?? G??rny wrote: Hello, A simple solution to a program long-unsolved. In GLEP form. Both attached and published as a gist: https://gist.github.com/2945569 (please note that github doesn't render GLEP headers correctly) -- Best regards, Micha?? G??rny GLEP: XXX Title: Optional runtime dependencies via runtime-switchable USE flags Version: $Revision:$ Last-Modified: $Date:$ Author: Micha?? G??rny mgo...@gentoo.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 17 Jun 2012 Post-History: Abstract This GLEP addresses the issue of referencing optional runtime dependencies in Gentoo packages and ebuilds. It does introduce a concept of runtime-switchable USE flags to achieve that goal. Motivation == Optional runtime dependencies are often found in packages installing various scripts (shell, python, perl). These are not strictly required for the particular package to work but installing them enables additional functionality. Unlike in compiled programs, enabling or disabling those features (dependencies) does not affect the files installed by the package. They can be installed and uninstalled independently of the package, resulting in changes of functionality without a need to rebuild the package. Currently such dependencies are usually expressed only through ``pkg_postinst()`` messages. This forces user to manually install the necessary dependencies, and uninstall them when they are no longer necessary. Another solution is using regular USE flags. Those flags do not strictly follow the principles of USE flags because they do not affect files installed by the package and are not entirely effective to the package (a disabled feature will still be available if necessary dependency is installed). Additionally, it requires unnecessary rebuilds of the package in order to change the dependencies. Specification = The ebuilds aiming to provide features enabled through optional runtime dependencies should: 1. create regular USE flags for all those features, following appropriate specifications for Gentoo ebuilds, and including the flags in the ``IUSE`` variable; 2. introduce additional ``IUSE_RUNTIME`` variable listing names of USE flags related to optional runtime dependencies (without prefixes related to IUSE defaults). Additionally, the ebuilds must obey the following rules: 1. all flags listed in ``IUSE_RUNTIME`` have to be listed in ``IUSE``, 2. flags listed in ``IUSE_RUNTIME`` can be referred in ``RDEPEND``, ``PDEPEND`` and ``REQUIRED_USE`` variables, 3. flags listed in ``IUSE_RUNTIME`` must not be referred in phase functions, ``DEPEND`` or ``SRC_URI``, 4. flags listed in ``IUSE_RUNTIME`` may be referred through USE dependencies by other packages' ``DEPEND``, ``RDEPEND`` and ``PDEPEND`` variables. Unless I'm on crack, you're stating that essentially an optional use flag (one you label 'runtime'), is able to be used transitively during DEPEND. That's not particularly here's some suggested pkgs to install- that's rebuild the fucker for this changed DEPEND, which is the existing situation. The package manager should treat flags listed in ``IUSE_RUNTIME`` as regular USE flags, except for the following: 1. the state of the flags must be re-evaluated each time the package dependency graph is considered, 2. enabling or disabling any of the flags must not involve rebuilding the package, 3. the flags may be listed in the visual output in a distinct way to inform the user that they affect runtime dependencies only. Rationale = The proposed solution tries to solve the issue of handling runtime dependencies while reusing the existing infrastructure. Most importantly, users will be able to reuse the existing tools and configuration files to enable and disable optional runtime and build-time dependencies alike. The remaining reused features include: - dependency syntax, If you invent new syntax, I'm going to be unhappy. KISS as it were. - ability to use ``REQUIRED_USE``, USE dependencies, - ability to describe flags in `metadata.xml`, - global flag names (and descriptions). Alternative proposed solution involved creating additional ``SDEPEND`` variable. That proposition had the following disadvantages: - being package-oriented rather than feature-oriented, No; use flags are our configuration space, and they turn on/off sections of the given pkgs graph. Your proposal relies on the same concept; bluntly, what you're proposing is just as 'package oriented'. Effectively, you can't dismiss SDEPEND/ODEPEND via changing the rules between your proposal and ODEPEND's proposal. Nice try though. :) - lack of ability to express multiple packages required by a single feature, Eh? SDEPEND=my_feature? ( pkg1 pkg 2 ) - lack of ability to
Re: [gentoo-dev] About forcing rebuilds of other packages issue
On Sun, Jun 10, 2012 at 01:25:55PM +0100, Ciaran McCreesh wrote: On Sat, 09 Jun 2012 13:55:53 -0700 Zac Medico zmed...@gentoo.org wrote: A dependency atom will have optional SLOT and ABI_SLOT parts. Using the dbus-glib depedency on glib:2 as an example [1], the dbus-glib dependency will be expressed with an atom such as dev-libs/glib:2:= and the package manager will translate that atom to dev-libs/glib:2:=2.32 at build time. So, ':' is always used to distinguish SLOT deps, and ':=' is always used to distinguish ABI_SLOT deps. Is that syntax good? Here's a nicer syntax: no ABI_SLOT variable, and SLOT=2/2.32. Hate the slash; just looks ugly to me (so starts the bikeshed). Sans that naggle, notions fine however; not sure I'm a fan of people being able to specify the exact ABI they need from an ebuild while it's in source form, but may be of use for emul-* packages. ~harring
Re: [gentoo-dev] About forcing rebuilds of other packages issue
On Wed, Jun 06, 2012 at 05:43:49PM -0700, Zac Medico wrote: On 06/06/2012 12:23 PM, Ciaran McCreesh wrote: On Wed, 06 Jun 2012 21:16:05 +0200 Pacho Ramos pa...@gentoo.org wrote: Well, I think reading this thread is more or less clear what it would be supposed to do, also Zac suggested it and looks to have an idea about what should it do. There's a big leap from more or less clear and an idea to the kind of knowledge we want to have. Think REQUIRED_USE for how this can go wrong... If you think ABI_SLOT is essential, why not try implementing it and trying it out in a large number of packages, and reporting your results? It's pretty close to the SLOT operator model, and it seems like it should work fine. We can deploy EAPI 5_pre1 with ABI_SLOT support, and test it in an overlay before we include it in the final EAPI 5. I'd prefer you nailing down the details a bit more before slipping it into an EAPI called 5_pre1; aside from usual complaints, frankly I'd rather not have to figure out the design of it via raiding the patches out of portage history ;) If we're going to do this, there should be a way to represent the direction of compatibility. Might be overthinking it, but consider upgrades where new API is added; this does *not* break ABI, it extends it. Going in reverse however *would* break ABI for anything that was using the new additions. This issue can be avoided via usage of version operators w/ appropriate slot binding deps, just seems hanky in light of what we're talking about. I'm perfectly fine w/ ABI_SLOT and SLOT (I proposed a similar thing in '06/'07); I'd however suggest ensuring there is some buy in from devs on that one since that was the main argument against it in the past. That argument may no longer apply, but should be checked imo. ~harring
Re: [gentoo-dev] About forcing rebuilds of other packages issue
On Thu, Jun 07, 2012 at 08:15:28PM +0100, Ciaran McCreesh wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Thu, 07 Jun 2012 15:14:03 -0400 Ian Stakenvicius a...@gentoo.org wrote: How is the case of something like libpng going to be handled, where we only support one API (and so only one SLOT)? Either in the proposed ABI_SLOT thing or when using slot operators? Ideally, by you putting in the work and supporting more than one API, since doing so vastly improves the user experience. Failing that, SLOT plus blockers. Then if it turns out that really doesn't work (and it's not just developers being utterly lazy), either ABI_SLOT or parts in a future EAPI. SLOT + blockers only works for API breakages; for instances where API is the same but the ABI has shifted (a lib function switching from taking a short switching to a long for example), the scheme doesn't work and rebuilding is what's required. Thing is, the API breakage bit we already have sorted; point of this whole discussion is dealing w/ the latter scenario, which slot + blockers *doesn't* address; not unless your proposal is the clusterfuck notion of modifying the ebuild providing the lib, and sticking a shite ton of blockers for every known consumer. That approach is wrong on multiple levels to say the least. For the slot-operator case, will every consumer of libpng be forced to change their dep to libpng:= to ensure they get rebuilt when libpng bumps from 1.5 to 1.6?? Every consumer of libpng that wants to improve from the current situation, yes. Just going to point something out here; you've spent a lot of time stating Someone else has to go sort these problems out- problems that in many cases are decades old. You want to enforce this hard line, you do the work. Reminder: Ebuilds sole purpose are to make integrators jobs easier. Not to enforce the views of EAPI authors, but to enable people to get shit done. ABI_SLOT should *not* be used all over the place; it's basically a corner case variable that allows integrators to work around known cranky upstreams, or generally thorny ass problems. Aka, scenarios where the slotting solution doesn't fit. Unless ciaran's plan is to step up and fix all of these offending scenarios (and keep doing so), ABI_SLOT should be landed at the same time as SLOT operators. We already know it has uses, and when it *occurs*, it's painful to deal with it- specifically at the user level. Provide the PM the neccessary data, and it can lessen that pain. ~harring
Re: [gentoo-dev] [gentoo-portage-dev] About forcing rebuilds of other packages issue
On Tue, Jun 05, 2012 at 07:18:01PM -0700, Zac Medico wrote: On 06/05/2012 05:51 PM, Michael Weber wrote: Is there any chance to detect this ZLIB_VERSION problem with revdep-rebuild (worst case: add a list of possibly broken packages with tests)? I'd suggest a special ebuild phase to check for ABI changes, like the pre_pkg_preinst_abi_check phase suggested here: https://bugs.gentoo.org/show_bug.cgi?id=192319#c20 Same thing I said in '07; I don't have a problem w/ hooks for ebuilds to specify additional QA checks, but this *cannot* be the user's end solution- it needs to be purely for making it easier for devs to spot their screwups. In other words, revdep-rebuild shouldn't be involved; this should spot/complain that zlib (for example) changed abi w/out a matching metadata setting/whatever, rather than having checks done in the consumers. Using this for anything other than a QA check of the originating package, basically has an end result of us going towards a non-deterministic resolution model- which is a clusterfuck, frankly. ~harring
Re: [gentoo-dev] Git braindump: 2 of N: developer interaction (merge co-ordinators)
On Tue, Jun 05, 2012 at 09:04:33AM +0200, Micha?? G??rny wrote: On Mon, 4 Jun 2012 15:57:53 -0700 Brian Harring ferri...@gmail.com wrote: Btw, good catch on package.mask. Hhadn't thought of that, that *will* be the most contentious point. That can be dealt w/ via having git on portage-1 profile format so we'd have package.mask as directories (which Ciaran will validly hate, and I won't like due to having to write the portage-1 - PMS translater for rsync distribution), or coming up w/ a different way to split the commits across multiple files, rather than a single. That doesn't like a 'lesser evil' to me. I'd rather see those few conflicts and fix them once in a while rather than having to lookup multiple package.mask files with entries split in a semi-random manner. The rate of package.mask commits was calculated yesterday- it's around 4.5 a day. Non issue, move along... ~harring
Re: [gentoo-dev] Re: metadata/md5-cache
On Mon, Jun 04, 2012 at 09:27:10AM +0200, Micha?? G??rny wrote: On Sun, 3 Jun 2012 09:48:26 + Robin H. Johnson robb...@gentoo.org wrote: On Sun, Jun 03, 2012 at 11:34:07AM +0200, Micha?? G??rny wrote: I means using separate proto for metadata, not necesarrily git. In any case, if it comes to transferring a lot of frequently-changing files, rsync is not that efficient... It does NOT send any of the intermediate states. But it does have to check all the files. Which is a pretty minimal cost in the grand scheme of things. You also need to figure out what 'efficiency' you're going to talk about here; network io, disk io, cpu io, etc. Most people in this case care about network IO; rsync's not perfect, but for reasons described below, it's the best of breed for the usage scenario. Did I mention I'm not talking necessarily about git? Git would be sanest if you were after this; it already does point to point delta transformations sanely. No point in reinventing a VCS; if you can't force the tree back to a known good state (aka, distributed VCS), you can't apply deltas to it, which case you need an rsync like algo. Rather anything which would just lookup our timestamp, revision or whatever and just send what have changed, in a packed manner. This would be reinventing git/VCS, or more likely, pretending that a timestamp file automatically means the repository is *unmodified*, and trying to do a point to point transformation on it. Where you're notion breaks down is that fun little bit about unmodified. This is why rsync is used; it's not limited to a point to point transformation, it's able to work from any starting point *efficiently*. Either way, suggest you do some research into this- including efficiencies of rsync, git, existing snapshot delta rsync machinery (tarsync, diffball, etc), study the trade offs inherint in each. Your initial email frankly reaks of NIH, hence my suggestions to go investigate what exists now. ~harring
Re: [gentoo-dev] Git braindump: 2 of N: developer interaction (merge co-ordinators)
On Mon, Jun 04, 2012 at 03:49:31AM +, Kent Fredric wrote: On 3 June 2012 09:46, Robin H. Johnson robb...@gentoo.org wrote: If there are enough Alice developers, is it a possibility that Bob will never have a chance to get his commit in? All this requires, is that in the time it takes Bob to do 'git pull', Alice manages to do 'git push' again. Alice can thus deprive Bob of a fair chance to get his commit in. Bob becomes an unhappy developer and gives up. There's an easier solution here: Bob pushes to a branch or to a public repo ( ie: github ) , and then contacts Alice ( or somebody else ) who pulls their changes into the tree on their behalf. Its not ideal but better than nothing. And certainly better than being stuck on SVN where this case is virtually guaranteed and with no viable workarounds when it is encountered. Kent, you did read Robin's email fully before commenting, right? ;) You just proposed 'merge lieutenants', which Robin already covered in the originating email of this thread: http://archives.gentoo.org/gentoo-dev/msg_f478e9cbb14feb01ad0771c5d24222c4.xml For the record, I'm against any form of merge lieutenant reliant on someone pulling shit in; automated (QA of some form) I'd be fine w/, although that's not simple machinery to slap into the proposals. While I do grok the potential issue of someone being a hog (specifically via blasting commit by commit rather than building up work locally, then pushing it in chunks), frankly... I'm not that concerned about it, and would rather deal w/ it if/when it occurs. The nature of our commits for the most part are standalone from others- that's not true of the kernel/mozilla, thus why I don't think their issues are necessarily ours. ~harring
Re: [gentoo-dev] Git braindump: 1 of N: merging git signing
On Mon, Jun 04, 2012 at 08:45:42PM +0200, Dirkjan Ochtman wrote: On Mon, Jun 4, 2012 at 7:25 PM, Rich Freeman ri...@gentoo.org wrote: Anything we do has to be automated to be of any real value. ??Ideally if something goes wrong it should be as detectable as possible. Yeah, but you'd have to part of that at every developer's box. Can we just agree that having the tip of the main tree always signed will be enough for now, and postpone the rest of the discussion until later? ToT is always going to be signed. If it *isn't* signed, either the infra machinery is broken and not rejecting commits that it should reject, or someone is trojaning the repo (either via an infra compromise, local compromise, or via man in the middle). One thing people need to keep in mind here is that when you sign the commit, you're signing off on the history implicitly. Directly addressing freeman's comment about people sign the manifest but don't look at what they're signing, when it comes to git signage, bluntly, people doing that shouldn't have access- if they can't be arsed to validate what they're signing, then trusting them w/ the tree is probably questionable. Harsh, but frankly, sane people don't sign enforcable contracts w/out verifying what they're signing (note the 'enforcable' bit, stated to head off the EULA rathole discussion); this isn't any different frankly. ~harring
Re: [gentoo-dev] Git braindump: 1 of N: merging git signing
On Mon, Jun 04, 2012 at 03:27:03PM -0400, Rich Freeman wrote: On Mon, Jun 4, 2012 at 3:10 PM, Brian Harring ferri...@gmail.com wrote: One thing people need to keep in mind here is that when you sign the commit, you're signing off on the history implicitly. ?Directly addressing freeman's comment about people sign the manifest but don't look at what they're signing, when it comes to git signage, bluntly, people doing that shouldn't have access- if they can't be arsed to validate what they're signing, then trusting them w/ the tree is probably questionable. I suspect that you're missing my point. The argument was made that as long as merge commits are signed you know that unsigned commits referenced by them are OK. However, some of those commits might have been already in gentoo-x86 and I doubt that anybody is going to check those. Going over the details on the offchance you're misunderstanding part of it: A signed commit is a signing of the git metadata; tree hash (literally, the state of the tree), committer, author, message, and parent sha1. Each git commit includes it's parent sha1 in it; this gives a locked history for a given commit sha1 (unless someone preimages sha1). What matters is that the leaf node, the final point in the graph, is signed- that's a dev sign off on effectively that they created that particular locked history. Realistically signing of each node is preferable, but the leaf is the minimal required. The dev, prior to signing that, should be verifying what they're adding (moreso, what exists between last signed rev and theirs), they agree to and know of. Specifically, they're asserting their addition. When the dev resync's, part of the process *should* be validating that the new remote ToT is actually signed by a trusted key; aka, another dev. For merge commits, the same applies here- due to the rather nice sha1 hack of git, signing the merge commit has the same angle, the dev signs off on the new addition effectively (for merge commits, moreso that the resultant integration is what they intended). Now to pick at your statement; the dev is signing off on commits already in x86; think through this scheme; each dev signs off as they go on their specific changes, validating backwards is doable- meaning you can validate that all commits in the vcs were signed off by trusted folk (ie, dev's). Your argument is either based on misunderstanding how this works (potentially just ignoring the pull validation devs have to do), or assuming that the dev is responsible for signing off on *everything* in the tree's proceeding history just because they're signing a specific history/tree sha. As indicated above, they're signing off essentially on each gap between signing, the delta. If I have a perfect commit, I do a git pull and a git push and the result is a merge that references whatever was in gentoo-x86 before, whether placed there by dev, or hacker, or whatever. Unless I go back and review the existing gentoo-x86 history (and likely have to repeat the process when somebody else does a push before I do), I can't vouch for what was in there already - just what I'm adding. For someone to inject something into the history requires basically a sha1 preimage attack, or the dev to be hit by an injection attack, along w/ the dev *skipping* validation of the pull to current ToT. If we require every commit to be signed, that blocks the potential for a careless dev skipping the valdation to sign off on things that were injected. This is a bit honerous, and is why signed pushes were the preference (and is a bit of an abuse what we're doing now). The reason I mentioned maifests is that they have the same issue. If I keyword an arch on foo-1.4.5, I sign the manifest. That doesn't mean that I checked every file in the package's directory tree for issues. At most I checked foo-1.4.5, but I can't sign off on just 1.4.5 - I have to sign off on everything. Also, when I sign off on 1.4.5, I'm really just signing off for the keyword change, not the piece of buggy code I didn't write on line 37 of the ebuild. Manifests aren't particularly comparable, nor is it worth trying to do so. Technically w/ CVS manifests you can isolate down just the delta, and infer that the signage was for that, but the tools don't particularly exist for it, nor is it worth it; cvs just isn't designed sanely to allow for this. With git however, since the signing can be integrated directly into the revs themselves, you *are* directly signing the deltas; as said, w/ cvs technically you are (even if you're signing all of the results). An additional note btw; to be clear, the only manifest file that exists in a git tree is a thin manifest holding checksums for the distilfes involved. If you're thinking of rsync distributed trees (which aren't thin manifests), the plan was to enhance the existing rsync snapshot generation scripts so
Re: [gentoo-dev] Git braindump: 2 of N: developer interaction (merge co-ordinators)
On Tue, Jun 05, 2012 at 12:36:04AM +0200, Michael Weber wrote: On 06/04/2012 03:25 PM, Brian Harring wrote: While I do grok the potential issue of someone being a hog (specifically via blasting commit by commit rather than building up work locally, then pushing it in chunks), frankly... I'm not that concerned about it, and would rather deal w/ it if/when it occurs. The nature of our commits for the most part are standalone from others- that's not true of the kernel/mozilla, thus why I don't think their issues are necessarily ours. True. We already have maintainers and herds as responsible (sole editors) entities for locations (packages). But, we have arch teams editing ebuild/KEYWORDS, which alters Manifest/EBUILD lines. Resulting in potential clashes (not fast-forwardable), if the herd or maintainer does bumps or cleanups. Will these Manifest lines (and the arch team inflicted Manifest changes)? Converting to git, we'll switch over to thin manifests- they're *just* the checksums for the distfiles, no need for the rest since git already provides that verification implicitly. That just leaves conflict w/in ebuilds, which is a valid the dev needs to deal with this themselves scenario imo. According to robbat2 data (gentoo-commit tarball) we have ~400k commits in gentoo-x86 (w/o proj,xml) in 4.7 years, that's 6.2 per hour averaged. But I've to look into the data to see trends (# developers, daylight). One thing to note- that's *individual* commits, and probably a mildly jacked up number due to the double tap requirement of commiting manifests to CVS. What I'm driving at is that there's a difference between commits/revisions, and pushs; I expect our push rate to be less; I'd be surprised if we're doing 1:1 commit/push rate. The conflict rate should be less painful for people in that light, or at least has been in my experience thus far. Btw, good catch on package.mask. Hhadn't thought of that, that *will* be the most contentious point. That can be dealt w/ via having git on portage-1 profile format so we'd have package.mask as directories (which Ciaran will validly hate, and I won't like due to having to write the portage-1 - PMS translater for rsync distribution), or coming up w/ a different way to split the commits across multiple files, rather than a single. That's assuming package.mask becomes a significant conflict point also. Frankly I'd rather deal w/ that problem when it arrises, rather than trying to optimize for it now. ~harring
Re: [gentoo-dev] Re: metadata/md5-cache
On Sun, Jun 03, 2012 at 09:25:43AM +, Robin H. Johnson wrote: On Sun, Jun 03, 2012 at 08:31:43AM +, Duncan wrote: Micha?? G??rny posted on Sun, 03 Jun 2012 09:22:04 +0200 as excerpted: Even if only the files metatdata changes, that still adds a significant cost to an rsync. I wonder when it will come to the point where git will be more efficient than rsync. Or maybe it would be already? Handwavey guess, but I've figured git to be more efficient client-side for some time. Server-side I don't know about, but I've presumed that's the reason the switch-to-git plans haven't included switching the default for user-syncs to git. I expect user/client side, git would be more efficient already, but as I said, that's handwavey guesses. No, the switch to git will NOT help users, it isn't more efficient. They will still be best served by rsync, for a couple of reasons: 1. metadata cache is NOT available in Git. Sidenote, and this is mildly insane, I'd thought about submodules for this; basically every rsync window, we dump the metadata into vcs, which devs can pull down and make use of. I've also not experimented w/ this workflow, so it could be batshit insane. Anyone game to experiment? ~harring
Re: [gentoo-dev] multiprocessing.eclass: doing parallel work in bash
On Sat, Jun 02, 2012 at 03:50:06PM -0700, Zac Medico wrote: On 06/02/2012 02:31 PM, Micha?? G??rny wrote: On Sat, 2 Jun 2012 15:54:03 -0400 Mike Frysinger vap...@gentoo.org wrote: # @FUNCTION: redirect_alloc_fd # @USAGE: var file [redirection] # @DESCRIPTION: (...and a lot of code) I may be wrong but wouldn't it be simpler to just stick with a named pipe here? Well, at first glance you wouldn't be able to read exactly one result at a time but is it actually useful? I'm pretty sure that the pipe has remain constantly open in read mode (which can only be done by assigning it a file descriptor). Otherwise, there's a race condition that can occur, where a write is lost because it's written just before the reader closes the pipe. There isn't a race; write side, it'll block once it exceeds pipe buf size; read side, bash's read functionality is explicitly byte by byte reads to avoid consuming data it doesn't need. That said, Mgorny's suggestion ignores that the the code already is pointed at a fifo. Presume he's suggesting Just open it everytime you need to fuck with it... which, sure, 'cept that complicates the read side (either having to find a free fd, open to it, then close it), or abuse cat or $() to pull the results and make the reclaim code handle multiple results in a single shot. Frankly, don't see the point in doing that. The code isn't that complex frankly, and we *need* the overhead of this to be minimal- the hand off/reclaim is effectively the bottleneck for scaling. If the jobs you've backgrounded are a second a piece, it matters less; if they're quick little bursts of activity, the scaling *will* be limited by how fast we can blast off/reclaim jobs. Keep in mind that the main process has to go find more work to queue up between the reclaims, thus this matters more than you'd think. Either way, that limit varies dependent on time required for each job vs # of cores; that said, you run code like this on a 48 core and you see it start becoming an actual bottleneck (which is why I came up with this hacky bash semaphore). ~harring
Re: [gentoo-dev] multiprocessing.eclass: doing parallel work in bash
On Fri, Jun 01, 2012 at 06:41:22PM -0400, Mike Frysinger wrote:
# @FUNCTION: multijob_post_fork
# @DESCRIPTION:
# You must call this in the parent process after forking a child process.
# If the parallel limit has been hit, it will wait for one to finish and
# return the child's exit status.
multijob_post_fork() {
[[ $# -eq 0 ]] || die ${FUNCNAME} takes no arguments
: $(( ++mj_num_jobs ))
if [[ ${mj_num_jobs} -ge ${mj_max_jobs} ]] ; then
multijob_finish_one
fi
return $?
}
Minor note; the design of this (fork then check), means when a job
finishes, we'll not be ready with more work. This implicitly means
that given a fast job identification step (main thread), and a slower
job execution (what's backgrounded), we'll not breach #core of
parallelism, nor will we achieve that level either (meaning
potentially some idle cycles left on the floor).
Realistically, the main thread (what invokes post_fork) is *likely*,
(if the consumer isn't fricking retarded) to be doing minor work-
mostly just poking about figuring out what the next task/arguments
are to submit to the pool. That work isn't likely to be a full core
worth of work, else as I said, the consumer is being a retard.
The original form of this was designed around the assumption that the
main thread was light, and the backgrounded jobs weren't, thus it
basically did the equivalent of make -jcores+1, allowing #cores
background jobs running, while allowing the main thread to continue on
and get the next job ready, once it had that ready, it would block
waiting for a slot to open, then immediately submit the job once it
had done a reclaim.
On the surface of it, it's a minor difference, but having the next
job immediately ready to fire makes it easier to saturate cores.
Unfortunately, that also changes your API a bit; your call.
~harring
Re: [gentoo-dev] multiprocessing.eclass: doing parallel work in bash
On Fri, Jun 01, 2012 at 06:41:22PM -0400, Mike Frysinger wrote:
regenerating autotools in packages that have a lot of AC_CONFIG_SUBDIRS is
really slow due to the serialization of all the dirs (which really isn't
required). so i took some code that i merged into portage semi-recently
(which is based on work by Brian, although i'm not sure he wants to admit it)
I've come up with worse things in the name of speed (see the
daemonized ebuild processor...) ;)
and put it into a new multiprocessing.eclass. this way people can generically
utilize this in their own eclasses/ebuilds.
it doesn't currently support nesting. not sure if i should fix that.
i'll follow up with an example of parallelizing of eautoreconf. for
mail-filter/maildrop on my 4 core system, it cuts the time needed to run from
~2.5 min to ~1 min.
My main concern here is cleanup during uncontrolled shutdown; if the
backgrounded job has hung itself for some reason, the job *will* just
sit; I'm not aware of any of the PMs doing process tree killing, or
cgroups containment; in my copious free time I'm planning on adding a
'cjobs' tool for others, and adding cgroups awareness into pkgcore;
that said, none of 'em do this *now*, thus my concern.
-mike
# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
# @ECLASS: multiprocessing.eclass
# @MAINTAINER:
# base-sys...@gentoo.org
# @AUTHORS:
# Brian Harring ferri...@gentoo.org
# Mike Frysinger vap...@gentoo.org
# @BLURB: parallelization with bash (wtf?)
# @DESCRIPTION:
# The multiprocessing eclass contains a suite of functions that allow ebuilds
# to quickly run things in parallel using shell code.
if [[ ${___ECLASS_ONCE_MULTIPROCESSING} != recur -_+^+_- spank ]] ; then
___ECLASS_ONCE_MULTIPROCESSING=recur -_+^+_- spank
# @FUNCTION: makeopts_jobs
# @USAGE: [${MAKEOPTS}]
# @DESCRIPTION:
# Searches the arguments (defaults to ${MAKEOPTS}) and extracts the jobs
number
# specified therein. Useful for running non-make tools in parallel too.
# i.e. if the user has MAKEOPTS=-j9, this will show 9.
# We can't return the number as bash normalizes it to [0, 255]. If the flags
# haven't specified a -j flag, then 1 is shown as that is the default `make`
# uses. Since there's no way to represent infinity, we return 999 if the user
# has -j without a number.
makeopts_jobs() {
[[ $# -eq 0 ]] set -- ${MAKEOPTS}
# This assumes the first .* will be more greedy than the second .*
# since POSIX doesn't specify a non-greedy match (i.e. .*?).
local jobs=$(echo $* | sed -r -n \
-e
's:.*[[:space:]](-j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p' \
-e 's:.*[[:space:]](-j|--jobs)[[:space:]].*:999:p')
echo ${jobs:-1}
}
This function belongs in eutils, or somewhere similar- pretty sure
we've got variants of this in multiple spots. I'd prefer a single
point to change if/when we add a way to pass parallelism down into the
env via EAPI.
# @FUNCTION: multijob_init
# @USAGE: [${MAKEOPTS}]
# @DESCRIPTION:
# Setup the environment for executing things in parallel.
# You must call this before any other multijob function.
multijob_init() {
# Setup a pipe for children to write their pids to when they finish.
mj_control_pipe=${T}/multijob.pipe
mkfifo ${mj_control_pipe}
exec {mj_control_fd}${mj_control_pipe}
rm -f ${mj_control_pipe}
Nice; hadn't thought to wipe the pipe on the way out.
# See how many children we can fork based on the user's settings.
mj_max_jobs=$(makeopts_jobs $@)
mj_num_jobs=0
}
# @FUNCTION: multijob_child_init
# @DESCRIPTION:
# You must call this first in the forked child process.
multijob_child_init() {
[[ $# -eq 0 ]] || die ${FUNCNAME} takes no arguments
trap 'echo ${BASHPID} $? '${mj_control_fd} EXIT
trap 'exit 1' INT TERM
}
Kind of dislike this form since it means consuming code has to be
aware of, and do the () trick.
A helper function, something like
multijob_child_job() {
(
multijob_child_init
$@
)
multijob_post_fork || die game over man, game over
}
Doing so, would conver your eautoreconf from:
for x in $(autotools_check_macro_val AC_CONFIG_SUBDIRS) ; do
if [[ -d ${x} ]] ; then
pushd ${x} /dev/null
(
multijob_child_init
AT_NOELIBTOOLIZE=yes eautoreconf
)
multijob_post_fork || die
popd /dev/null
fi
done
To:
for x in $(autotools_check_macro_val AC_CONFIG_SUBDIRS) ; do
if [[ -d ${x} ]]; then
pushd ${x} /dev/null
AT_NOELIBTOOLIZE=yes multijob_child_job eautoreconf
popd
fi
done
Note, if we used an eval in multijob_child_job, the pushd/popd could
be folded in. Debatable.
# @FUNCTION: multijob_post_fork
# @DESCRIPTION:
# You must call this in the parent process after forking a child process.
# If the parallel limit has been hit, it will wait
Re: [gentoo-dev] Stability of /sys api
On Mon, May 14, 2012 at 03:53:53AM -0400, Walter Dnes wrote: After some Google-searching, I think I've figured out how to implement automounting under mdev. I'd like to put in as much sanity-checking into the script as possible. Right now I have 1 USB stick plugged in as /dev/sdb. Th hard drive is /dev/sda. The removable data is readable like so... cat /sys/block/sda/removable 0 cat /sys/block/sdb/removable 1 My question... is this API stable or deprecated? I.e. can I count on it being around for a while? I figure this question is a developer type question rather than ordinary user type. Api is stable although last I dealt with that crap it was reliant on chipsets/controllers not sucking and misreporting (mmc in particular comes to mind, although perhaps the hardware sucks less these days). Suggest you start studying udev source in addition since your questions of that sort are likely to be answered there. Aka, most likely wind up asking udev upstream (likely gregkh assuming he hasn't killfile'd everyone from that thread). Now the unfun part; this isn't really the right place to be asking. I get you're doing this w/ a gentoo intent, but you're dancing that line mightily fine. People asking can I safely use nested context managers in python2.6 even if it's orientated towards a potential gentoo bit (say the ephemeral gentoo stats project), typically will be told dunno. Ask the people who wrote the damn thing. Point there is that the ml shouldn't be used as tech help for the guts of I don't want udev and am trying to replace it with mdev; pkgcore nor paludis internal questions don't come here (format does since this is the appropriate venue) under the same logic. Forums come to mind, or appropriate upstreams as mentioned. Barring that, use the source luke, and start reading the lkml. If you're trying to do this, you'll likely need to track discussions there. Not trying to be a dick mind you, and perhaps others view othewise, but this isn't the place for it nor do I suspect people care to see more of this particular war play out on our ml. ~harring
Re: [gentoo-dev] RFC: new feature to disable merging into stray locations
On Mon, Apr 30, 2012 at 03:36:17PM +0200, Ulrich Mueller wrote:
On Mon, 30 Apr 2012, Thomas Sachau wrote:
Krzysztof Pawlik schrieb:
On 30/04/12 10:39, Ulrich Mueller wrote:
If the eclass doesn't work with FEATURES=collision-protect
then it needs to be fixed.
Long story short: older eclass compiled Python byte code in live
file system, new one does it in src_install so .pyo/.pyc gets
properly recorded, so the package *has to overwrite* files that
are not owned by anyone (no package owns them).
The usual approach to this problem was to have some cleanup code
in pkg_setup or pkg_preinst that would remove the old files.
Interesting idea - I see two problems:
- how I would get the list of files for already installed version
(the file list can (and will) differ between versions)
Simple workaround in pkg_preinst (just example code, nothing to
copy/paste):
if has_version first_good_version ; then
for i in $(find ${D} -name *.{pyc,pyo}) ; do
[[ -e ${i/${D}/${ROOT}/} ]] rm ${i/${D}/${ROOT}/}
done
fi
This wouldn't work, because collision protection runs _before_
pkg_preinst.
However, one could imagine a (somewhat dirty) workaround:
- At the end of src_install, rename all *.pyc and *.pyo in ${D} to
names that don't collide with the ones in ${ROOT}, so that collision
protection won't be triggered.
- At the beginning of pkg_preinst, rename them back to their original
names.
- Run above code.
Alternatively, you could take all .py files of the installed
versions and blindly remove the pyo/pyc files for them in
pkg_preinst (this should also prevent leaving dead files around).
s/preinst/setup/ otherwise the same argument as above would apply.
The checks above can misbehave.
Pretty sure I've already suggested this route for this specific issue,
but QA vars in the ebuild that portage can use for hints is a sane,
non-cluster-fucky way to deal with this. Something along the lines of
QA_COLLISION_PROTECT_IGNORE='*.pyc'
Still has the potential for things going boom, but warnings can be
setup for when collision-protect is on, and the portage version
doesn't look new enough (those being details; point is the general
approach).
~harring
Re: [gentoo-dev] New eclass for Python
On Wed, Apr 04, 2012 at 10:41:03AM -0400, Mike Gilbert wrote: On Wed, Apr 4, 2012 at 4:50 AM, Corentin Chary corentin.ch...@gmail.com wrote: I have a feature request for distutil-ng (or maybe it's already possible but I don't know how). I have a package that depends on python-dateutil:python-2 for python2_x and python-dateutil:python-3 for python3_x. Would it be possible to have virtual targets like python, python2, python3, pypi, jithon ? One way to implement this would be to have a separate function that you would call to generate abi-specific dependencies. Think through the performance implications of that, and the fact that moves control/decisions outside of the PM resolvers purview. If it's use controlled and is part of metadata, sure, but anything else? Not so much. :) ~harring
Re: [gentoo-dev] New eclass for Python
On Wed, Apr 04, 2012 at 08:36:37PM -0400, Mike Gilbert wrote: On Wed, Apr 4, 2012 at 8:07 PM, Brian Harring ferri...@gmail.com wrote: On Wed, Apr 04, 2012 at 10:41:03AM -0400, Mike Gilbert wrote: On Wed, Apr 4, 2012 at 4:50 AM, Corentin Chary corentin.ch...@gmail.com wrote: I have a feature request for distutil-ng (or maybe it's already possible but I don't know how). I have a package that depends on python-dateutil:python-2 for python2_x and python-dateutil:python-3 for python3_x. Would it be possible to have virtual targets like python, python2, python3, pypi, jithon ? One way to implement this would be to have a separate function that you would call to generate abi-specific dependencies. Think through the performance implications of that, and the fact that moves control/decisions outside of the PM resolvers purview. If it's use controlled and is part of metadata, sure, but anything else? ??Not so much. :) All I'm talking about is a function or function(s) to generate deps like these: python2_6? ( dev-python/python-dateutil[python2_6] ) python2_7? ( dev-python/python-dateutil[python2_7] ) I'm not sure how that would affect performance. Nah, that's fine- that's metadata level representation of it. Sorry, aparenytly I misread your original email- interpretted it as shelling out for that decision (something people propose occasionally, and usually needs to be stomped out w/ maximal force), rather than representing it properly in dependencies. What you're intending there is perfectly sane/acceptable; pardon the noise ;) ~brian
Re: [gentoo-dev] About suggesting to create a separate partition for portage tree in handbook
On Sat, Mar 31, 2012 at 08:44:02AM +, Sven Vermeulen wrote: On Fri, Mar 30, 2012 at 10:06:18AM +0200, Pacho Ramos wrote: Looks then that there are several alternatives for portage tree, then, maybe the option would be to add a note to Gentoo Handbook explaining the cons of having portage tree on a standard partition and, then, put a link to a wiki page (for example) where all this alternatives are explained. What do you think about this approach? I don't like the cons approach, as it gives the impression that users are pushed into a negative solution, whereas the current situation works just fine for almost all users. The approach for a different partition is for performance reasons (which most users don't have any negative feelings about) and as such might be read as a ricer approach. For modern hardware w/ a modern kernel (or at least =2.6.38 for the dcache resolution optimizations)... does anyone actually have real performance stats for this? If the notion is a seperate FS, one tailored to the portage tree's usage models (tail packing for example), sure, grok that although I question how much people really are getting out of it. In the past, situation definitely differed- I'm just wondering if the gain is actually worth debating it, rather than just ignoring it (or sticking it in a foot note for people trying to use durons). ~harring
Re: [gentoo-dev] Happy 10th birthday (in advance)
On Sat, Mar 31, 2012 at 10:52:53AM +0100, Ciaran McCreesh wrote: On Sat, 31 Mar 2012 12:44:03 +0300 Alex Alexander alex.alexan...@gmail.com wrote: @preserved-libs works very well and is awesome. hack or not. IMO it should be in stable already. I've been using it on stable production boxes for years without any issues :) ...and here we see the problem. You think that I haven't noticed it break means it works. The problem with preserved-libs (and emerge --jobs, for that matter) is that the design is I can think of a few ways where it might break, so I'll hard-code in special cases to handle those, but in general I can't think of what other problems there are so it's fine. That's a bad way of doing things. Then don't use it. Reality is, gentoo does. If you don't like that fact, I suggest you stick to exherbo. Related, why the hell are you still even around here? You literally send more mail to our dev ml then to exherbos. I wouldn't care if it weren't the fact your gentoo dev posts generally consist of xyz is stupid, as is the people behind it whether it be portage, udev, council, etc, take your pick. ~harring (being rather tired of the broken record).
Re: [gentoo-dev] [GSoC2012] Cross Container Support Project
On Fri, Mar 23, 2012 at 01:46:17PM +0300, Alexey Shvetsov wrote: Hi! Well i have 2 arm lxc containers on amd64 machine. Its works good if qemu support most of needed cross arch instructions I'd be curious how much of that is native, vs emulated. The hybrid approach of scratchbox/obs has some definite gains. If we had a clean way to mark which can be native (toolchain), the perf gain is definitely worth the work... ~brian
