Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Fri, 02 Jun 2017 15:55:17 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On pią, 2017-06-02 at 13:27 +0200, Alexis Ballier wrote:
> > On Thu, 01 Jun 2017 23:31:25 +0200
> > Michał Górny <mgo...@gentoo.org> wrote:
> > [...]  
> > > > There are probably dozens of ways to make that non
> > > > deterministic. Here's one: USE='-*'. Apply '|| ( cli cgi fpm
> > > > apache2 embed phpdbg )' last; this enables 'cli'. Since it's
> > > > the last one, REQUIRED_USE is not satisfied because of 'cli?
> > > > ( ^^ ( readline libedit ) )'. If you process it first, then you
> > > > enable cli and readline and are good.
> > > 
> > > You just take a second iteration, and things fall into place.
> > > That's not a problem.
> > >   
> > > > > > Also, what happens if we applied all the constraints and
> > > > > > obtained some useflags setting that still fails REQUIRED_USE
> > > > > > check ?  
> > > > > 
> > > > > It can't happen. If you can apply all the constraints, then
> > > > > implicitly REQUIRED_USE is satisfied. If you can't apply all
> > > > > the constraints, then it just fails. Of course, we want to
> > > > > ultimately avoid that case.
> > > > 
> > > > See the php example above. How do you ensure this does not
> > > > happen?
> > > > 
> > > > Note that your assertion 'If you can apply all the constraints,
> > > > then implicitly REQUIRED_USE is satisfied.' is false: You're
> > > > changing the values of useflags, thus might violate some
> > > > previously checked constraint.
> > > 
> > > So you reiterate. Either you reach a solution (preferably there's
> > > only a single valid solution you can reach) or you hit a previous
> > > state which indicates a loop and you fail.  
> > 
> > 
> > So, PM, for every ebuild, will need to store the (at most) 2^n
> > states it has already seen while trying to solve REQUIRED_USE and
> > iterate until it cannot proceed anymore or falls into a previous
> > state (so that's 2^n time too). That way we go from a linear time &
> > linear space algorithm to one in exponential time & space. That's
> > probably not a good idea.  
> 
> I don't think that's actually going to happen. You'd have to try
> really hard to get over n-1 iterations. I mean, the only simple way I
> can think of is:
> 
>   b? ( a ) c? ( b ) d? ( c ) e? ( d ) ...
> 
> and this only means n-1 iterations. Can you think of a better way to
> force multiple iterations that can be written without making
> REQUIRED_USE completely unreadable? How likely is that going to happen
> accidentally?

I don't see any reason why it should terminate after n iterations; I
don't see any example of how to do more either. I can try to think more
about it, but maybe it is not even needed, see below.

> > I think it's better to limit the number of iterations to some
> > constant. I'd say 1, then fail if REQUIRED_USE is still not
> > satisfied. Maybe 2 or 3 can be useful but I think it'd be much
> > harder to provide automated checks for that and much harder for
> > ebuild writers to understand what will happen with the REQUIRED_USE
> > they're about to write. 
> 
> Well, I don't mind setting that. All my tests (that weren't
> deliberately abusing the algorithm) were able to finish in a single
> iteration. 2 or 3 should probably be safe for all the reasonable
> uses. However, if we're not going to verify all possible values on
> repoman side, I think it would be better to have a larger limit for
> users, to not have things explode on them unnecessarily.


Look, if we assume left to right solving, only one iteration possible
and all the constraints to be of the form 'p_1? p_2? ... p_n? ( q_1 ...
q_m )' where p_i and q_i are either 'useflag' or '!useflag', then we
only have to check when such a clause can change a previously satisfied
clause to unsatisfied.

For two clauses: 
"p_1? p_2? ... p_n? ( q_1 ... q_m )" and "p'_1? p'_2? ... p'_{n'}?
( q'_1 ... q'_{m'} )", assuming the first is satisfied, when can solving
the 2nd break the 1st?

It must be that:
1.The conditions are compatible: No p_i is the negation of a p'_j.
2.Solving the 1st does not make the 2nd trivially true: No q_i is
  the negation of a p'_j.
3.Solving the 2nd does not make the 1st trivially true afterwards: No
  p_i is the negation of a q'_j.
4.Solving the 2nd does break the 1st assumption: (A q_i is
  the negation of a q_'j) or (a q'_j is some p_i and one of q_1 ... q_m
  might be false).

The 

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Sat, 03 Jun 2017 17:33:09 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On sob, 2017-06-03 at 13:00 +0200, Alexis Ballier wrote:
> > This whole thing definitely needs more thought and feedback but for
> > now those extra restrictions seem quite natural to me, allow easy
> > solving on the PM side and allow to have useful feedback from
> > repoman. 
> 
> Well, I'll try to figure out the magic you were telling me later but
> as a quick note, my specific use case for this are Python targets, so
> I'm going throw a few basic concepts that need to work for you to
> play with ;-).
> 
> In the following samples pt1,2,.. stands for PYTHON_TARGETS;
> pst1,2,... for PYTHON_SINGLE_TARGET. Eventually I'd like to kill the
> latter but that depends on how well the autosolving works.
> 
> 1. ^^ ( pst1 pst2 pst3.. ) pst1? ( pt1 ) pst2? ( pt2 ) pst3? ( pt3 )..

the pt{1,2,...} part does not matter here: they all fail point 4 when
compared as 2nd clause to the others (each ptX appears only once in the
whole expression), so this boils down to solving n-ary ^^, and to how
those are translated: you can have several translations but some would
not work properly. Let's try:
(a) pst1? ( !pst2 !pst3 )
(b) pst2? ( !pst3 )
(c) !pst3? ( !pst2? ( pst1 ) )

Between (a) and (c), points 1 to 3 hold (those points are reflexive).
For point 4, "a q'_j is some p_i" will hold in both ways, so we do
actually have a cycle between them. Bad luck.

Let's write (c) as:
(c) !pst3? ( !pst2? ( !pst1? ( pst1 ) ) )

Now, (c) can't break (a) because of point 1: pst1 in (a) vs !pst1 in
(c).


(b) can't break (a) because of point 2: q_i=!pst2 is the negation of
p_j=pst2.

(c) can't break (b) because pst2 vs !pst2 (point 1).


So, this formulation works:
(a) pst1? ( !pst2 !pst3 )
(b) pst2? ( !pst3 )
(c) !pst3? ( !pst2? ( !pst1? ( pst1 ) ) )

USE="pst1 whatever" will enable only pst1. USE="-pst1 pst2 whatever"
will enable only pst2. USE="-pst1 -pst2 pst3" will leave it alone.
USE="-pst{1,2,3}" will enable pst1.



Note that "pst1? ( pt1 ) pst2? ( pt2 ) pst3? ( pt3 ) ^^ ( pst1 pst2
pst3.. )" is a different story: (c) expanded as above can break 'pst1?
( pt1 )' (point 4: a q'_j is some p_i) and you can actually check that
with USE="-pt* -pst*"; you'll need 2 passes to get the proper solution.
Fortunately, this is still a DAG and repoman would be able to propose an
ordering requiring only one pass.


[...]
> 2. ^^ ( pst1 pst2.. ) pst1? ( pt1 ) pst2? ( pt2 ).. ^^ ( pt1 pt2 )
> 
> This is a possible extension of the above for the migration period.
> The idea is that exactly one PST must be selected, and only the
> matching PT must be selected (others are implicitly disabled).

If we expand '^^ ( pt1 pt2 )' as above we get:
(d) pt1? ( !pt2 )
(e) !pt2? ( !pt1? ( pt1 ) )

Here, (d) is annoying: it can break and be broken by 'pst2? ( pt2 )'.
There would be a cycle and this would be rejected/notified.

If you think about it, this would mean I have set USE="-* pt1 pst2";
pst2 forcing to enable pt2 but '^^ ( pt1 pt2 )' with pt1 enabled would
prefer pt1 and disable pt2 again... This hints the solution: You need
to define who wins between pt and pst.

Instead you could write it:
^^ ( pst1 pst2.. ) pst1? ( pt1 !pt2 ... ) pst2? ( !pt1 pt2 ... )..
But then 'pst2? ( !pt1 pt2 ... )' can break each other with 'pst1?
( pt1 !pt2 ... )' in the sense I defined because of point 4 (A q_i is
the negation of a q_'j); you'll get the repoman notification about a
cycle. This is a case of a perfectly valid constraint that is rejected
by the restriction.

It is valid because we know we are guaranteed exactly one pstX will
be enabled. We can hint the solver with that by writing:
^^ ( pst1 pst2.. )
pst1? ( pt1 !pt2 ... )
pst2? ( !pt1? ( pt2 !pt3 ... ) )
pst3? ( !pt1? ( !pt2? ( pt3 !pt4 ... ) ) )


Now we're good: For j>i, solving a pst{j} line does not break a pst{i}
one because of point 2: A q_i (pt{i}) is the negation of a p'_j
(!pt{i}).



It's getting a bit ugly but it's probably bearable with good reporting
from static checkers (repoman).



> 3. doc? ( || ( pt3 pt4 ) ) || ( pt1 pt2 pt3 pt4 )
> 
> This is distutils-r1 with USE=doc requiring python2. Note that it's
> an example where the second || is added via eclass [NB: we've checked
> and PMS says eclass values are appended to ebuild value].


Much simpler here:
(a) doc? ( !pt4? ( pt3 ) )
(b) !pt4? ( !pt3? ( !pt2? ( pt1 ) )

(b) can't break (a) because of point 4: Neither 'a q_i is
  the negation of a q_'j' nor 'a q'_j is some p_i' hold. We're good.


USE="-* doc" will enable pt3 only. USE="-* pt{whatever} doc" will
enable pt3 (if not enabled) unless pt{whatever} contains pt4.

Alexis.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

Here's a quick n dirty code to play with, based on yours: 
https://github.com/aballier/required-use

On Sat, 3 Jun 2017 18:58:35 +0200
Alexis Ballier <aball...@gentoo.org> wrote:

> > 1. ^^ ( pst1 pst2 pst3.. ) pst1? ( pt1 ) pst2? ( pt2 ) pst3? ( pt3
> > )..  

$ python3 ./nsolve.py '^^ ( pst1 pst2 pst3 ) pst1? ( pt1 ) pst2? ( pt2
) pst3? ( pt3 )'

[[!pst1, !pst2, !pst3]? => [pst1], [pst1]? => [!pst2],
[pst1]? => [!pst3], [!pst1, pst2]? => [!pst3], [pst1]? => [pt1],
[pst2]? => [pt2], [pst3]? => [pt3]]


> > 2. ^^ ( pst1 pst2.. ) pst1? ( pt1 ) pst2? ( pt2 ).. ^^ ( pt1 pt2 )



$ python3 ./nsolve.py '^^ ( pst1 pst2 ) pst1? ( pt1 ) pst2? ( pt2 ) ^^
( pt1 pt2 )'

[[!pst1, !pst2]? => [pst1], [pst1]? => [!pst2], [pst1]? => [pt1],
[pst2]? => [pt2], [!pt1, !pt2]? => [pt1], [pt1]? => [!pt2]]

[[pt1]? => [!pt2]] can break [[pst2]? => [pt2]]


> ^^ ( pst1 pst2.. ) pst1? ( pt1 !pt2 ... ) pst2? ( !pt1 pt2 ... )..

$ python3 ./nsolve.py '^^ ( pst1 pst2 ) pst1? ( pt1 !pt2 ) pst2? ( !pt1
pt2 )'

[[!pst1, !pst2]? => [pst1], [pst1]? => [!pst2], [pst1]? => [pt1],
[pst1]? => [!pt2], [pst2]? => [!pt1], [pst2]? => [pt2]]

[[pst2]? => [!pt1]] can break [[pst1]? => [pt1]]
[[pst2]? => [pt2]] can break [[pst1]? => [!pt2]]

> pst1? ( pt1 !pt2 ... )
> pst2? ( !pt1? ( pt2 !pt3 ... ) )
> pst3? ( !pt1? ( !pt2? ( pt3 !pt4 ... ) ) )

$ python3 ./nsolve.py '^^ ( pst1 pst2 ) pst1? ( pt1 !pt2 ) pst2?
( !pst1? ( !pt1 pt2 ) )'

[[!pst1, !pst2]? => [pst1], [pst1]? => [!pst2], [pst1]? => [pt1],
[pst1]? => [!pt2], [pst2, !pst1]? => [!pt1], [pst2, !pst1]? => [pt2]]


All good.


> > 3. doc? ( || ( pt3 pt4 ) ) || ( pt1 pt2 pt3 pt4 )

$ python3 ./nsolve.py 'doc? ( || ( pt3 pt4 ) ) || ( pt1 pt2 pt3 pt4 )'
[[doc, !pt3, !pt4]? => [pt3], [!pt1, !pt2, !pt3, !pt4]? => [pt1]]




Note: the code only reports errors if a clause can break another clause
left to it. It doesn't do topological sorting nor cycle reporting yet.

Alexis.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Mon, 05 Jun 2017 16:10:25 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On pon, 2017-06-05 at 09:55 +0200, Alexis Ballier wrote:
> > On Sun, 4 Jun 2017 10:59:38 +0200
> > Alexis Ballier <aball...@gentoo.org> wrote:
> >   
> > > Here's a quick n dirty code to play with, based on yours: 
> > > https://github.com/aballier/required-use  
> > 
> > I've run that on the whole tree (considering all ebuilds with non
> > empty REQUIRED_USE), some stats:
> > 
> > $ time python3 classify.py requsel 
> > Stats:
> > Parse error: 16  
> 
> Hmm, how did you get those numbers? I just tested parsing and found
> only 7 unique REQUIRED_USE entries that fail. However, my sample file
> is only around 1000 entries long, so either I failed to get all of
> them, or you didn't deduplicate your list ;-). More on it below.

I did not deduplicate anything. I took every single ebuild and
generated a list of req use out of it. Meaning if you had 10 ebuilds
for 1 package with the same req use that'd count as 10 above.

[...]
> > The cycle is mostly due to:
> > $ python3 nsolve.py 'llvm? ( gallium ) gallium? ( llvm )'
> > [...]
> > toposort.CircularDependencyError: Circular dependencies exist among
> > these items: {[gallium]? => [llvm]:{[llvm]? => [gallium]}, [llvm]?
> > => [gallium]:{[gallium]? => [llvm]}}
> > 
> > 
> > This is something I had overseen when replacing 'a q'_j is some p_i
> > and one of q_1 ... q_m might be false' by only 'a q'_j is some
> > p_i'; it can be replaced without changing anything in the way PM
> > would solve it by "a q'_j is some p_i and the set of {q_j} is not a
> > subset of q' union p'" (that is, {q_i} is not trivially true if the
> > 2nd clause is applied). Extending that, we get those stats:  
> 
> I'm not even trying to understand the things you say with indexes but
> I trust you know what you're doing.

With that kind of things it's good to have someone having a second
look. It's so easy to forget a case or to miss a simplification.

> For completeness, we need to
> consider three cross-dependent states:
> 
> a. a? ( b ) b? ( a )


that's exactly the above :p

{q_j} is {b}, {q'} is {a}, {p'} is {b}; {b} is a subset of {a} union
{b}


[...]
> b. !a? ( !b ) !b? ( !a )

that's also the above with s/!b/b/ s/!a/a/ :=)

> c. a? ( b ) !a? ( !b )

that's already handled.

[...]
> > I'll let you play with it, but for me it seems this would work quite
> > nicely.
> >   
> 
> Well, I guess it's time to hit the next level. For a start, we have to
> handle all-of groups, i.e.:
> 
>   ( a b c )
> 
> Stand-alone makes little sense (and little trouble) but as you could
> have seen it's used nested in other thingies:
> 
> 1. || ( ( a b ) ( c d ) e )
> 
> 2. ?? ( ( a b ) ( c d ) e )
> 
> 3. ^^ ( ( a b ) ( c d ) e )

Yeah that's the nesting problem causing a parse error.
Those should be expanded to implications. What I'm relying onto is all
clauses to be of the form '[list of conditions]? [list of constraints]'


> For verifying constraints, they are not bad. We just follow the
> generic rule that the branch evaluates to true if all subexpressions
> are true. 
> 
> For solving, it might be a little unclear on how to proceed with
> partially true branches but for the sake of simplicity I would just
> ignore them and behave as if they were false. That is, case (1) with
> USE='c' would result in 'a b' being enabled.
> 
> The practical uses I've seen are:
> 
> a. || ( deprecated ( gtk3 introspection ) )
> 
> I guess this one would be equivalent to:
> 
>   !gtk3? ( !introspection? ( deprecated ) )

Unfortunately no. Favoring leftmost, you need to enable 'deprecated'
when either gtk3 or introspection is false.

That'd be:
!gtk3? ( deprecated )
!introspection? ( deprecated )


Fortunately we can distribute \/ and /\:
|| ( deprecated ( gtk3 introspection ) )
is equivalent to:
|| ( deprecated gtk3 )
|| ( deprecated introspection )

giving the above implication translation.

This can be extended to 
|| ( ( a1 a2 a3 ... ) ( b1 b2 b3 ... ) ... ) to handle all cases.



> b. ^^ ( ( !32bit 64bit ) ( 32bit !64bit ) ( 32bit 64bit ) )
> 
> This looks like a crazy way of saying:
> 
>   || ( 32bit 64bit )

Hmm yes


> c. ^^ ( ( !ruby !s7 ) ( ruby !s7 ) ( !ruby s7 ) )
> 
> This looks like an insane version of:
> 
>   ?? ( ruby s7 )


yes too it seems



> except that per my solver it just disables both when both are set ;-).

That's kind of the point. And that's why it is important to have the
solving rules well defined. Depending on how REQUIRED_USE is written,
it will be solved differently even for equivalent logical formulas.


> Not sure i

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Tue, 06 Jun 2017 19:39:04 +0200
Michał Górny  wrote:

> > [...]  
> > > > > The question is whether we want to:
> > > > > 
> > > > > a. actually try to solve this nesting insanity,
> > > > > 
> > > > > b. declare it unsupported and throw REQUIRED_USE mismatch on
> > > > > user,
> > > > > 
> > > > > c. ban it altogether.
> > > > 
> > > > 
> > > > I don't think it is *that* insane to support nesting :)
> > > > 
> > > > > ( ^^ ( ?? ( a b ) c ( d e ) ) f )  
> > 
> > If you really need that then you'd need to expand it manually. It
> > seems better to have it expanded internally automatically.
> > Remember you were the one wanting to keep || & co because they're
> > simpler to read and write ;)
> >   
> 
> Well, I was able to implement the logic for all-of blocks outside
> and inside other n-ary constraints, including the necessary logic
> transformations. Fun fact is, I was able to do it without implementing
> a complete set of logic functions and transformations in AST ;-).
> 
> I've just made it fail (correctly this time) with any other kind of
> nesting -- I don't think it's going to have a real use case and even
> if it did, there are more readable ways of solving the same problem.
> 
> The question is -- will you rebase now on top of my changes

yes that should be the goal but there are a lot of things to do prior
to that I think (thanks for doing it btw)

> (and preferably use nice logical changes with good commit messages),

Heh, I really meant 'quickndirty' :) it's actually good there is an
actual semi relevant commit message :p

> or should I try later to merge the rest of your code in? ;-)
> 
> Also, do I presume correctly that for all supported cases (i.e. those
> which your nsolve does not reject), solve and nsolve are compatible?
> 

Not sure what you mean here. nsolve does not solve anything, it just
validates REQUIRED_USE so that it is guaranteed it can be solved.


We first need to define properly & formally how to solve requse
constraints if we want ppl to be able to rely on it (or rather write
requse that give the expected result).

The way I see it, REQUIRED_USE ast looks like:
(assuming ^^ is already expanded to || + ??)

clause =
AllOf(list of clauses)
  | AnyOf(list of clauses)
  | AtMostOne(list of clauses)
  | Implication(useflag, clause)
  | useflag

Now, portage already has the function 'eval(input, clause)'. We need to
define 'trueify(input, clause)' that modifies input so that 'eval(input,
clause)' is always true afterwards. Since this is SAT, there is no
hope to make this work for all clauses. From the discussions here, a
good algorithm would be:

trueify(input, clause) = match clause with
  AllOf(l) -> for i in l: trueify(input, i)
| AnyOf(l) -> if not eval(input, clause): trueify(input, l[0])
| AtMostOne(l) -> f = (lambda x,y: pass)
  for i in l:
f(input, i)
if eval(input, i): f = falsify
| Implication(useflag, consequence) ->
 if input[useflag]: trueify(input, consequence)
| useflag -> input[useflag] = True


Now you see that for the AtMostOne case we need its dual, the
'falsify(input, clause)' function:

falsify(input, clause) = match clause with
  AllOf(l)   -> falsify(input, l[0])
| AnyOf(l)   -> for i in l: falsify(input, i)
| AtMostOne(l) -> for i in l:
 if eval(input, clause): trueify(input, i)
| Implication(useflag, consequence) ->
 if not input[useflag]: raise "impossible"
 else: falsify(input, consequence)
| useflag -> input[useflag] = False
  


Note how the above favors leftmost in all cases. If it needs to change
something, it always tries to leave the leftmost untouched. Note also
that it processes everything left to right (the AllOf case where
REQUIRED_USE="AllOf(list of clauses)" ).


Now, what I need is basically a -funroll-all-loops on that algorithm
(We're Gentoo!): This explains why I butchered your code any why all
this is so dependent on the solving method chosen.

After unrolling the loops for a given REQUIRED_USE, this code will look
like:
if useflag1: if useflag2: ... : input[foo] = True; input[bar] = False
if baz: if biz: input[x] = True

In my notation this is:
[
  [useflag1, useflag2, ...]?[foo,!bar]
  [baz, biz]?[x]
]

And that's where 'nsolve' comes into play. Portage will do:

trueify(input, requse)
if not eval(input, requse) then rant

We want to guarantee 'requse' is in a form so that portage never has to
rant. My nsolve is only a sufficient condition for that. I was
expecting more like 10% of problematic cases, but considering the data
obtained is that less than .5% of required use in the tree would
require non trivial change and in all the cases I've seen those
required changes are legit, I believe we can move forward with this.


So, the very first thing to do is to agree that the above solver
(the trueify function) is what we want to 

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Wed, 07 Jun 2017 11:27:59 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On śro, 2017-06-07 at 10:17 +0200, Alexis Ballier wrote:
> > > Also, do I presume correctly that for all supported cases (i.e.
> > > those which your nsolve does not reject), solve and nsolve are
> > > compatible? 
> > 
> > Not sure what you mean here. nsolve does not solve anything, it just
> > validates REQUIRED_USE so that it is guaranteed it can be solved.  
> 
> What I mean is whether you can guarantee that:
> 
> a. for every X that nsolve(X) == ok, solve() will be able to find
> a valid solution,

yes

> b. for every X that solve() can solve reliably, nsolve(X) == ok.

no and that's not really possible

> > We first need to define properly & formally how to solve requse
> > constraints if we want ppl to be able to rely on it (or rather write
> > requse that give the expected result).
> > 
> > The way I see it, REQUIRED_USE ast looks like:
> > (assuming ^^ is already expanded to || + ??)
> > 
> > clause =
> > AllOf(list of clauses)
> >   | AnyOf(list of clauses)
> >   | AtMostOne(list of clauses)
> >   | Implication(useflag, clause)
> >   | useflag
> > 
> > Now, portage already has the function 'eval(input, clause)'. We
> > need to define 'trueify(input, clause)' that modifies input so that
> > 'eval(input, clause)' is always true afterwards. Since this is SAT,
> > there is no hope to make this work for all clauses. From the
> > discussions here, a good algorithm would be:
> > 
> > trueify(input, clause) = match clause with
> >   AllOf(l) -> for i in l: trueify(input, i)  
> > > AnyOf(l) -> if not eval(input, clause): trueify(input, l[0])
> > > AtMostOne(l) -> f = (lambda x,y: pass)  
> > 
> >   for i in l:
> > f(input, i)
> > if eval(input, i): f = falsify  
> > > Implication(useflag, consequence) ->  
> > 
> >  if input[useflag]: trueify(input, consequence)  
> > > useflag -> input[useflag] = True  
> > 
> > 
> > Now you see that for the AtMostOne case we need its dual, the
> > 'falsify(input, clause)' function:
> > 
> > falsify(input, clause) = match clause with
> >   AllOf(l)   -> falsify(input, l[0])  
> 
> That's a debatable case. My solve() actually 'falsifies' all
> the subexpressions which might be more reliable.

Best way to debate this is probably to write the implication
translation and feed that to nsolve from a few test cases.

Intuition is that falsifying all of them adds more pressure on the
solver and you might end up failing to solve it for no good reason, so
falsifying only one of them seems safer.

> > > AnyOf(l)   -> for i in l: falsify(input, i)
> > > AtMostOne(l) -> for i in l:  
> > 
> >  if eval(input, clause): trueify(input, i)  
> 
> Do I read this correctly that it pretty much implies enabling the
> first two subexpressions?

or the leftmost first false if one is already true in there, yes

> > > Implication(useflag, consequence) ->  
> > 
> >  if not input[useflag]: raise "impossible"  
> 
> Why impossible? Unless I'm missing something, it's false already.

'foo? bar' is always true if foo is false; so it's impossible to make
it false

it's really a corner case as I think we don't allow nested implications
inside ||, ^^, () or ??, which is the only way to reach that.

> >  else: falsify(input, consequence)  
> > > useflag -> input[useflag] = False  
> 
> Looks mostly sane. You've missed '!flag' but that's trivial to add.

yeah, i realized after sending the email

> > 
> > Note how the above favors leftmost in all cases. If it needs to
> > change something, it always tries to leave the leftmost untouched.
> > Note also that it processes everything left to right (the AllOf
> > case where REQUIRED_USE="AllOf(list of clauses)" ).  
> 
> You need to be able to reorder the clauses to handle use.force
> and use.mask.

Not sure if reorder is the best way. It sure works, but maybe we'd want
a repoman error if e.g. 'foo? ( bar )' is in REQUIRED_USE, bar is
masked but not foo. That'd be a matter of eliminating the constants in
the ast and if we get 'false' for a profile we error out.

> > So, the very first thing to do is to agree that the above solver
> > (the trueify function) is what we want to implement and set this in
> > stone. There's no point in implementing a proper requse checker if
> > the algorithm is meant to change. Having a form

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Wed, 31 May 2017 21:02:24 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On śro, 2017-05-31 at 19:39 +0200, Alexis Ballier wrote:
> > > >  Again, you *need* to process the constraints in order. '!a?
> > > > ( b ) !b? ( a )' is not deterministic when none of a and b are
> > > > enabled otherwise.
> > > 
> > > You can't rely on any particular order of constraints, especially
> > > when eclass stacking comes into play. You could try defining some
> > > constraint- sorting algorithm but it's going to be complex and
> > > it's usefulness will be limited anyway due to various kinds of
> > > grouping.  
> > 
> > 
> > Better have some order: If half of the above constraint comes from
> > ebuild and the other half comes from eclass, then PM might toggle
> > 'a' or 'b' depending on the phase of the moon which is specifically
> > what we're trying to avoid.  
> 
> No, it can't. That's the whole point. The algorithm must be defined so
> that it is always predictable independently of order (maybe except
> the ordering inside ^^, ||, ??) and independently of how it's nested
> (i.e. 'a? ( b? ( c ) )' must give the same result as 'b? ( a? ( c )
> )').

This is a lot of handwaving without real description of how it would
work. This starts to look a lot like confluence in rewriting systems
and you want developers to write only confluent rules and repoman to
decide that. Good luck with that.

Let's look at real world examples:
gtk+:

REQUIRED_USE="
|| ( aqua wayland X )
xinerama? ( X )
"

Let's assume aqua is masked, which reduces to:
REQUIRED_USE="
|| ( wayland X )
xinerama? ( X )
"

With USE='-* xinerama' this one is invalid: applying the first
constraint first enables wayland, then the 2nd enables X, giving a
result of USE="xinerama wayland X". Applying the 2nd first enables X,
then the 1st does nothing, giving a result of USE="xinerama X".


Now the funny one:
$ portageq metadata / ebuild dev-lang/php-7.0.18 REQUIRED_USE
cli? ( ^^ ( readline libedit ) ) truetype? ( gd ) webp? ( gd ) cjk?
( gd ) exif? ( gd ) xpm? ( gd ) gd? ( zlib ) simplexml? ( xml ) soap?
( xml ) wddx? ( xml ) xmlrpc? ( || ( xml iconv ) ) xmlreader? ( xml )
xslt? ( xml ) ldap-sasl? ( ldap ) mhash? ( hash ) phar? ( hash ) qdbm?
( !gdbm ) readline? ( !libedit ) recode? ( !imap !mysqli ) sharedmem?
( !threads ) mysql? ( || ( mysqli pdo ) ) || ( cli cgi fpm apache2
embed phpdbg )



There are probably dozens of ways to make that non deterministic.
Here's one: USE='-*'. Apply '|| ( cli cgi fpm apache2 embed phpdbg )'
last; this enables 'cli'. Since it's the last one, REQUIRED_USE is not
satisfied because of 'cli? ( ^^ ( readline libedit ) )'.
If you process it first, then you enable cli and readline and are good.

> If you start relying on stuff like ordering, you're one step from
> making stuff suddenly fail or change meaning due to minor changes,
> like sorting stuff.

What we want to ensure is that for 2 identical inputs the results are
identical. If ordering is specified, then re-ordering REQUIRED_USE in
an ebuild is not minor anymore. That's a trade off.


> > eclass stacking is not a problem: specify if it's append or prepend
> > and be done.  
> 
> What about multiple inherits with guards? Next thing I know, we end up
> putting REQUIRED_USE outside guards (like we have to do with
> EXPORT_FUNCTIONS now) because you need a specific order, and guards
> make it unpredictable.

guards wont make much of a difference: They'll simply de-duplicate
constraints by keeping only the rightmost one in the prepend case and
the leftmost one in the append case I think.

> > Note that if you want to remove the need for an order, you'll need
> > to ensure that all orderings of the constraints give the same
> > result. It's not sufficient to "only" check all possible inputs.  
> 
> That's the matter of the algorithm.

For what I know, this algorithm does not even exist.

> > Also, what happens if we applied all the constraints and obtained
> > some useflags setting that still fails REQUIRED_USE check ?  
> 
> It can't happen. If you can apply all the constraints, then implicitly
> REQUIRED_USE is satisfied. If you can't apply all the constraints,
> then it just fails. Of course, we want to ultimately avoid that case.

See the php example above. How do you ensure this does not happen?

Note that your assertion 'If you can apply all the constraints, then
implicitly REQUIRED_USE is satisfied.' is false: You're changing the
values of useflags, thus might violate some previously checked
constraint.


> > > The point would be: by definition, the solver should be able to
> > > touch *any* USE flag. If it can't, it mostly implies that you
> >

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Sun, 4 Jun 2017 10:59:38 +0200
Alexis Ballier <aball...@gentoo.org> wrote:

> Here's a quick n dirty code to play with, based on yours: 
> https://github.com/aballier/required-use

I've run that on the whole tree (considering all ebuilds with non
empty REQUIRED_USE), some stats:

$ time python3 classify.py requsel 
Stats:
Parse error: 16
Good: 8316
Need topo sort: 140
Cyclic: 57

real0m2.996s
user0m2.950s
sys 0m0.004s


Running time is good I think.
Parse error is some nested construct not supported by your parser that
I have not fixed either.


~2.5% simply need to be reordered. ~0.6% require more clever thinking.


Let's have a look at a few of them:

sys-fs/cryptsetup-1.7.5
^^ ( gcrypt kernel nettle openssl ) python? ( ||
( python_targets_python2_7 python_targets_python3_4
python_targets_python3_5 python_targets_python3_6 ) )
static? ( !gcrypt )

USE='-* static' is painful: the first ^^ will enable gcrypt, static?
( !gcrypt ) will disable it. Reordering the ^^ so that kernel appears
first fixes the cycle.


media-libs/mesa-13.0.6
|| ( gcrypt libressl nettle openssl ) d3d9?
( dri3 gallium ) llvm? ( gallium ) opencl? ( gallium llvm ) openmax?
( gallium ) gles1? ( egl ) gles2? ( egl ) vaapi? ( gallium ) vdpau?
( gallium ) vulkan? ( video_cards_i965 ) wayland? ( egl gbm ) xa?
( gallium ) video_cards_freedreno? ( gallium ) video_cards_intel?
( classic ) video_cards_i915? ( || ( classic gallium ) )
video_cards_i965? ( classic ) video_cards_nouveau? ( || ( classic
gallium ) ) video_cards_radeon? ( || ( classic gallium ) gallium?
( x86? ( llvm ) amd64? ( llvm ) ) ) video_cards_r100? ( classic )
video_cards_r200? ( classic ) video_cards_r300? ( gallium x86? ( llvm )
amd64? ( llvm ) ) video_cards_r600? ( gallium ) video_cards_radeonsi?
( gallium llvm ) video_cards_vmware? ( gallium )

The cycle is mostly due to:
$ python3 nsolve.py 'llvm? ( gallium ) gallium? ( llvm )'
[...]
toposort.CircularDependencyError: Circular dependencies exist among
these items: {[gallium]? => [llvm]:{[llvm]? => [gallium]}, [llvm]? =>
[gallium]:{[gallium]? => [llvm]}}


This is something I had overseen when replacing 'a q'_j is some p_i and
one of q_1 ... q_m might be false' by only 'a q'_j is some p_i'; it can
be replaced without changing anything in the way PM would solve it by
"a q'_j is some p_i and the set of {q_j} is not a subset of q' union
p'" (that is, {q_i} is not trivially true if the 2nd clause is
applied). Extending that, we get those stats:

Stats:
Parse error: 16
Good: 8325
Need topo sort: 146
Cyclic: 42

real0m1.575s
user0m1.563s
sys 0m0.012s


Now we seem to get only valid warnings (I have not checked them all
though):

sys-firmware/seabios-1.10.1:
debug? ( !binary ) !amd64? ( !x86? ( binary ) )

USE="debug -amd64 -x86" ?


sci-libs/ceres-solver-1.11.0:
test? ( gflags ) sparse? ( lapack ) abi_x86_32? ( !sparse !lapack )

USE='-* sparse -lapack abi_x86_32' shows a conflict between the last 2
clauses. Better write:

test? ( gflags ) !abi_x86_32? ( sparse? ( lapack ) ) abi_x86_32?
( !sparse !lapack )

which makes the test work



mail-mta/exim-4.89
dane? ( !gnutls ) dmarc? ( spf dkim ) pkcs11? ( gnutls ) spf?
( exiscan-acl ) srs? ( exiscan-acl )

USE="dane pkcs11" ?

sci-libs/hdf5-1.8.18:
threads? ( !cxx !mpi !fortran !hl ) fortran2003? ( fortran )

USE="threads fortran2003" ?





I'll let you play with it, but for me it seems this would work quite
nicely.


Alexis.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Mon, 29 May 2017 17:33:13 +0200
Michał Górny  wrote:

> 2.4. Backwards compatibility
> 


Considering the discussions in that thread, a natural way to move
forward now seems to be:

1. Define a way to solve constraints deterministically
2. Get a warning check into repoman to ensure REQUIRED_USE are solved
   that way.
3. Get those repoman warnings fixed.
4. Implement an optional FEATURES in portage for automatic solving
   of REQUIRED_USE.
5. If all goes well, mandate that solving in next EAPI and make the
   repoman warning an error for those EAPIs.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Fri, 09 Jun 2017 14:54:07 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On pią, 2017-06-09 at 13:41 +0200, Alexis Ballier wrote:
> > On Fri, 09 Jun 2017 11:19:20 +0200
> > Michał Górny <mgo...@gentoo.org> wrote:
> >   
> > > On śro, 2017-06-07 at 11:56 +0200, Alexis Ballier wrote:  
> > > > On Wed, 07 Jun 2017 11:27:59 +0200
> > > > Michał Górny <mgo...@gentoo.org> wrote:
> > > > 
> > > > > On śro, 2017-06-07 at 10:17 +0200, Alexis Ballier wrote:
> > > > > > > Also, do I presume correctly that for all supported cases
> > > > > > > (i.e. those which your nsolve does not reject), solve and
> > > > > > > nsolve are compatible? 
> > > > > > 
> > > > > > Not sure what you mean here. nsolve does not solve
> > > > > > anything, it just validates REQUIRED_USE so that it is
> > > > > > guaranteed it can be solved.  
> > > > > 
> > > > > What I mean is whether you can guarantee that:
> > > > > 
> > > > > a. for every X that nsolve(X) == ok, solve() will be able to
> > > > > find a valid solution,
> > > > 
> > > > yes
> > > > 
> > > > > b. for every X that solve() can solve reliably, nsolve(X) ==
> > > > > ok.
> > > > 
> > > > no and that's not really possible
> > > 
> > > I thought so. To expand it a little, could you confirm whether I
> > > correctly presume that:
> > > 
> > > a. for all 'good' results, the plain iterative solve() should be
> > > able to find a solution with a single iteration?  
> > 
> > yes that's the point of it
> >   
> > > b. for all 'need toposort' results, the solve() should be able to
> > > find a solution with n>=1 iterations?  
> > 
> > yes; though n is only bounded by the # of clauses (expanded as
> > implications) while it can be 1 if reorderer properly; I wouldn't
> > bother doing several iterations and just reject that at repoman
> > side since it's easily solved  
> 
> I would prefer not having Portage fail randomly on local ebuilds where
> the cost of multiple iterations is practically zero, and certainly
> it's not worth the effort to ensure a particular ordering.


Yep, makes sense.

It makes it harder to guess though: "a? ( b ) !b? ( a )" will
enable both a and b if USE=-*. This is not obvious to me from a single
read of the constraint. I need to tell myself "Oh, I've now enabled 'a'
and I need to recheck the first clause.".

I was thinking more from a spec perspective where I would definitely
not want to mandate PM to do fixpoint(solve) or having to detect a
cycle before failing. Also, I believe that forcing a single pass solver
will help in ensuring that the solver solves it the way the developer
writing the constraint meant it.


[...]
> > > > > > > Implication(useflag, consequence) ->  
> > > > > > 
> > > > > >  if not input[useflag]: raise
> > > > > > "impossible"  
> > > > > 
> > > > > Why impossible? Unless I'm missing something, it's false
> > > > > already.
> > > > 
> > > > 'foo? bar' is always true if foo is false; so it's impossible to
> > > > make it false
> > > 
> > > Yes, you are correct. I was actually thinking of 'if LHS is
> > > false, we do not enforce RHS'.  
> > 
> > I'm wrong actually. It can be falsified by setting foo to True and
> > bar to False. More on it below.  
> 
> Well, I'm not sure if it can still plainly apply here but the generic
> contract was that in implication clauses only RHS is mutable.


Yeah, that's basically what I inferred from trying to figure out the
meaning out of it later on.

[...]
> > > > > > 
> > > > > > Note how the above favors leftmost in all cases. If it
> > > > > > needs to change something, it always tries to leave the
> > > > > > leftmost untouched. Note also that it processes everything
> > > > > > left to right (the AllOf case where
> > > > > > REQUIRED_USE="AllOf(list of clauses)" ).  
> > > > > 
> > > > > You need to be able to reorder the clauses to handle use.force
> > > > > and use.mask.
> > > > 
> > > > Not sure if reorder is the best way. It sure works, but maybe
> > > > we

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Mon, 12 Jun 2017 21:17:16 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On pon, 2017-06-12 at 11:08 +0200, Alexis Ballier wrote:
> > On Sun, 11 Jun 2017 18:05:18 +0200
> > Alexis Ballier <aball...@gentoo.org> wrote:
> >   
> > > I think this handles all the cases. I'll try to update the repo
> > > with that algo.  
> > 
> > I've updated my fork. It'd be good to merge it and rebase solve() on
> > top of the output of to_impl.convert_to_implications if you're happy
> > with it.
> > 
> > $ time python3 classify.py requsel 
> > Stats:
> > Parse error: 0
> > Good: 8334
> > Need topo sort: 152
> > Cyclic: 43
> > 
> > real0m1.874s
> > user0m1.869s
> > sys 0m0.005s
> > 
> > 
> > 
> > It works better, no more parse error, and only 43 problematic
> > cases.  
> 
> Thanks for doing it. It's certainly an interesting case study. I've
> merged it and pushed the result.
> 
> However, I personally think it's only going to work against your case.

What do you mean here ?

> You can clearly see now how complex the code has become. Even
> in the pseudo-ocaml you presented it already is complex. Now imagine
> having to retype it in cleartext suitable for the PMS.

The code needs cleanup and probably some refactoring. I think it can be
made much simpler.

> I've actually started typing the initial specification yesterday [1].
> As you can see, banning the extra constraints has made the algorithms
> much simpler. In particular:
> 
> 1. You do not have to define 'falsify' for anything other than pure
> flags -- which makes it easy to inline it.
> 
> 2. ||, ??, ^^ groups are only flat lists of flags -- which makes
> reordering and processing them trivial.
> 
> 3. The algorithm is recursive only on USE-conditional groups. This
> makes it trivial to make it iterative. Optimizations become trivially
> possible.


While you're right in one sense, you're mixing two different things
here. What you wrote *is* recursive. It does not recurse just because
you're assuming a restricted syntax. You're only saving two things:
you don't need to define how to enforce to false (that is 3 lines not 3
pages :=) ) and you're avoiding the nested use conditionals that are
already ill defined per the current spec (foo? bar is equivalent to
&& ( foo bar ) when nested) which I believe is already another problem.

Then, remember how I wanted to be much more drastic than you in the
beginning by killing all ||,&&,^^ etc. and keep only use
conditionals in REQUIRED_USE ?  Well, that's where the complexity comes.
The whole deal then is to define rewriting rules for the AST so that
the algorithm you describe executes the exact same instructions but the
new AST only has use conditionals. This is more like writing a compiler
for the spec, so this does not belong to the spec and there is no
issue here.

[BTW: checking the rewrite rules behave properly is what I meant by
rebasing solve() on top of it and being happy with it]



> Nevertheless, feel free to play with the full implementation. If
> you're interested, you can play with the complex cases more. In
> particular, I'm wondering whether nsolve will actually consider most
> of them solvable.
> 
> As for the results, I think it is the point where we start preparing
> pull requests with REQUIRED_USE changes to see whether the developers
> agree with such changes.

If by that you also include code cleanup and writing tests then yes :)

> [1]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse

I really don't like the reordering thing. Even the restricted
syntax does not fix the issue with '^^ ( a b ) b? ( a )' already
mentioned here. It'd be much better and simpler for the spec just to
assign a fixed value and use the solving rules with those.


Alexis.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Thu, 15 Jun 2017 17:13:57 +0100
Ciaran McCreesh <ciaran.mccre...@googlemail.com> wrote:

> On Thu, 15 Jun 2017 18:07:00 +0200
> Alexis Ballier <aball...@gentoo.org> wrote:
> > > The best way to convince me is through valid examples.
> > 
> > It is also easier to be convinced when you try to understand and ask
> > for clarifications instead of just rejecting without thinking :)  
> 
> The problem with this entire proposal is that it's still in "well I
> can't think of how it could possibly go wrong" territory. We need a
> formal proof that it's sound. History has shown that if something can
> be abused by Gentoo developers, it will be abused...

Had you read the thread you would have noticed that I provided an
algorithm giving sufficient conditions for the solver to work. That
is, if developers pay attention to repoman warnings/errors, it will
never fail. Obviously, since we're still in the SAT space, you can
ignore the errors and make it fail, but it'll never be worse than what
we currently have.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Thu, 15 Jun 2017 17:22:26 +0100
Ciaran McCreesh <ciaran.mccre...@googlemail.com> wrote:

> On Thu, 15 Jun 2017 18:19:04 +0200
> Alexis Ballier <aball...@gentoo.org> wrote:
> > On Thu, 15 Jun 2017 17:13:57 +0100
> > Ciaran McCreesh <ciaran.mccre...@googlemail.com> wrote:  
> > > On Thu, 15 Jun 2017 18:07:00 +0200
> > > Alexis Ballier <aball...@gentoo.org> wrote:
> > > > > The best way to convince me is through valid examples.
> > > > 
> > > > It is also easier to be convinced when you try to understand and
> > > > ask for clarifications instead of just rejecting without
> > > > thinking :)  
> > > 
> > > The problem with this entire proposal is that it's still in "well
> > > I can't think of how it could possibly go wrong" territory. We
> > > need a formal proof that it's sound. History has shown that if
> > > something can be abused by Gentoo developers, it will be
> > > abused...
> > 
> > Had you read the thread you would have noticed that I provided an
> > algorithm giving sufficient conditions for the solver to work. That
> > is, if developers pay attention to repoman warnings/errors, it will
> > never fail. Obviously, since we're still in the SAT space, you can
> > ignore the errors and make it fail, but it'll never be worse than
> > what we currently have.  
> 
> You have shown that you produce a solution, not the solution that's
> actually wanted.
> 

Since 'wanted' is still undefined, I'd say it produces the defined
solution and you can adapt to the definition to get what you want.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Thu, 15 Jun 2017 17:59:13 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On śro, 2017-06-14 at 16:09 +0200, Alexis Ballier wrote:
> > On Wed, 14 Jun 2017 15:57:38 +0200
> > Michał Górny <mgo...@gentoo.org> wrote:
> > [...]  
> > > > [...]
> > > > > > > > > [1]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse  
> > > > > > > > >   
> > > > > > > > 
> > > > > > > > I really don't like the reordering thing. Even the
> > > > > > > > restricted syntax does not fix the issue with '^^ ( a b
> > > > > > > > ) b? ( a )' already mentioned here. It'd be much better
> > > > > > > > and simpler for the spec just to assign a fixed value
> > > > > > > > and use the solving rules with those.
> > > > > > > 
> > > > > > > You're not going to convince me by providing examples
> > > > > > > that are utterly broken by design and
> > > > > > > meaningless ;-).  
> > > > > > 
> > > > > > Well... if it's so obvious that the example is broken by
> > > > > > design that you don't even bother to explain why, I assume
> > > > > > you have an algorithm for that. Where is the code ? What
> > > > > > are the numbers ? How many ebuilds might fail after
> > > > > > reordering ? How can this be improved ?  
> > > > > 
> > > > > Are you arguing for the sake of arguing here? I just presumed
> > > > > that this example is so obviously broken there is no point
> > > > > wasting any more time on it. The code of nsolve clearly
> > > > > detects that, so I don't really understand what you're trying
> > > > > to prove here.
> > > > 
> > > > Those are real questions. You should take breath, think a bit
> > > > about it, and try to run the 2 possible orderings of the ^^
> > > > through nsolve or even solve.py. They both are very happy (and
> > > > are right to be) with the above ordering. You might want to
> > > > think a bit more about what is the relation between this broken
> > > > 10 chars example and the 10 lines python targets one below.
> > > > 
> > > > You should also realize that all the above questions have
> > > > already been answered in length if you do as I suggest.
> > > 
> > > No. I have already spent too much time on this. We're already long
> > > past all useful use cases, and now I feel like you're going to
> > > argue to death just to find a perfect algorithm that supports
> > > every absurd construct anyone can even write, if only to figure
> > > out the construct is completely useless.  
> > 
> > I'm not going to argue to death. It's already proven reordering is
> > broken.
> >   
> > > If you want to play with it more, then please by all means do
> > > so.  
> > 
> > There is nothing to do for reordering. It's broken by design.
> >   
> > > However, do not expect me to waste any more of my time on it. I've
> > > done my part, the code works for all reasonable use cases and
> > > solves all the problems I needed solving. If you want more, then
> > > it's your job to do it and solve the resulting issues.  
> > 
> > Like... writing code handling all the cases and describing how it
> > works ? We're past that. The only thing we're not past is that you
> > fail to understand it and attempt to block it.
> >   
> 
> Then please provide a single valid example that:

app-text/wklej-0.2.1-r1 ^^ ( python_single_target_pypy
python_single_target_pypy3 python_single_target_python2_7
python_single_target_python3_4 python_single_target_python3_5
python_single_target_python3_6 ) python_single_target_pypy?
( python_targets_pypy ) python_single_target_pypy3?
( python_targets_pypy3 ) python_single_target_python2_7?
( python_targets_python2_7 ) python_single_target_python3_4?
( python_targets_python3_4 ) python_single_target_python3_5?
( python_targets_python3_5 ) python_single_target_python3_6?
( python_targets_python3_6 ) vim? ( ^^ ( python_single_target_python2_7
) )


Simplified as:
^^ ( a b ) c? ( b )

(see the pattern now ? :) )

> a. is completely 'correct' (that is, provides a valid, predictable
> and acceptable solution) with the default ordering O_a,

c? ( b ) ^^ ( b a )


> b. is not 'correct' with at least one reordering O_b (assuming only
> ||, ^^, ?? is subject to reordering),

c? ( b ) ^^ ( a b )

> 
> c. nsolve reports O_a as all good, and O_b as not good.

I'll let you run this. It does.

> The best way to convince me is through valid examples.


It is also easier to be convinced when you try to understand and ask
for clarifications instead of just rejecting without thinking :)

Alexis.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Thu, 15 Jun 2017 19:38:48 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On czw, 2017-06-15 at 18:07 +0200, Alexis Ballier wrote:
> > On Thu, 15 Jun 2017 17:59:13 +0200
> > Michał Górny <mgo...@gentoo.org> wrote:
> >   
> > > On śro, 2017-06-14 at 16:09 +0200, Alexis Ballier wrote:  
> > > > On Wed, 14 Jun 2017 15:57:38 +0200
> > > > Michał Górny <mgo...@gentoo.org> wrote:
> > > > [...]
> > > > > > [...]  
> > > > > > > > > > > [1]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse  
> > > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > I really don't like the reordering thing. Even the
> > > > > > > > > > restricted syntax does not fix the issue with '^^
> > > > > > > > > > ( a b ) b? ( a )' already mentioned here. It'd be
> > > > > > > > > > much better and simpler for the spec just to assign
> > > > > > > > > > a fixed value and use the solving rules with
> > > > > > > > > > those.  
> > > > > > > > > 
> > > > > > > > > You're not going to convince me by providing examples
> > > > > > > > > that are utterly broken by design and
> > > > > > > > > meaningless ;-).
> > > > > > > > 
> > > > > > > > Well... if it's so obvious that the example is broken by
> > > > > > > > design that you don't even bother to explain why, I
> > > > > > > > assume you have an algorithm for that. Where is the
> > > > > > > > code ? What are the numbers ? How many ebuilds might
> > > > > > > > fail after reordering ? How can this be
> > > > > > > > improved ?
> > > > > > > 
> > > > > > > Are you arguing for the sake of arguing here? I just
> > > > > > > presumed that this example is so obviously broken there
> > > > > > > is no point wasting any more time on it. The code of
> > > > > > > nsolve clearly detects that, so I don't really understand
> > > > > > > what you're trying to prove here.  
> > > > > > 
> > > > > > Those are real questions. You should take breath, think a
> > > > > > bit about it, and try to run the 2 possible orderings of
> > > > > > the ^^ through nsolve or even solve.py. They both are very
> > > > > > happy (and are right to be) with the above ordering. You
> > > > > > might want to think a bit more about what is the relation
> > > > > > between this broken 10 chars example and the 10 lines
> > > > > > python targets one below.
> > > > > > 
> > > > > > You should also realize that all the above questions have
> > > > > > already been answered in length if you do as I
> > > > > > suggest.  
> > > > > 
> > > > > No. I have already spent too much time on this. We're already
> > > > > long past all useful use cases, and now I feel like you're
> > > > > going to argue to death just to find a perfect algorithm that
> > > > > supports every absurd construct anyone can even write, if
> > > > > only to figure out the construct is completely useless.
> > > > 
> > > > I'm not going to argue to death. It's already proven reordering
> > > > is broken.
> > > > 
> > > > > If you want to play with it more, then please by all means do
> > > > > so.
> > > > 
> > > > There is nothing to do for reordering. It's broken by design.
> > > > 
> > > > > However, do not expect me to waste any more of my time on it.
> > > > > I've done my part, the code works for all reasonable use
> > > > > cases and solves all the problems I needed solving. If you
> > > > > want more, then it's your job to do it and solve the
> > > > > resulting issues.
> > > > 
> > > > Like... writing code handling all the cases and describing how
> > > > it works ? We're past that. The only thing we're not past is
> > > > that you fail to understand it and attempt to block it.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Thu, 15 Jun 2017 18:04:35 +0100
Ciaran McCreesh <ciaran.mccre...@googlemail.com> wrote:

> On Thu, 15 Jun 2017 18:55:45 +0200
> Alexis Ballier <aball...@gentoo.org> wrote:
> > The guarantee comes from the fact that the output is always in the
> > space of all possible inputs from the user. So, if some output will
> > kill a kitten, so does some input.  
> 
> USE=minimal
> USE=mips
> USE=-ssl
> 

So what?

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Thu, 15 Jun 2017 18:48:42 +0100
Ciaran McCreesh <ciaran.mccre...@googlemail.com> wrote:

> On Thu, 15 Jun 2017 19:30:02 +0200
> Alexis Ballier <aball...@gentoo.org> wrote:
> > On Thu, 15 Jun 2017 18:04:35 +0100
> > Ciaran McCreesh <ciaran.mccre...@googlemail.com> wrote:  
> > > On Thu, 15 Jun 2017 18:55:45 +0200
> > > Alexis Ballier <aball...@gentoo.org> wrote:
> > > > The guarantee comes from the fact that the output is always in
> > > > the space of all possible inputs from the user. So, if some
> > > > output will kill a kitten, so does some input.  
> > > 
> > > USE=minimal
> > > USE=mips
> > > USE=-ssl
> > > 
> > 
> > So what?  
> 
> So, if the aim of this solution is to make things better for the user,
> what are you doing to establish that this will make things better for
> the user instead of recommending something awful?
> 

Considering that the way you write REQUIRED_USE defines how the solver
behaves, your problem is ill defined.

If I try to ask my crystal ball, I would say: USE=mips is either masked
or forced so never an option. Developer would not want USE=minimal to
be toggled randomly so would write a constraint so that it always
appears e.g. on the left part of an implication.

Re: [gentoo-dev] [RFC] toolchain-funcs.eclass / toolchain-glibc.eclass - gcc-6 bugfixes and updates

[Alexis Ballier]

On Fri, 16 Jun 2017 14:25:02 +0100
"M. J. Everitt"  wrote:

> On 16/06/17 09:27, Matthias Maier wrote:
> > On Wed, Jun 14, 2017, at 18:15 CDT, Matthias Maier
> >  wrote: 
> >> Hello all,
> >>
> >> this is a series of patches against the toolchian-funcs and
> >> toolchain-glibc eclasses, most notably
> >>  
> > Pushed.
> >
> > Best,
> > Matthias
> >  
> .. That was quick ...
> 
> I swore there was something in the devmanual about a nice long period
> of bikeshedding before changes to eclasses were approved ..
> 

Maintainers are not even required to send their changes to the ML
before committing. They do it because they think it makes sense to
have some review for eclasses having a wide usage. Sending trivial
changes here instead of b.g.o can be seen as spam.

Re: [gentoo-dev] Hardening a default profile

[Alexis Ballier]

On Sat, 17 Jun 2017 14:43:24 +0300
Andrew Savchenko  wrote:

> On Thu, 15 Jun 2017 19:52:07 -0500 Matthias Maier wrote:
> > > there should be a way of turning these off systematically.  the
> > > advantage of the current hardened gcc specs is that one can switch
> > > between them using gcc-config.  if these are forced on for the
> > > default profile then there will be no easy way to systematically
> > > turn them off.  
> > 
> > No - there won't be an easy way for systematically turning off
> > SSP and PIE in 17.0 profiles [1,2].
> > 
> > The hardened toolchain with its different gcc profiles came from a
> > time where SSP and PIE were relatively new security features and a
> > certain amount of fine-grained control was needed. Further, at that
> > time we were talking about external patches against gcc. Nowadays
> > everything is upstreamed and (almost) no patches to gcc for
> > hardened profiles are applied any more.
> > 
> > Given the fact that all major linux distributions are following the
> > path of improved default hardening features (see for example [1])
> > and that we have been using ssp/pie in hardened profiles for years
> > now the purpose of fine-grained control over ssp/pie is also highly
> > questionable.
> > 
> > The consensus at the moment is that PIE and SSP (as well as stricter
> > linker flags) will soon be standard (or, actually *are* already
> > standard) compilation options. A per-package override (if
> > absoluetely needed) is fine - and, in fact, already in place
> > everywhere where needed.  
> 
> Gentoo is all about choice, remember? :)
> 
> It is really good to have them by default, it is bad to force them
> on everyone. Security is not always of paramount importance
> comparing to other factors, sometimes performance matters more,
> e.g. in isolated and restricted non-public HPC environment.
> 
> PIE, SSP may lead up to 8% of performance loss[1]. The
> stack-protector (especially stack-protector-all or -strong) may
> cause even more damage. For compute nodes this may be equivalent to
> millions USD loss (depends on the system scale of course).

This can probably be fixed by a gcc-config target disabling those as it
used to be the case on hardened

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Fri, 09 Jun 2017 11:19:20 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On śro, 2017-06-07 at 11:56 +0200, Alexis Ballier wrote:
> > On Wed, 07 Jun 2017 11:27:59 +0200
> > Michał Górny <mgo...@gentoo.org> wrote:
> >   
> > > On śro, 2017-06-07 at 10:17 +0200, Alexis Ballier wrote:  
> > > > > Also, do I presume correctly that for all supported cases
> > > > > (i.e. those which your nsolve does not reject), solve and
> > > > > nsolve are compatible?   
> > > > 
> > > > Not sure what you mean here. nsolve does not solve anything, it
> > > > just validates REQUIRED_USE so that it is guaranteed it can be
> > > > solved.
> > > 
> > > What I mean is whether you can guarantee that:
> > > 
> > > a. for every X that nsolve(X) == ok, solve() will be able to find
> > > a valid solution,  
> > 
> > yes
> >   
> > > b. for every X that solve() can solve reliably, nsolve(X) == ok.  
> > 
> > no and that's not really possible  
> 
> I thought so. To expand it a little, could you confirm whether I
> correctly presume that:
> 
> a. for all 'good' results, the plain iterative solve() should be able
> to find a solution with a single iteration?

yes that's the point of it

> b. for all 'need toposort' results, the solve() should be able to find
> a solution with n>=1 iterations?

yes; though n is only bounded by the # of clauses (expanded as
implications) while it can be 1 if reorderer properly; I wouldn't bother
doing several iterations and just reject that at repoman side since it's
easily solved

> c. all of 'circular' results have at least one USE flag combination
> that can not be solved by solve()?

In theory no as that would imply your 1st b. In practice, I've only
seen cases like that.



> > > > We first need to define properly & formally how to solve requse
> > > > constraints if we want ppl to be able to rely on it (or rather
> > > > write requse that give the expected result).
> > > > 
> > > > The way I see it, REQUIRED_USE ast looks like:
> > > > (assuming ^^ is already expanded to || + ??)
> > > > 
> > > > clause =
> > > > AllOf(list of clauses)
> > > >   | AnyOf(list of clauses)
> > > >   | AtMostOne(list of clauses)
> > > >   | Implication(useflag, clause)
> > > >   | useflag
> > > > 
> > > > Now, portage already has the function 'eval(input, clause)'. We
> > > > need to define 'trueify(input, clause)' that modifies input so
> > > > that 'eval(input, clause)' is always true afterwards. Since
> > > > this is SAT, there is no hope to make this work for all
> > > > clauses. From the discussions here, a good algorithm would be:
> > > > 
> > > > trueify(input, clause) = match clause with
> > > >   AllOf(l) -> for i in l: trueify(input, i)
> > > > > AnyOf(l) -> if not eval(input, clause): trueify(input,
> > > > > l[0]) AtMostOne(l) -> f = (lambda x,y: pass)
> > > > 
> > > >   for i in l:
> > > > f(input, i)
> > > > if eval(input, i): f = falsify
> > > > > Implication(useflag, consequence) ->
> > > > 
> > > >  if input[useflag]: trueify(input,
> > > > consequence)
> > > > > useflag -> input[useflag] = True
> > > > 
> > > > 
> > > > Now you see that for the AtMostOne case we need its dual, the
> > > > 'falsify(input, clause)' function:
> > > > 
> > > > falsify(input, clause) = match clause with
> > > >   AllOf(l)   -> falsify(input, l[0])
> > > 
> > > That's a debatable case. My solve() actually 'falsifies' all
> > > the subexpressions which might be more reliable.  
> > 
> > Best way to debate this is probably to write the implication
> > translation and feed that to nsolve from a few test cases.  
> 
> Exactly my thought. Since both algorithms should be kept in sync, it
> probably makes sense to figure out the translation first and use
> whatever makes it consistent without hacking on the translation hard.
> I'll try to figure it out on paper today.
> 
> > Intuition is that falsifying all of them adds more pressure on the
> > solver and you might end up failing to solve it for no good reason,
> > so falsifying only one of them seems 

Re: [gentoo-dev] [RFC] Restricting allowed nesting of REQUIRED_USE

[Alexis Ballier]

On Sat, 10 Jun 2017 00:30:07 +0200
Michał Górny  wrote:

> Hi, everyone.
> 
> As you may or may not know, PMS says rather little about REQUIRED_USE
> [1,2]. The largest past of the definition is shared with other
> dependency-like specifications [3].
> 
> Similarly to regular dependency specifications, PMS is rather lax in
> nesting things. While this isn't a major problem for dependencies
> where the syntax is limited to any-of, all-of and USE-conditional
> groups (though it already may cause some confusion there), it allows
> quite a bit of a mayhem with the full set of REQUIRED_USE clauses.
> 
> We have five different kinds of clauses there: any-of, at-most-one-of,
> exactly-one-of, all-of and USE-conditional. Furthermore, unlike
> in dependency specifications, the last type is circular with flags
> enforced by REQUIRED_USE constraints.
> 
> While nesting all of those clauses is technically valid (and can be
> logically verified), it has no proven usability. As a result, it is
> either not used at all or has a few use cases which suffer from poor
> readability and can be easily replaced with *much simpler*
> constraints. In fact, allowing them is not solving any issues but
> only introducing more when developers fail at using them.
> 
> I would therefore like to discuss restricting nesting of REQUIRED_USE
> clauses.
> 
> 
> What's my take in this? As you have probably noticed (and stopped
> reading) I am working with Alexis on solving REQUIRED_USE constraints
> automatically. We're working towards a few goals: keeping things
> simple, giving predictable solutions, and being able to automatically
> validate whether the constraints are solvable.
> 
> While we're near solving almost everything, the complex clauses add
> unnecessary complexity (both to the spec and to the code) which does
> not really benefit anyone, and bring solutions that can not be
> predictable because the clauses are ambiguous by design.
> 
> To avoid adding this complexity, it would be reasonable to ban at
> least some of the non-useful combinations. This means either banning
> them completely (in a future EAPI + possibly repoman) so that
> developers do not even try to use them, or disabling autosolving when
> they are being used).

I'm not sure it is worth restricting too much in the spec, at least now.
It certainly has benefits, but the extra complexity they add forces to
thoroughly think about how to design the proper solver, which I don't
see as a bad thing.

The main problem is that the solver, in those complex cases, will
provide results that, at least to me, do not seem natural.

It'd probably be a very good thing to restrict the allowed nesting
since they add (runtime) complexity to the solver & checker, like a
repoman warning and/or error, depending on some threshold.

On the other hand, the syntax you propose seems way much saner. I like
it and consider it is a good way to guide developers into writing
easily predictable constraints. However, I would not disable auto
solving when this does not match, I would have a generic algorithm, and
wait for field testing before deciding if people are happy with the
results or if they prefer to rewrite their constraints in a saner way
to have a straightforward interpretation of the solver results.

Bests,

Alexis.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Fri, 09 Jun 2017 18:21:50 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> (cut off the parts where I agree and there's nothing to add)
> 
> On pią, 2017-06-09 at 16:16 +0200, Alexis Ballier wrote:
> > [...]  
> > > > In your example above, we'd call 'nsolve("|| ( X )")' and
> > > > 'nsolve("|| ( Y )")' (or even simpler, depending on how
> > > > simplify() is defined). If both X and Y are masked on a
> > > > profile, then that'd reduce to 'nsolve("False")' which would
> > > > rant.
> > > 
> > > So you're talking about reducing prior to transforming? Yes,
> > > that'd work. As I mentioned in one of the first mails wrt my
> > > reference implementation, I've used reordering (stable sort)
> > > instead of reducing since it was simpler.
> > > 
> > > If you reduce (simplify), you need to account for special cases
> > > like getting '|| ()' etc. If you reorder only, things just fail
> > > the normal way.  
> > 
> > While the reordering idea seems nice as it factors both user
> > preference and masks, the problem with reordering is that nothing
> > guarantees that the solver won't try to enable a masked flag. We'd
> > have to deal with that somehow.  
> 
> Well, yes and no.
> 
> The algorithm always needs to account for the possibility of
> constraints altering immutable flags. Of course, there's more than
> one way of doing it.
> 
> AFAIU you are aiming for separate processing of immutable flags
> and explicit failure if the constraints would attempt to force value
> of those flags. That surely makes sense for verification.

The semi-hidden goal here for me is to have purely ast rewriting rules
giving a list of implications. This makes the solver trivial as those
are read as "if condition then constraint" and can be used as input for
the checker. Failing that, this would need to be done on the checker
side anyway and then we might run into problems like the checker not
really checking reality since the solver behaves a little bit
differently.

> My approach was simpler -- marking the flags immutable, and failing if
> something actually tries to alter their value. I think it's a simpler
> solution for the plain solver and it works as well. After all, we do
> not want the solver to attempt to find workarounds for the problem
> but just fail.

This should be equivalent: masked flags will be toggled as last resort
and fail; eliminated flags will not be toggled at all and fail if
having them as immutable causes a contradiction

> The above applies clearly to the plain conflicts such as:
> 
>   foo? ( bar )
> 
> where bar is immutable. The n-ary operators can be flexed a little.
> That's what reordering achieves -- it makes sure they come as the most
> or the least preferred as appropriate. Which means that the same
> algorithm either succeeds (by not having to touch them) or fails at
> attempting to flip them.
> 
> Think of:
> 
>   ?? ( a b c )
> 
> with both b in use.force. This gets reordered to:
> 
>   ?? ( b c a )
> 
> The order between b doesn't matter. Since b comes first now, it
> forces c being disabled. Since c is immutable, the solver fails with
> ImmutabilityError. We solve the problem with minimal code redundancy.

Considering that code should ideally be checked, that'd be '?? ( a true
true )' reducing to 'false' and a repoman error.


> > I think reordering should be kept for user preferences
> > (soft-enable/soft-disable) while masks for hard-no's or hard-yes'es.
> > 
> > 
> > Be careful with reordering though:
> > '^^ ( a b ) b? ( a )' can be solved in one pass.
> > (it disables b if both are set and enables a if none are set)
> > 
> > while:
> > '^^ ( b a ) b? ( a )' loops
> > (if both are set it disables 'a' for the 1st clause but then
> > enables it for the 2nd)
> > 
> > This is not checked by nsolve().  
> 
> Yes, this is a problem I was considering. I planned to think a bit if
> we could possibly generate some more complex transformations to
> trigger nsolve to notice this kind of issues.


Except feeding the n! possible reorderings to nsolve() and checking them
all I don't see many other possibilities.

We could think about a transformation that would be order agnostic,
like '|| ( a b c )' giving the same output as '|| ( b c a )' but then
this would not express any preference anymore. Remember: The whole
point of the solver is to break the symmetry of SAT formulas so that
there is a natural way to solve them instead of just "figure out some
useflags that make it work". In other words, order does actually
matter a lot, otherwise you fall into th

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Sun, 11 Jun 2017 18:05:18 +0200
Alexis Ballier <aball...@gentoo.org> wrote:

> I think this handles all the cases. I'll try to update the repo with
> that algo.

I've updated my fork. It'd be good to merge it and rebase solve() on
top of the output of to_impl.convert_to_implications if you're happy
with it.

$ time python3 classify.py requsel 
Stats:
Parse error: 0
Good: 8334
Need topo sort: 152
Cyclic: 43

real0m1.874s
user0m1.869s
sys 0m0.005s



It works better, no more parse error, and only 43 problematic cases.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Wed, 14 Jun 2017 15:57:38 +0200
Michał Górny  wrote:

> No. I have already spent too much time on this. We're already long
> past all useful use cases

Also, if you feel that way, then please stop working on this entirely
for now. You're not bringing any good to anyone, yourself first, by
pushing yourself when you don't have the will to reach a proper
solution. You'll only end up with a solution having the same kind of
problems you're trying to fix with required use.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Wed, 14 Jun 2017 15:57:38 +0200
Michał Górny  wrote:
[...]
> > [...]  
> > > > > > > [1]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse  
> > > > > > 
> > > > > > I really don't like the reordering thing. Even the
> > > > > > restricted syntax does not fix the issue with '^^ ( a b )
> > > > > > b? ( a )' already mentioned here. It'd be much better and
> > > > > > simpler for the spec just to assign a fixed value and use
> > > > > > the solving rules with those.  
> > > > > 
> > > > > You're not going to convince me by providing examples that are
> > > > > utterly broken by design and meaningless ;-).
> > > > 
> > > > Well... if it's so obvious that the example is broken by design
> > > > that you don't even bother to explain why, I assume you have an
> > > > algorithm for that. Where is the code ? What are the numbers ?
> > > > How many ebuilds might fail after reordering ? How can this be
> > > > improved ?
> > > 
> > > Are you arguing for the sake of arguing here? I just presumed that
> > > this example is so obviously broken there is no point wasting any
> > > more time on it. The code of nsolve clearly detects that, so I
> > > don't really understand what you're trying to prove here.  
> > 
> > Those are real questions. You should take breath, think a bit about
> > it, and try to run the 2 possible orderings of the ^^ through
> > nsolve or even solve.py. They both are very happy (and are right to
> > be) with the above ordering. You might want to think a bit more
> > about what is the relation between this broken 10 chars example and
> > the 10 lines python targets one below.
> > 
> > You should also realize that all the above questions have already
> > been answered in length if you do as I suggest.  
> 
> No. I have already spent too much time on this. We're already long
> past all useful use cases, and now I feel like you're going to argue
> to death just to find a perfect algorithm that supports every absurd
> construct anyone can even write, if only to figure out the construct
> is completely useless.

I'm not going to argue to death. It's already proven reordering is
broken.

> If you want to play with it more, then please by all means do so.

There is nothing to do for reordering. It's broken by design.

> However, do not expect me to waste any more of my time on it. I've
> done my part, the code works for all reasonable use cases and solves
> all the problems I needed solving. If you want more, then it's your
> job to do it and solve the resulting issues.

Like... writing code handling all the cases and describing how it
works ? We're past that. The only thing we're not past is that you fail
to understand it and attempt to block it.


Alexis.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Wed, 14 Jun 2017 14:24:48 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On śro, 2017-06-14 at 11:06 +0200, Alexis Ballier wrote:
> > On Wed, 14 Jun 2017 00:13:42 +0200
> > Michał Górny <mgo...@gentoo.org> wrote:
> >   
> > > On wto, 2017-06-13 at 12:27 +0200, Alexis Ballier wrote:  
> > > > On Mon, 12 Jun 2017 21:17:16 +0200
> > > > Michał Górny <mgo...@gentoo.org> wrote:
> > > > 
> > > > > I've actually started typing the initial specification
> > > > > yesterday [1]. As you can see, banning the extra constraints
> > > > > has made the algorithms much simpler. In particular:
> > > > > 
> > > > > 1. You do not have to define 'falsify' for anything other than
> > > > > pure flags -- which makes it easy to inline it.
> > > > > 
> > > > > 2. ||, ??, ^^ groups are only flat lists of flags -- which
> > > > > makes reordering and processing them trivial.
> > > > > 
> > > > > 3. The algorithm is recursive only on USE-conditional groups.
> > > > > This makes it trivial to make it iterative. Optimizations
> > > > > become trivially possible.
> > > > 
> > > > 
> > > > While you're right in one sense, you're mixing two different
> > > > things here. What you wrote *is* recursive. It does not recurse
> > > > just because you're assuming a restricted syntax. You're only
> > > > saving two things: you don't need to define how to enforce to
> > > > false (that is 3 lines not 3 pages :=) ) and you're avoiding
> > > > the nested use conditionals that are already ill defined per
> > > > the current spec (foo? bar is equivalent to && ( foo bar ) when
> > > > nested) which I believe is already another problem.
> > > > 
> > > > Then, remember how I wanted to be much more drastic than you in
> > > > the beginning by killing all ||,&&,^^ etc. and keep only use
> > > > conditionals in REQUIRED_USE ?  Well, that's where the
> > > > complexity comes. The whole deal then is to define rewriting
> > > > rules for the AST so that the algorithm you describe executes
> > > > the exact same instructions but the new AST only has use
> > > > conditionals. This is more like writing a compiler for the
> > > > spec, so this does not belong to the spec and there is no issue
> > > > here.
> > > 
> > > I'm looking for a compromise here. Killing those groups
> > > completely is going to make things harder for users. Keeping them
> > > with functionality limited to what's used in ~99.9% ebuilds
> > > (based on your numbers) is IMO a better choice.  
> > 
> > I already said I see the limited syntax as a good thing because it
> > forces devs to write constraints that have a natural interpretation
> > in how it is solved. However, you can't limit the syntax without a
> > new EAPI, and more importantly, properly solving does not even
> > require limiting the syntax.  
> 
> Actually, you can, via a Gentoo policy. Since solving is not required
> by the PMS, there is no rule saying it has to work for every
> constraint allowed by the PMS.

Indeed. But you're trying to make rules that it has *not* to work for
some of them for reasons that look more like laziness in trying to
understand the problem than anything else.

> Much like, you can't force a particular ordering or forbid circular
> constraints without a new EAPI. Yet you do it because it gives
> a practical improvement.

I don't see this being enforced by a new EAPI. It's more a matter of QA
tools like repoman. The main difference is that there are real reasons
for forbidding circular constraints: For true circular ones, no solver
will ever be able to solve it...

[...]
> >   
> > > > [BTW: checking the rewrite rules behave properly is what I
> > > > meant by rebasing solve() on top of it and being happy with
> > > > it]
> > > 
> > > Could you reiterate the current solving rules (trueify/falsify)?
> > > Are they equal to the ones you listed last, or does the current
> > > implementation change anything?  
> > 
> > Let's recap a bit. nsolve() is poorly named and does not solve
> > anything. It translates to implications and checks whether the
> > implications solver will always provide a valid result in one pass.
> > So, if you only care about solving rules, read your spec man. For
> > the more general case it should behave like those tr

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Thu, 15 Jun 2017 17:32:40 +0100
Ciaran McCreesh <ciaran.mccre...@googlemail.com> wrote:

> On Thu, 15 Jun 2017 18:30:10 +0200
> Alexis Ballier <aball...@gentoo.org> wrote:
> > On Thu, 15 Jun 2017 17:22:26 +0100
> > Ciaran McCreesh <ciaran.mccre...@googlemail.com> wrote:  
> > > On Thu, 15 Jun 2017 18:19:04 +0200
> > > Alexis Ballier <aball...@gentoo.org> wrote:
> > > > On Thu, 15 Jun 2017 17:13:57 +0100
> > > > Ciaran McCreesh <ciaran.mccre...@googlemail.com> wrote:  
> > > > > On Thu, 15 Jun 2017 18:07:00 +0200
> > > > > Alexis Ballier <aball...@gentoo.org> wrote:
> > > > > > > The best way to convince me is through valid
> > > > > > > examples.
> > > > > > 
> > > > > > It is also easier to be convinced when you try to understand
> > > > > > and ask for clarifications instead of just rejecting without
> > > > > > thinking :)  
> > > > > 
> > > > > The problem with this entire proposal is that it's still in
> > > > > "well I can't think of how it could possibly go wrong"
> > > > > territory. We need a formal proof that it's sound. History has
> > > > > shown that if something can be abused by Gentoo developers, it
> > > > > will be abused...
> > > > 
> > > > Had you read the thread you would have noticed that I provided
> > > > an algorithm giving sufficient conditions for the solver to
> > > > work. That is, if developers pay attention to repoman
> > > > warnings/errors, it will never fail. Obviously, since we're
> > > > still in the SAT space, you can ignore the errors and make it
> > > > fail, but it'll never be worse than what we currently
> > > > have.  
> > > 
> > > You have shown that you produce a solution, not the solution
> > > that's actually wanted.  
> > 
> > Since 'wanted' is still undefined, I'd say it produces the defined
> > solution and you can adapt to the definition to get what you want.  
> 
> So you're saying that at the end of this, there's an ENFORCED_USE
> solver that spits out some answer that may or may not be in any way a
> sane solution to the conflict.
> 
> I don't see how that's helpful to a user.
> 

Define sane.
The definition of the solver is made to change the least possible of
the inputs and is completely and easily predictable by the person
writing the constraint. That is something I would call sane.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Thu, 15 Jun 2017 17:45:09 +0100
Ciaran McCreesh <ciaran.mccre...@googlemail.com> wrote:

> On Thu, 15 Jun 2017 18:37:16 +0200
> Alexis Ballier <aball...@gentoo.org> wrote:
> > > So you're saying that at the end of this, there's an ENFORCED_USE
> > > solver that spits out some answer that may or may not be in any
> > > way a sane solution to the conflict.
> > > 
> > > I don't see how that's helpful to a user.  
> > 
> > Define sane.
> > The definition of the solver is made to change the least possible of
> > the inputs and is completely and easily predictable by the person
> > writing the constraint. That is something I would call sane.  
> 
> The problem is not just writing a resolver that spits out a valid
> output. The problem is writing a resolver which will never go and
> uninstall bash as a result of unintended combinations of inputs (which
> Portage used to do, but there's now a special exception for system
> packages, so it will only occasionally unexpectedly uninstall critical
> packages that aren't explicitly in system due to virtuals instead).
> This is *hard*.

We're not talking about solving deps here.

> A bad suggestion to the user is worse than no suggestion at all.
> Unless you can safely determine that there aren't any unintended
> consequences of your rule, the focus needs to be on producing good
> error messages so the user can figure the problem out, not on
> producing bad solutions that will confuse things even more.

The guarantee comes from the fact that the output is always in the
space of all possible inputs from the user. So, if some output will
kill a kitten, so does some input.

Of course, implementation can decide to error out and propose a
solution, to continue but print big fat warnings, etc. I like the
initial proposal in that regard.


Alexis.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Wed, 14 Jun 2017 00:13:42 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> On wto, 2017-06-13 at 12:27 +0200, Alexis Ballier wrote:
> > On Mon, 12 Jun 2017 21:17:16 +0200
> > Michał Górny <mgo...@gentoo.org> wrote:
> >   
> > > I've actually started typing the initial specification yesterday
> > > [1]. As you can see, banning the extra constraints has made the
> > > algorithms much simpler. In particular:
> > > 
> > > 1. You do not have to define 'falsify' for anything other than
> > > pure flags -- which makes it easy to inline it.
> > > 
> > > 2. ||, ??, ^^ groups are only flat lists of flags -- which makes
> > > reordering and processing them trivial.
> > > 
> > > 3. The algorithm is recursive only on USE-conditional groups. This
> > > makes it trivial to make it iterative. Optimizations become
> > > trivially possible.  
> > 
> > 
> > While you're right in one sense, you're mixing two different things
> > here. What you wrote *is* recursive. It does not recurse just
> > because you're assuming a restricted syntax. You're only saving two
> > things: you don't need to define how to enforce to false (that is 3
> > lines not 3 pages :=) ) and you're avoiding the nested use
> > conditionals that are already ill defined per the current spec
> > (foo? bar is equivalent to && ( foo bar ) when nested) which I
> > believe is already another problem.
> > 
> > Then, remember how I wanted to be much more drastic than you in the
> > beginning by killing all ||,&&,^^ etc. and keep only use
> > conditionals in REQUIRED_USE ?  Well, that's where the complexity
> > comes. The whole deal then is to define rewriting rules for the AST
> > so that the algorithm you describe executes the exact same
> > instructions but the new AST only has use conditionals. This is
> > more like writing a compiler for the spec, so this does not belong
> > to the spec and there is no issue here.  
> 
> I'm looking for a compromise here. Killing those groups completely is
> going to make things harder for users. Keeping them with functionality
> limited to what's used in ~99.9% ebuilds (based on your numbers) is
> IMO a better choice.

I already said I see the limited syntax as a good thing because it
forces devs to write constraints that have a natural interpretation in
how it is solved. However, you can't limit the syntax without a new
EAPI, and more importantly, properly solving does not even require
limiting the syntax.

BTW, I don't know how you get that info from my data because I never
voluntarily checked for a restricted syntax :)

> > [BTW: checking the rewrite rules behave properly is what I meant by
> > rebasing solve() on top of it and being happy with it]  
> 
> Could you reiterate the current solving rules (trueify/falsify)? Are
> they equal to the ones you listed last, or does the current
> implementation change anything?

Let's recap a bit. nsolve() is poorly named and does not solve
anything. It translates to implications and checks whether the
implications solver will always provide a valid result in one pass.
So, if you only care about solving rules, read your spec man. For the
more general case it should behave like those trueify/falsify with
the change that nested implications are interpreted as && (so no
more !(a -> b) crap to worry about).

If you take solve() as an implementation of your spec, you have:
solve(x) <=> solve(to_impl.convert_to_implications(x)) when solve(x)
is defined; with the added benefit that
'solve(to_impl.convert_to_implications(x))' is defined and should
provide proper results on the whole REQUIRED_USE syntax as currently
defined (granted that nsolve(x) does not report anything wrong).

> > > Nevertheless, feel free to play with the full implementation. If
> > > you're interested, you can play with the complex cases more. In
> > > particular, I'm wondering whether nsolve will actually consider
> > > most of them solvable.
> > > 
> > > As for the results, I think it is the point where we start
> > > preparing pull requests with REQUIRED_USE changes to see whether
> > > the developers agree with such changes.  
> > 
> > If by that you also include code cleanup and writing tests then
> > yes :)  
> 
> I'm not sure if we're talking about the same thing. I'm talking about
> filing pull requests against ebuilds whose REQUIRED_USE is rejected by
> nsolve. I think it'd serve as a reasonable field test for whether
> developers agree with the imposed restrictions.

I was talking about PRs against portage & repoman.

> > > [1]:https://wiki.gentoo.or

Re: [gentoo-dev] autotools.eclass: automatically move configure.in to configure.ac

[Alexis Ballier]

On Sun, 11 Jun 2017 17:20:49 +0200
Kristian Fiskerstrand  wrote:

> On 06/11/2017 05:17 PM, Mart Raudsepp wrote:
> >> We can always patch the eclass at that point if that is still a big
> >> concern, but I fundamentally agree with William on this, starting
> >> point
> >> should be fixing it upstream, so can start with a tracking bug on
> >> affected packages.  
> > That's a complete useless waste of time, to track some ancient
> > packages that don't get any upstream update anyway. The active ones
> > have updated it long ago. And it'd be a joke to propose last riting
> > for the reason of a file being named configure.in instead of
> > configure.ac.
> > 
> >   
> 
> That determination can be made on a package-by-package basis and fixed
> in ebuild if needed.
> 

Funny thing is that packages still using autoconf 2.1* don't get
any warning and packages setting WANT_AUTOCONF to some older version
will never break...

Re: [gentoo-dev] crossdev: installing _host_ build dependencies not automatic?

[Alexis Ballier]

On Wed, 3 May 2017 22:47:18 +0100
James Le Cuirot <ch...@gentoo.org> wrote:

> On Wed, 3 May 2017 17:56:43 +0200
> Alexis Ballier <aball...@gentoo.org> wrote:
> 
> > On Wed, 3 May 2017 12:05:48 +0200
> > "Paweł Hajdan, Jr." <phajdan...@gentoo.org> wrote:
> >   
> > > I encountered <https://bugs.gentoo.org/show_bug.cgi?id=617276>
> > > while working on some cross-compiling project.
> > > 
> > > Admittedly, it may not be that easy to handle host package
> > > dependencies fully automatically.
> > > 
> > > I'm wondering - is it documented what portage guarantees, and
> > > what I'm expected to just manually handle to provide host build
> > > dependencies?
> > > 
> > > Any other advice about properly using crossdev would also be
> > > appreciated. I'd be happy to test and help improve things.
> > 
> > 
> > From man emerge:
> > 
> > 
> >--root-deps[=rdeps]
> >   If  no argument is given then build-time dependencies
> > of packages for ROOT are installed to ROOT instead of /.  If the
> >rdeps argument is given then discard all build-time
> > dependencies of packages for ROOT.  This option is only meaningful
> > when used together with ROOT and it should not be enabled under
> > normal circumstances!
> > 
> >   Does not affect EAPIs that support HDEPEND.
> > Experimental EAPI 5-hdepend provides HDEPEND as a new means to
> > adjust installation into "/" and ROOT.  If ebuilds using EAPIs
> >   which do not support HDEPEND are built  in  the same
> >   emerge run as those using EAPIs which do support
> > HDEPEND, this option affects only the former.
> > 
> > 
> > crossdev wrappers set --root-deps=rdeps (read cross-emerge, this
> > can be overriden), but be careful: If you only care about getting
> > all the deps and maybe more then removing --root-deps might help
> > you. However, when cross compiling you will likely run into broken
> > deps since / and ROOT will not use the same keyword visibility.  
> 
> I was going to point to crossdev's use of --root-deps=rdeps too. I did
> wonder why on earth this was even added. I overrode it for quite a
> while and didn't have any issue. History showed that it was added by
> solar without much of an explanation. He's no longer around to ask. It
> wasn't until I tried to build a brand new ppc64le system recently that
> I finally found a reason for it, though I'm not sure it was the
> original reason. The multilib ABI USE flags start conflicting horribly
> in cross situations and this option seems to be the only way around it
> at present.

there's that and e.g. you might want to enable USE=mmal for cross
compiling to a raspberry pi; raspberrypi-userland will be in DEPEND but
you wont be able to merge it into /

> I doubt keyword visibility is an issue. Portage uses a different
> configuration between / and ROOT when cross-compiling. I don't think
> it tries to force the same package versions beyond what is specified
> in the ebuild. For pure build-time dependencies, the package will
> only be installed to / anyway (i.e. you don't need cmake in ROOT) so
> there is nothing to enforce here.


The issue is that we can't distinguish between target build time
dependencies (libraries, headers, etc.) and host build time
dependencies (cmake, autotools, etc.). While --root-deps=rdeps avoids
most of the problems at the cost of missing host deps, it still fails
horribly for target build time only deps (e.g. pure headers libraries).

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Tue, 30 May 2017 20:11:38 +0200
Michał Górny  wrote:
[...]
> > > Of course, we could just validate all the possible cases via
> > > repoman, and reject the ebuild if there's at least one conflict
> > > between them. Not sure how to express that properly in the spec
> > > though. Not sure how it would work practically.  
> > 
> > Adding a 2^n check to repoman isn't gonna work well.
> >   
> 
> I'm not saying it's the most optimal algorithm of verifying
> the correctness of the constraints. It's just the one that's quite
> obvious -- relatively simple and reliable. If someone can come up with
> something better that covers at least the most common cases, I'm all
> for it.
> 
> That said, this wouldn't be that much of a problem if we can keep the
> n low. For a start, we can rule out all flags that don't appear
> in REQUIRED_USE at all. Furthermore, I think we could also ignore
> the constraints for flags that don't appear there at least 'twice',
> and so on.

:)

You're applying classical techniques to lower the size of a SAT
instance so that your exponential algorithm does not explode, but it's
still hard.

I'm not sure what you want: If you want to detect that there is an
impossible constraint, well, ebuild writer will notice soon enough when
testing it. If you want to detect that there is a way to have a
conflict between useflags, then there will be valid cases where this
will happen.

That said, assuming we have REQUIRED_USE in CNF form, its negation is
in DNF form. Solving SAT on DNF formulas is easy (linear I think), and
this would give you an assignment of useflags triggering an impossible
constraint. e.g. 'foo? ( bar )' with USE='foo -bar' in make.conf.
This could be used to trigger a repoman warning but basically every
single ebuild would trigger those.

Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Mon, 05 Jun 2017 20:10:12 +0200
Michał Górny  wrote:
[...]
> > > Stand-alone makes little sense (and little trouble) but as you
> > > could have seen it's used nested in other thingies:
> > > 
> > > 1. || ( ( a b ) ( c d ) e )
> > > 
> > > 2. ?? ( ( a b ) ( c d ) e )
> > > 
> > > 3. ^^ ( ( a b ) ( c d ) e )  
> > 
> > Yeah that's the nesting problem causing a parse error.
> > Those should be expanded to implications. What I'm relying onto is
> > all clauses to be of the form '[list of conditions]? [list of
> > constraints]'  
> 
> I've noticed that you turned the implications into multi-conditions,
> breaking all my scripts ;-). Is the [list of conditions] conjunctive
> or disjunctive?

conjunctive as in foo? ( bar? ( baz ) ) -> [foo,bar]?[baz]


[...]
> > > The question is whether we want to:
> > > 
> > > a. actually try to solve this nesting insanity,
> > > 
> > > b. declare it unsupported and throw REQUIRED_USE mismatch on user,
> > > 
> > > c. ban it altogether.  
> > 
> > 
> > I don't think it is *that* insane to support nesting :)
> >   
> 
> || ( ^^ ( ?? ( a b ) c ( d e ) ) f )

If you really need that then you'd need to expand it manually. It seems
better to have it expanded internally automatically.
Remember you were the one wanting to keep || & co because they're
simpler to read and write ;)

Alexis.

Re: [gentoo-dev] glibc-2.26 and changes with SunRPC, libtirpc, ntirpc, libnsl (NIS and friends), ...

[Alexis Ballier]

On Mon, 18 Sep 2017 11:56:30 +0200
"Andreas K. Huettel"  wrote:

> Porting a package means adding a dependency in the style of 
> || ( 

Re: [gentoo-dev] Reviving the Sandbox project

[Alexis Ballier]

On Fri, 22 Sep 2017 06:07:18 +0200
Michał Górny  wrote:

> W dniu czw, 21.09.2017 o godzinie 15∶41 -0700, użytkownik Matt Turner
> napisał:
> > On Thu, Sep 21, 2017 at 2:25 PM, Michał Górny 
> > wrote:  
> > > Given that sandbox is utterly broken by design, I don't really
> > > want to put too much effort in trying to make it a little better.
> > > I'd rather put the minimal effort required to make it
> > > not-much-worse.  
> > 
> > You said in your initial email that you weren't an expert in its
> > internals, but here you say it's broken by design. Why do you think
> > that?
> >   
> 
> Because it uses LD_PRELOAD which is a huge hack and which causes
> guaranteed issues we can't really fix. All we can do is disable it for
> emacs, for compiler-rt and I'm afraid this list will grow because
> overriding random library functions is never a good idea.
> 

I think we're all ears for a better solution. There are probably much
better ways to do sandboxing these days than 15 years ago.

LD_PRELOAD does not work with static binaries. Hence the non
portable ptrace stuff. Hence bugs. Etc. The point is, that's the
best we have now.


Alexis.

Re: [gentoo-dev] Reviving the Sandbox project

[Alexis Ballier]

On Fri, 22 Sep 2017 12:38:54 +0100
Sergei Trofimovich <sly...@gentoo.org> wrote:

> On Fri, 22 Sep 2017 12:57:21 +0200
> Alexis Ballier <aball...@gentoo.org> wrote:
> 
> > On Fri, 22 Sep 2017 06:07:18 +0200
> > Michał Górny <mgo...@gentoo.org> wrote:
> >   
> > > W dniu czw, 21.09.2017 o godzinie 15∶41 -0700, użytkownik Matt
> > > Turner napisał:
> > > > On Thu, Sep 21, 2017 at 2:25 PM, Michał Górny
> > > > <mgo...@gentoo.org> wrote:  
> > > > > Given that sandbox is utterly broken by design, I don't really
> > > > > want to put too much effort in trying to make it a little
> > > > > better. I'd rather put the minimal effort required to make it
> > > > > not-much-worse.  
> > > > 
> > > > You said in your initial email that you weren't an expert in its
> > > > internals, but here you say it's broken by design. Why do you
> > > > think that?
> > > >   
> > > 
> > > Because it uses LD_PRELOAD which is a huge hack and which causes
> > > guaranteed issues we can't really fix. All we can do is disable
> > > it for emacs, for compiler-rt and I'm afraid this list will grow
> > > because overriding random library functions is never a good idea.
> > > 
> > 
> > I think we're all ears for a better solution. There are probably
> > much better ways to do sandboxing these days than 15 years ago.
> > 
> > LD_PRELOAD does not work with static binaries. Hence the non
> > portable ptrace stuff. Hence bugs. Etc. The point is, that's the
> > best we have now.  
> 
> Some other distros try harder to isolate build environment either
> through chroot and/or private mount/user/network namespace that
> contains only explicitly specified files in build environment.
> 
> That would require more cooperation from package manager to fetch
> list of all visible depends.
> 
> Don't know if drop-in relacement could be implemented for sandbox
> that way. I like clear sandbox error reporting.


We definitely do need a kind of drop-in replacement since PMS
mandates some parts of the sandbox API (addwrite/addpredict & co for
instance)

Re: [gentoo-dev] Reviving the Sandbox project

[Alexis Ballier]

On Fri, 22 Sep 2017 17:20:23 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> W dniu pią, 22.09.2017 o godzinie 12∶57 +0200, użytkownik Alexis
> Ballier napisał:
> > On Fri, 22 Sep 2017 06:07:18 +0200
> > Michał Górny <mgo...@gentoo.org> wrote:
> >   
> > > W dniu czw, 21.09.2017 o godzinie 15∶41 -0700, użytkownik Matt
> > > Turner napisał:  
> > > > On Thu, Sep 21, 2017 at 2:25 PM, Michał Górny
> > > > <mgo...@gentoo.org> wrote:
> > > > > Given that sandbox is utterly broken by design, I don't really
> > > > > want to put too much effort in trying to make it a little
> > > > > better. I'd rather put the minimal effort required to make it
> > > > > not-much-worse.
> > > > 
> > > > You said in your initial email that you weren't an expert in its
> > > > internals, but here you say it's broken by design. Why do you
> > > > think that?
> > > > 
> > > 
> > > Because it uses LD_PRELOAD which is a huge hack and which causes
> > > guaranteed issues we can't really fix. All we can do is disable
> > > it for emacs, for compiler-rt and I'm afraid this list will grow
> > > because overriding random library functions is never a good idea.
> > >   
> > 
> > I think we're all ears for a better solution. There are probably
> > much better ways to do sandboxing these days than 15 years ago.
> > 
> > LD_PRELOAD does not work with static binaries. Hence the non
> > portable ptrace stuff. Hence bugs. Etc. The point is, that's the
> > best we have now.
> >   
> 
> I know of two obvious alternatives: ptrace and filesystem layer (e.g.
> FUSE).
> 
> For the former, there's sydbox. I'm going to look into integrating it
> into Portage when I have more time.

From: https://github.com/alip/pinktrace/blob/master/configure.ac
case "$host_cpu" in
i[[3456]]86|pentium)
x86?64*|amd64)
ia64)
powerpc64*)
powerpc*)
arm*)
 [add support for those arches]
*)
AC_MSG_RESULT([NO!])
AC_MSG_ERROR([Architecture $host_cpu is not supported by
pinktrace]) ;;

sandbox keywords:
2.11-r5:0: ~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc
 ~ppc64 ~s390 ~sh ~sparc ~sparc-fbsd ~x86 ~x86-fbsd


Good luck adding the missing bits!


> For the latter, I have writing one in TODO. But I'm not sure when I'll
> have enough time to do work on it.

Not sure how that would work, but you'll likely need some kind of
chroot/container since you don't want to trust a random binary ran as
root to respect environment variables.

Alexis.

Re: [gentoo-dev] Reviving the Sandbox project

[Alexis Ballier]

On Fri, 22 Sep 2017 19:39:16 +0200
Michał Górny <mgo...@gentoo.org> wrote:

> W dniu pią, 22.09.2017 o godzinie 19∶15 +0200, użytkownik Alexis
> Ballier napisał:
> > On Fri, 22 Sep 2017 17:20:23 +0200
> > Michał Górny <mgo...@gentoo.org> wrote:
> >   
> > > W dniu pią, 22.09.2017 o godzinie 12∶57 +0200, użytkownik Alexis
> > > Ballier napisał:  
> > > > On Fri, 22 Sep 2017 06:07:18 +0200
> > > > Michał Górny <mgo...@gentoo.org> wrote:
> > > > 
> > > > > W dniu czw, 21.09.2017 o godzinie 15∶41 -0700, użytkownik Matt
> > > > > Turner napisał:
> > > > > > On Thu, Sep 21, 2017 at 2:25 PM, Michał Górny
> > > > > > <mgo...@gentoo.org> wrote:  
> > > > > > > Given that sandbox is utterly broken by design, I don't
> > > > > > > really want to put too much effort in trying to make it a
> > > > > > > little better. I'd rather put the minimal effort required
> > > > > > > to make it not-much-worse.  
> > > > > > 
> > > > > > You said in your initial email that you weren't an expert
> > > > > > in its internals, but here you say it's broken by design.
> > > > > > Why do you think that?
> > > > > >   
> > > > > 
> > > > > Because it uses LD_PRELOAD which is a huge hack and which
> > > > > causes guaranteed issues we can't really fix. All we can do
> > > > > is disable it for emacs, for compiler-rt and I'm afraid this
> > > > > list will grow because overriding random library functions is
> > > > > never a good idea. 
> > > > 
> > > > I think we're all ears for a better solution. There are probably
> > > > much better ways to do sandboxing these days than 15 years ago.
> > > > 
> > > > LD_PRELOAD does not work with static binaries. Hence the non
> > > > portable ptrace stuff. Hence bugs. Etc. The point is, that's the
> > > > best we have now.
> > > > 
> > > 
> > > I know of two obvious alternatives: ptrace and filesystem layer
> > > (e.g. FUSE).
> > > 
> > > For the former, there's sydbox. I'm going to look into
> > > integrating it into Portage when I have more time.  
> > 
> > From: https://github.com/alip/pinktrace/blob/master/configure.ac
> > case "$host_cpu" in
> > i[[3456]]86|pentium)
> > x86?64*|amd64)
> > ia64)
> > powerpc64*)
> > powerpc*)
> > arm*)
> >  [add support for those arches]
> > *)
> > AC_MSG_RESULT([NO!])
> > AC_MSG_ERROR([Architecture $host_cpu is not supported by
> > pinktrace]) ;;
> > 
> > sandbox keywords:
> > 2.11-r5:0: ~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc
> >  ~ppc64 ~s390 ~sh ~sparc ~sparc-fbsd ~x86 ~x86-fbsd
> > 
> > 
> > Good luck adding the missing bits!
> > 
> >   
> > > For the latter, I have writing one in TODO. But I'm not sure when
> > > I'll have enough time to do work on it.  
> > 
> > Not sure how that would work, but you'll likely need some kind of
> > chroot/container since you don't want to trust a random binary ran
> > as root to respect environment variables.
> >   
> 
> Environment variables? What for?
> 

I don't know :)
I have no idea how you would provide a virtual FS that would be the
only thing ever seen by all processes spawned.

Re: [gentoo-dev] glibc-2.26 and changes with SunRPC, libtirpc, ntirpc, libnsl (NIS and friends), ...

[Alexis Ballier]

On Mon, 18 Sep 2017 11:56:30 +0200
"Andreas K. Huettel"  wrote:

> sunrpc - build against glibc

Now that I think about it: What about other libcs ? musl, uclibc,
freebsd or even the prefix ones ?

[...]
> Porting a package means adding a dependency in the style of 
> || ( 

Re: [gentoo-dev] Of death and prerm

[Alexis Ballier]

On Tue, 29 Aug 2017 16:38:15 -0400
Michael Orlitzky  wrote:

> What should happen if an ebuild calls "die" in pkg_prerm?
> 
> The issue arose while trying to create a package that could not be
> uninstalled except as part of an upgrade. The first thing that came to
> mind was to have it die in pkg_prerm.
> 
> What portage does is *appear* to crash, but then continue along as if
> nothing happened.
> 
> Does the PMS cover this indirectly? (Is there a reliable way to make
> package removal fail?)

Is there any point in dying in any phase after (or during)
pkg_postinst ?
files are already live by then; what would this achieve ?

[gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: sys-libs/zlib/

[Alexis Ballier]

On Sat,  9 Sep 2017 21:07:02 + (UTC)
"Lars Wendler"  wrote:

> commit: b5f0471fe97de820f888702e1938989b9fd25522
> Author: David Seifert  gentoo  org>
> AuthorDate: Wed Sep  6 11:48:56 2017 +
> Commit: Lars Wendler  gentoo  org>
> CommitDate: Sat Sep  9 21:06:46 2017 +
> URL:
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=b5f0471f
> 
> sys-libs/zlib: Add sublot
[...]
> -SLOT="0"
> +SLOT="0/1" # subslot = SONAME

Not much to do in this case now, but can we please stop this insanity
in the future ?

You're not adding any subslot here, you're changing it from its default
${SLOT} (0) to 1, causing this:

 (sys-libs/zlib-1.2.11:0/1::gentoo, ebuild scheduled for merge) causes
 rebuilds for: (app-text/ghostscript-gpl-9.21:0/0::gentoo, ebuild
 scheduled for merge) (media-libs/libpng-1.6.32:0/16::gentoo, ebuild
 scheduled for merge) (dev-libs/libxml2-2.9.5:2/2::gentoo, ebuild
 scheduled for merge) (dev-db/mariadb-10.2.8:0/18::gentoo, ebuild
 scheduled for merge) (x11-libs/libpciaccess-0.13.5:0/0::gentoo, ebuild
 scheduled for merge) (dev-lang/python-2.7.13:2.7/2.7::gentoo, ebuild
 scheduled for merge) (media-libs/openjpeg-2.2.0:2/7::gentoo, ebuild
 scheduled for merge) (media-libs/tiff-4.0.8:0/0::gentoo, ebuild
 scheduled for merge) (net-misc/openssh-7.5_p1-r2:0/0::gentoo, ebuild
 scheduled for merge) (sys-devel/llvm-4.0.1:4/4::gentoo, ebuild
 scheduled for merge) (sys-auth/consolekit-1.2.0:0/0::gentoo, ebuild
 scheduled for merge) (dev-lang/python-3.4.6:3.4/3.4m::gentoo, ebuild
 scheduled for merge) (media-gfx/imagemagick-7.0.6.9:0/7.0.6.9::gentoo,
 ebuild scheduled for merge)



Thank you.


Alexis.

Re: [gentoo-dev] [QA] New policy: 'files' directory must not be larger than 32 KiB

[Alexis Ballier]

On Tue, 19 Dec 2017 16:00:16 -0500
"Aaron W. Swenson"  wrote:

> On 2017-12-17 14:21, Michał Górny wrote:
> >   Total size of 'files' subdirectory of a package should not be
> > larger than 32 KiB. If the package needs more auxiliary files, they
> > should be put into SRC_URI e.g. via tarballs.  
> 
> I don’t have any strong opinions about this either way.

note that the announcement fails to mention why this has been a
self-imposed rule on some packages for a long time: tools like quilt or
git are made to track large patchsets. Once scripting is in place, it is
much more convenient to generate patch tarballs from those tools; epatch
supporting numbered patchsets and applying them all is great for that
too.

> However, what alternative do we have to throwing the patches up in a
> devspace?

mirror://gentoo, aka /space/distfiles-local/

> Having previously done so with dev-db/postgresql, it was annoying
> having to fix the SRC_URI because I wasn’t the one who did a slot
> bump.

or even that way: you can add several URIs for the same file, e.g.:
SRC_URI="pecker/~deva/patches.tar.bz2 pecker/~devb/patches.tar.bz2"

mirrors will pick the one available and usually users wont be impacted
by a 404 delay if devb hosts the patches; but I would still recommend
distfiles-local, esp. since it autocleans when the file is not used
anymore.

Re: [gentoo-dev] [QA] New policy: 'files' directory must not be larger than 32 KiB

[Alexis Ballier]

On Wed, 20 Dec 2017 08:34:14 -0500
Rich Freeman <ri...@gentoo.org> wrote:

> On Wed, Dec 20, 2017 at 4:42 AM, Alexis Ballier <aball...@gentoo.org>
> wrote:
> > On Tue, 19 Dec 2017 16:00:16 -0500
> > "Aaron W. Swenson" <titanof...@gentoo.org> wrote:
> >  
> >> However, what alternative do we have to throwing the patches up in
> >> a devspace?  
> >
> > mirror://gentoo, aka /space/distfiles-local/
> >  
> 
> That isn't a great option.  This has been discussed previously:
> 
>https://lists.gt.net/gentoo/dev/224673?do=post_view_threaded
> 
>
> https://devmanual.gentoo.org/general-concepts/mirrors/index.html#suitable-download-hosts
> 
> 
> I would point out that a devspace isn't the only alternative -
> anything stable would be reasonable.

So... 6 years later... still no solution ? Kid me not.

I don't see what any other hosting style changes here: next time infra
checks pecker homedirs, everyone will spend time manually removing old
distfiles. For archival of sources, there has always been
gentoo/src/patchsets. For extreme cases, there's distfiles-whitelist.

Re: [gentoo-dev] [PATCH 1/2] preserve-libs.eclass: Split off preserve_old_lib from eutils.

[Alexis Ballier]

On Thu, 4 Jan 2018 10:35:23 -0500
Mike Gilbert  wrote:

> On Thu, Jan 4, 2018 at 5:23 AM, Pacho Ramos  wrote:
> > I have seen this is only used by:
> > app-arch/xz-utils
> > dev-libs/gmp
> > dev-libs/libpcre
> > dev-libs/mpc
> > dev-libs/mpfr
> > net-nds/openldap
> > sys-libs/gdbm
> > sys-libs/ncurses
> > sys-libs/readline
> > sys-process/audit
> >
> > Maybe we could deprecate it and try to drop it in the future :/
> >  
> 
> As Soap touched on earlier, this should probably not be
> deprecated/removed until a solution compatible with Paludis and
> pkgcore is implemented.
> 
> A couple of options for that:
> 
> 1. Add functionality similar to preserve-libs to these alternate
> package managers. This is unlikely to happen.

IMHO having profiles mandate a preserve-libs behavior (and thus PMS
define this) for a few select packages (the ones above) is the way to go
here: preserve-libs is evil but really the least evil in this case.
If you miss the warning from the eclass, you can very easily end up
with binaries loading the two versions of the library; preserve-libs
has the advantage that portage is very loud about this.

Re: [gentoo-dev] News Item: Portage Dynamic Deps

[Alexis Ballier]

On Sun, 21 Jan 2018 23:01:08 -0800
Zac Medico  wrote:

> Please review.
> 
> Title: Portage Dynamic Deps
> Author: Zac Medico 
> Posted: 2018-01-28
> Revision: 1
> News-Item-Format: 2.0
> Display-If-Installed:  
> Beginning with Portage 2.3.20, the previous default --dynamic-deps=y
> setting has changed to --dynamic-deps=n. Due to this change, some
> users may experience emerge dependency calculation failures triggered
> by installed packages that have outdated dependencies. In order to
> avoid problems of this nature, use the emerge --changed-deps=y option
> with your next deep @world update.

What's the rationale behind this ?

What I mean is: while '--dynamic-deps=n --changed-deps=n' is the
technically correct behavior, this just seems like throwing unbearable
dep calculation failure messages at users' faces while we could default
to '--dynamic-deps=n --changed-deps=y' and get the already
policy-mandated behavior of 'force a rebuild when you change deps'.

Re: [gentoo-dev] rfc: [QA] Ban policy introduction SLIPUP

[Alexis Ballier]

On Tue, 31 Jul 2018 14:36:37 +0100
Amy Liffey  wrote:

> Hello folks,
> 
> I apologize to everyone for sending this proposal before it was
> finished. It was not voted on by the QA team hence it was not an
> official proposal by the QA team. There was probably some
> misunderstanding in communication.
> 
> After we finish the official draft and it is accepted by QA team
> members, we will be very happy to accept comments on the mailing list
> in the future.

During that drafting process, please clarify how this changes and
differs from the current GLEP48 wording and if the glep needs to be
updated:

If a particular developer persistently causes breakage, the QA team may
request that Comrel re-evaluates that developer's commit rights.
Evidence of past breakages will be presented with this request to
Comrel.





Alexis.

Re: [gentoo-dev] autotools.eclass and EAPI 7

[Alexis Ballier]

On Thu, 16 Aug 2018 21:36:56 +0300
Mart Raudsepp  wrote:

> Ühel kenal päeval, N, 16.08.2018 kell 14:27, kirjutas Brian Evans:
> > There are currently a handful of ebuilds using EAPI 7 and the
> > autotools
> > eclass.
> > 
> > I believe that this eclass should be reviewed for adding BDEPEND or
> > having BDEPEND replace the DEPEND statement as the default action of
> > the
> > eclass.
> > 
> > Other items might be needed, but that's doubtful.
> > 
> > Someone please advise the best course of action and either do it or
> > I will propose a patch based on the discussion.
> >   
> 
> Could or did someone also check through all the other eclasses that
> don't have any global EAPI compatibility checks?
> For the future, maybe it's better to add such a check - just accepting
> 0-7 or so, but unsure about all these custom EAPIs out there, might
> force more eclass copying to some overlays.

I don't really like that kind of checks: untested after usually small
changes != broken.

IMHO a better solution could be to have council members review all
eclasses prior to approving an eapi and either adding a comment why + a
die when used with the not-yet-approved-eapi if the eclass requires
changes or do nothing about it if it's fine. This has the double
advantage to give council members first hands experience with an eapi
before approving/voting it and ensures better migration when eapi is
approved.

Re: [gentoo-dev] Re: What means bup?

[Alexis Ballier]

On Wed, 18 Jul 2018 14:20:56 +0200
Kristian Fiskerstrand  wrote:

> On 07/18/2018 02:10 PM, Alexis Ballier wrote:
> > I often find myself in the
> > need to use/invent some abbreviation in order to fit the limit.
> > Considering this is an error, this sends the message that short is
> > preferred over clear.  
> 
> Or that the summary should be concise and a longer proper description
> can be written in the body of the commit message instead. Potentially
> mixed in with multiple commits for different logical changes etc etc.
> 

Sure, but that somewhat defeats the point of a summary if it's not
possible to clearly express what it is about without relying on the
long description.

Re: [gentoo-dev] Re: What means bup?

[Alexis Ballier]

On Wed, 18 Jul 2018 10:35:41 +0200
Ulrich Mueller  wrote:

> > On Wed, 18 Jul 2018, Matthew Thode wrote:  
> 
> > On 18-07-18 09:16:07, Johannes Huber wrote:  
> >> english is not my mother language, so please clarify what bup
> >> means, just seen here:
> >> 
> >> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=ee0e0401478cf30b3ced0405f6b89391e05d2397
> >> 
> >> Just checked if it was a typo, but can't be:
> >> git log --oneline --grep bup | count -l
> >> Expected 0 lines, got 1248.  
> 
> > It's similiar to a sound you make when you touch something's nose.
> > *boop*  
> 
> > I just prefer bup instead.  I generally only use it when doing
> > simple bumps of packages (copy ebuilds with only keyword edits).  
> 
> We used to have the rule that ChangeLog messages "should be well
> explained and written in clean English". I think this applies to
> commit messages, too.


While it does not really apply in this precise case, this rule's
weight has been lowered a lot with repoman enforcing 'cat/pkg: ...' +
hard limit on the commit message length. I often find myself in the
need to use/invent some abbreviation in order to fit the limit.
Considering this is an error, this sends the message that short is
preferred over clear.

Re: [gentoo-dev] How to provide a recent TeXLive for Gentoo for our users?

[Alexis Ballier]

On Mon, 26 Mar 2018 11:57:49 +0200
Jonas Stein  wrote:
> An installation via tlmgr provides many updates per day, but
> our distributed TeXLive is unfortunately always behind.
> Typically TeXLive on gentoo is 6-12 months behind upstream, because we
> have to bump a lot manually.

That is not the real reason. Most of it has been scripted for 10+ years.
The real reason is that we need to go through ~arch testing, fixing rev
deps that might need update to their .tex files because the underlying
packages they use has changed, adapt the deps for some potential
changes, and then a stablereq round. Following the yearly release cycle
leaves time for it to happen. It could move faster, but I don't see any
need for it as this would mean shortening stabilization cycles
potentially allowing for more bugs to enter stable and increasing the
load on arch teams.


[...]
> This is not a very nice solution, but it works so far. One difficulty
> is, that there is no 1:1 relation between the texlive distribution and
> dev-texlive/* at the moment.

There is a 1:1 relation.


> How can we enable our users to run a recent TeXLive in a clean way?

/usr/local/share/texmf has been supported by texlive on Gentoo from day
one. You can use that. It's an overlay that takes precedence on
anything else, so per the above, you lose all the QA & testing done
behind the scenes if you use this.


Alexis.

Re: [gentoo-dev] Proper dependencies for TeX related ebuilds

[Alexis Ballier]

On Mon, 26 Mar 2018 11:40:12 +0200
Jonas Stein  wrote:

> Hi,
> 
> I just read
> https://bugs.gentoo.org/625908#c5
> and saw that we do not have a good documentation on this and  many
> packages in the tree ship with strange tex dependencies.
> 
> I started a documentation on our wiki
> https://wiki.gentoo.org/wiki/Notes_on_TeX_related_ebuilds

Well, your example (inkscape) is wrong: dep on tl-core is *never*
sufficient and certainly does not provide a working latex command. First
step to get it right is to use the already existing virtuals
(virtual/{la,}tex-base).

Re: [gentoo-dev] How to provide a recent TeXLive for Gentoo for our users?

[Alexis Ballier]

On Mon, 26 Mar 2018 20:07:07 +0200
Jonas Stein  wrote:

> > [...]  
> >> This is not a very nice solution, but it works so far. One
> >> difficulty is, that there is no 1:1 relation between the texlive
> >> distribution and dev-texlive/* at the moment.  
> > There is a 1:1 relation.  
> 
> A full installation of TeXLive installs packages, which are in
> dev-tex/* and dev-texlive/* on gentoo.
> On the other hand single packages are bundled some times.
> Some programs/packages can be found in dev-tex/* and dev-texlive/*
> 
> I remember, that I last we had for example dev-tex/notoccite in the
> tree and dev-texlive/texlive-latexextra including shipped the same,
> but newer files.
> This is what I meant with "no 1:1 relation"

The idea with dev-tex/* packages is that they can be split out of
texlive and installed/updated independently as long as they are
actively maintained. While you might have a need for specific updates
on some packages, I seriously doubt you need (and can find the manpower
to maintain) a live ebuild for the whole CTAN and can live with yearly
texlive updates.

The notoccite example is what happens over the years: package is left
unmaintained, texlive catches up and the package becomes useless.
Worse: if done properly it will overlay the more recent version from
texlive! Those packages should be reported and removed (or updated).


> >> How can we enable our users to run a recent TeXLive in a clean
> >> way?  
> > /usr/local/share/texmf has been supported by texlive on Gentoo from
> > day one. You can use that. It's an overlay that takes precedence on
> > anything else, so per the above, you lose all the QA & testing done
> > behind the scenes if you use this.  
> 
> And packages which depend on a specific LaTeX package will not
> install. So the user has to provide a package.provided list, which is
> not so nice to maintain for so many packages.

They will install, only they will pull older unused files from
texlive. As said above, the proper way to avoid this is to step up as
maintainer of some dev-tex/* packages, fix the deps to add a || or
(probably after convincing me it's worth it because the package is
updated very frequently) remove it entirely from texlive and switch the
deps.


Alexis.

Re: [gentoo-dev] Unmasking >=media-video/ffmpeg-4.0

[Alexis Ballier]

On Tue, 6 Nov 2018 11:09:17 -0500
Rich Freeman  wrote:

> On Tue, Nov 6, 2018 at 10:57 AM Alexis Ballier 
> wrote:
> >
> > On Tue, 06 Nov 2018 17:08:22 +0200
> > Mart Raudsepp  wrote:
> >  
> > > It is not GStreamer fault that ffmpeg breaks API and ABI without
> > > parallel installability, much less so the distro maintainers of
> > > it. If you/upstream don't make it parallel installable, then this
> > > is what you get.  
> >
> > Are you, seriously, suggesting this is the solution to all problems
> > here ?
> >  
> 
> It isn't the only solution, but it is one sane upgrade path.  You
> can't expect everybody to update their software overnight when the API
> changes.  That means you have to support the old API for a while when
> you introduce a new one, otherwise you end up with some software that
> doesn't work with the old version, and some software that doesn't work
> with the new version.


These days, only symbols/constants that have been deprecated (and
marked as such) for a couple of releases are removed. This means people
see warnings for more than one year before seeing them gone for good.
The problem here is not "overnight changes" but rather consumers not
paying attention to those warnings, or worse, nobody ever seeing those
because it's unmaintained.


[...]

Alexis.

Re: [gentoo-dev] Unmasking >=media-video/ffmpeg-4.0

[Alexis Ballier]

On Mon, 05 Nov 2018 20:38:58 -0500
Craig Andrews  wrote:

> I think it's time to remove the mask on >=media-video/ffmpeg-4.0
> 
> ffmpeg 4 is in many distros (including Debian, Arch) and FreeBSD.
> Upstreams are starting to require it, too. For example, Kodi 18 and 
> media-plugins/gst-plugins-libav-16 will require it.
> 
> The blocker issue https://bugs.gentoo.org/653678 has only 5 blockers 
> against it right now:
> * 654628 has a pull request out to fix it: 
> https://github.com/gentoo/gentoo/pull/10341
> * 655170 is for a package that (IMHO) should be last-rited: 
> media-libs/mediastreamer
> * 666166 is for a package that (IMHO) should be last-rited: 
> media-plugins/vdr-image
> * 655442 and 658128 are for media-video/mplayer and fixed upstream, 
> requiring a new snapshot release
> 
> What do we say? Can we set a date when the hard mask will be removed?


I think 6 months qualifies for ETIMEOUT, so go ahead and unmask it :)

merge the github PR; I'll take care of mplayer

Re: [gentoo-dev] Unmasking >=media-video/ffmpeg-4.0

[Alexis Ballier]

On Wed, 7 Nov 2018 09:57:36 -0500
Ian Stakenvicius  wrote:

> On 2018-11-06 11:21 a.m., Alexis Ballier wrote:
> > On Tue, 6 Nov 2018 11:09:17 -0500
> > Rich Freeman  wrote:
> >   
> >> On Tue, Nov 6, 2018 at 10:57 AM Alexis Ballier
> >>  wrote:  
> >>>
> >>> On Tue, 06 Nov 2018 17:08:22 +0200
> >>> Mart Raudsepp  wrote:
> >>>
> >>>> It is not GStreamer fault that ffmpeg breaks API and ABI without
> >>>> parallel installability, much less so the distro maintainers of
> >>>> it. If you/upstream don't make it parallel installable, then this
> >>>> is what you get.
> >>>
> >>> Are you, seriously, suggesting this is the solution to all
> >>> problems here ?
> >>>
> >>
> >> It isn't the only solution, but it is one sane upgrade path.  You
> >> can't expect everybody to update their software overnight when the
> >> API changes.  That means you have to support the old API for a
> >> while when you introduce a new one, otherwise you end up with some
> >> software that doesn't work with the old version, and some software
> >> that doesn't work with the new version.  
> > 
> > 
> > These days, only symbols/constants that have been deprecated (and
> > marked as such) for a couple of releases are removed. This means
> > people see warnings for more than one year before seeing them gone
> > for good. The problem here is not "overnight changes" but rather
> > consumers not paying attention to those warnings, or worse, nobody
> > ever seeing those because it's unmaintained.
> >   
> 
> But we aren't upstream most of the time, and if upstreams are pegging
> their ffmpeg to a single version they don't bother to try the newer
> one to find out the errors.  Take Kodi, v17.x is pegged to no newer
> than ffmpeg-3.3.x as I recall, and has been blocking even v3.4's
> installation for the year'ish it's been in the gentoo repo.
> 
> So this "people see warnings" thing, it really doesn't apply, unless
> you (A) have the desire and resources to build and maintain a patch
> for upstream, and (B) have an upstream with the desire and resources
> to support more than the one version of ffmpeg for a given release
> set.  Both, IMO, are in very short supply.
> 


Believe me, it's far easier to do this than maintaining several slotted
versions. See the amount of work done for e.g. gtk, qt, wxwidgets, etc.
and I'm not even counting backporting security fixes nor patching tens
of packages to "use the proper slot".

Note that the (B) desire is usually mostly killed by people claiming
multiple versions should be supported without any idea of the
implications this has. Fortunately, there's only very few upstreams
left that consider ffmpeg so special that they want to bundle or force
an old buggy and vulnerable version.

Re: [gentoo-dev] Unmasking >=media-video/ffmpeg-4.0

[Alexis Ballier]

On Tue, 06 Nov 2018 17:08:22 +0200
Mart Raudsepp  wrote:

> Ühel kenal päeval, T, 06.11.2018 kell 12:00, kirjutas Alexis Ballier:
> > On Mon, 05 Nov 2018 20:38:58 -0500
> > Craig Andrews  wrote:
> >   
> > > I think it's time to remove the mask on >=media-video/ffmpeg-4.0
> > > 
> > > ffmpeg 4 is in many distros (including Debian, Arch) and FreeBSD.
> > > Upstreams are starting to require it, too. For example, Kodi 18
> > > and 
> > > media-plugins/gst-plugins-libav-16 will require it.
> > > 
> > > The blocker issue https://bugs.gentoo.org/653678 has only 5
> > > blockers 
> > > against it right now:
> > > * 654628 has a pull request out to fix it: 
> > > https://github.com/gentoo/gentoo/pull/10341
> > > * 655170 is for a package that (IMHO) should be last-rited: 
> > > media-libs/mediastreamer
> > > * 666166 is for a package that (IMHO) should be last-rited: 
> > > media-plugins/vdr-image
> > > * 655442 and 658128 are for media-video/mplayer and fixed
> > > upstream, 
> > > requiring a new snapshot release
> > > 
> > > What do we say? Can we set a date when the hard mask will be
> > > removed?  
> > 
> > 
> > I think 6 months qualifies for ETIMEOUT, so go ahead and unmask
> > it :)
> > 
> > merge the github PR; I'll take care of mplayer  
> 
> I'm sorry, but what?
> PR was filed yesterday, what 6 months are you talking about here?
> Please respect your fellow developers. We have other commitments and
> priorities and can't zero-day review a PR.


Oh, sorry, I did not even check what the PR was about nor its
uptime. It should definitely read as "get the PR merged" then.

6 months is when bugs started to be filled which by most standards a
package not fixed within that time can be considered dead/unmaintained.
I usually take no response for a couple of months as a green light to
do it myself.


> It is not GStreamer fault that ffmpeg breaks API and ABI without
> parallel installability, much less so the distro maintainers of it.
> If you/upstream don't make it parallel installable, then this is what
> you get.


Are you, seriously, suggesting this is the solution to all problems
here ?

Bests,

Alexis.

Re: [gentoo-dev] Unmasking >=media-video/ffmpeg-4.0

[Alexis Ballier]

On Tue, 06 Nov 2018 11:04:17 -0500
Craig Andrews  wrote:

> On 06.11.2018 06:00, Alexis Ballier wrote:
> > On Mon, 05 Nov 2018 20:38:58 -0500
> > Craig Andrews  wrote:
> >   
> >> I think it's time to remove the mask on >=media-video/ffmpeg-4.0
> >> 
> >> ffmpeg 4 is in many distros (including Debian, Arch) and FreeBSD.
> >> Upstreams are starting to require it, too. For example, Kodi 18 and
> >> media-plugins/gst-plugins-libav-16 will require it.
> >> 
> >> The blocker issue https://bugs.gentoo.org/653678 has only 5
> >> blockers against it right now:
> >> * 654628 has a pull request out to fix it:
> >> https://github.com/gentoo/gentoo/pull/10341
> >> * 655170 is for a package that (IMHO) should be last-rited:
> >> media-libs/mediastreamer
> >> * 666166 is for a package that (IMHO) should be last-rited:
> >> media-plugins/vdr-image
> >> * 655442 and 658128 are for media-video/mplayer and fixed upstream,
> >> requiring a new snapshot release
> >> 
> >> What do we say? Can we set a date when the hard mask will be
> >> removed?  
> > 
> > 
> > I think 6 months qualifies for ETIMEOUT, so go ahead and unmask
> > it :)
> > 
> > merge the github PR; I'll take care of mplayer  
> 
> Since Alexis added the mask I believe he can authorize its removal,
> so I have done so:
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=deb9d06fb461dc57291498b327e2eeb667444a5f


I forgot to mention it, but you should probably have checked the re-kw
bug if any and created one after dropping keywords with broken deps :/

Re: [gentoo-dev] [PATCH] opam.eclass: unbreak on EAPI=7

[Alexis Ballier]

On Sat,  2 Feb 2019 21:27:29 -0800
Georgy Yakovlev  wrote:

> Since D, ED, ROOT, EROOT no longer have a trailing slash in EAPI=7
> This eclass is terribly broken, installing things into
> imageusr/...


You might want to check https://github.com/aballier/ml-overlay

Re: [gentoo-dev] [PATCH] opam.eclass: unbreak on EAPI=7

[Alexis Ballier]

On Mon, 04 Feb 2019 13:42:21 -0800
Georgy Yakovlev  wrote:

> On Monday, February 4, 2019 2:52:37 AM PST Alexis Ballier wrote:
> > On Sat,  2 Feb 2019 21:27:29 -0800
> > 
> > Georgy Yakovlev  wrote:
> > > Since D, ED, ROOT, EROOT no longer have a trailing slash in EAPI=7
> > > This eclass is terribly broken, installing things into
> > > imageusr/...
> > 
> > You might want to check https://github.com/aballier/ml-overlay
> Hi!
> 
> I don't really use this eclass in any way.
> A user reported breakage, I wrote a patch and submitted here for
> review.
> 
> there is a commit in overlay that adds slashes, but paths will end up
> having 
> 
> // two slashes on EAPI6 and single on EAPI7 
> 
> https://github.com/aballier/ml-overlay/commit/
> 98c0f16bc490349f17afdd7a7675b9b5264d267e
> 
> also probably the docdir subdir part of opam_src_install() will  also
> fail, as there are no slashes.
> 
> 
> ::gentoo version needs some kind of fix as any EAPI7 ebuild that uses 
> opam.eclass is broken.
> 
> If you are ok with my patches I can commit, just let me know.


yes go ahead (modulo list comments ofc =)

Re: [gentoo-dev] Announcing RISC-V

[Alexis Ballier]

On Wed, 29 May 2019 10:27:34 -0700 (PDT)
Palmer Dabbelt  wrote:

> On Mon, 20 May 2019 02:44:18 PDT (-0700), aball...@gentoo.org wrote:
> > On Sat, 18 May 2019 20:47:28 +0200
> > Michał Górny  wrote:
> >
> >> On Fri, 2019-05-03 at 23:34 +0200, Andreas K. Huettel wrote:
> >> > * We will initially add two profiles to profile.desc: 
> >> >   default/linux/riscv/17.0/rv64gc/lp64d (non-multilib, 64bit
> >> > hardfloat) default/linux/riscv/17.0/rv64gc (multilib lp64d/lp64,
> >> > i.e. hard/softfloat)
> >> 
> >> I still don't understand the purpose of this multilib.  If you have
> >> a hardfloat CPU, why would you ever build some of the software
> >> softfloat?
> >
> > One reason I could imagine is that the hardfloat isn't IEEE 754
> > compliant. Searching through the RISC-V spec, it does not seem to be
> > the case here (ie: it is required to be compliant) so I'm also
> > wondering what is the point here.
> 
> The RISC-V floating-point extensions are IEEE-754 compliant, but
> they're optional.  We have chips without floating-point units, but
> right now all the Linux capable chips have FPUs.  As far as I know
> there are no Linux binaries that anyone cares about that are compiled
> for systems without hardware floating-point units, but I may be wrong
> about that one.


It was my understanding that FPU is not optional for rv64gc, is that
correct ?

> The non-FPU systems are much more interesting in embedded land, where
> lots of users don't have FPUs.  That's less relevant for Gentoo, but
> I do use crossdev embedded toolchains.


You'll probably not be using multilib here but rather a specific CHOST
and/or flags to enable softfloat everywhere.


[...]


Alexis.

Re: [gentoo-dev] Rethinking multilib flags in Gentoo

[Alexis Ballier]

On Wed, 8 May 2019 11:08:47 -0700
Matt Turner  wrote:

> On Wed, May 8, 2019 at 3:19 AM Alexis Ballier 
> wrote:
> >
> > On Wed, 08 May 2019 12:01:21 +0200
> > Michał Górny  wrote:
> >
> > > On Wed, 2019-05-08 at 11:54 +0200, Alexis Ballier wrote:
> > > > On Wed, 08 May 2019 11:41:41 +0200
> > > > Michał Górny  wrote:
> > > >
> > > > > > There's multilib that adds a lot of flags with a single
> > > > > > eclass change, but I'd guess the number of packages and
> > > > > > flags is constantly growing, so sooner or later you'll be
> > > > > > hit by this again and no multilib killing will help you
> > > > > > then.
> > > > > >
> > > > > > I think it is more future proof to use the addition of
> > > > > > multilib flags to fix pkgcheck rather than actively
> > > > > > reducing the number of multilib flags to cope with its
> > > > > > limitations.
> > > > >
> > > > > Then please do it, by all means.  The reality is simple.  If
> > > > > the tool is broken, you either fix it or stop doing what you
> > > > > know that breaks it. Being unable to do the former, and
> > > > > having no good replacement, I'd go for the latter.
> > > >
> > > > Well, why is it slow ? IO ? CPU ? Did you collect profiling
> > > > data ?
> > >
> > > CPU definitely.  More detail than that, I don't and I don't have
> > > time to investigate.
> >
> > So you don't have time to change 3 lines to add cProfile but do have
> > time to send various emails and rework the entire multilib system ?
> > weird.
> 
> This isn't productive.
> 
> If you'd like to do the work you're suggesting, I'm sure Michał will
> support that, but as is you're just passive-aggressively questioning
> his choices in the regards to the multilib system he created and the
> CI system he created.
> 


You're right about the passive-aggressive part, sorry about it. But
maybe there's a single not so important check that's killing the
performance and could be, instead, disabled meanwhile. No data has been
shown except that adding useflags to packages make CI explode.
I am indeed questioning the choices of making tree wide changes because
a CI system is under performing and see nothing wrong about this.

Re: [gentoo-dev] Announcing RISC-V

[Alexis Ballier]

On Sat, 18 May 2019 20:47:28 +0200
Michał Górny  wrote:

> On Fri, 2019-05-03 at 23:34 +0200, Andreas K. Huettel wrote:
> > * We will initially add two profiles to profile.desc: 
> >   default/linux/riscv/17.0/rv64gc/lp64d (non-multilib, 64bit
> > hardfloat) default/linux/riscv/17.0/rv64gc (multilib lp64d/lp64,
> > i.e. hard/softfloat)
> 
> I still don't understand the purpose of this multilib.  If you have
> a hardfloat CPU, why would you ever build some of the software
> softfloat?


One reason I could imagine is that the hardfloat isn't IEEE 754
compliant. Searching through the RISC-V spec, it does not seem to be
the case here (ie: it is required to be compliant) so I'm also
wondering what is the point here.


Note that hard vs soft float will be the least of our concerns
here: Based on wikipedia, I count 4 base ISA (2 stable, 2 dev) and 13
optional extensions (6 stable, 7 dev) so we potentially have an
exponential number of ABIs here...

Re: [gentoo-dev] Rethinking multilib flags in Gentoo

[Alexis Ballier]

On Tue, 07 May 2019 23:47:30 +0200
Michał Górny  wrote:

> While the large number of flags is practically invisible to user with
> all the USE_EXPAND hiding, it negatively impacts pkgcheck.  When
> the number reached 10, CI became unusable.  We're currently back down
> to 8, thanks to powerpc team, but the problem is going to happen again
> sooner or later.  Ideally we'd improve pkgcheck but I'm not aware of
> anyone having a good idea how to do it.


While I don't disagree with your rationale below, I think this
motivation is the wrong one: What sort of algorithm does it use
to explode when going from 8 to 10 flags ?!?

There's multilib that adds a lot of flags with a single eclass change,
but I'd guess the number of packages and flags is constantly growing,
so sooner or later you'll be hit by this again and no multilib killing
will help you then.

I think it is more future proof to use the addition of multilib flags
to fix pkgcheck rather than actively reducing the number of multilib
flags to cope with its limitations.

Also, remember that multilib is not entirely about skype or slack,
this was made with multibin in mind too: for example an ABI may perform
better than another one on specific workflows (x32) and it may make
sense to use this abi for a specific binary (which would be manually
built for now).

Alexis.

Re: [gentoo-dev] Rethinking multilib flags in Gentoo

[Alexis Ballier]

On Wed, 08 May 2019 11:41:41 +0200
Michał Górny  wrote:

> > There's multilib that adds a lot of flags with a single eclass
> > change, but I'd guess the number of packages and flags is
> > constantly growing, so sooner or later you'll be hit by this again
> > and no multilib killing will help you then.
> > 
> > I think it is more future proof to use the addition of multilib
> > flags to fix pkgcheck rather than actively reducing the number of
> > multilib flags to cope with its limitations.
> 
> Then please do it, by all means.  The reality is simple.  If the tool
> is broken, you either fix it or stop doing what you know that breaks
> it. Being unable to do the former, and having no good replacement,
> I'd go for the latter.


Well, why is it slow ? IO ? CPU ? Did you collect profiling data ?
Where are the scripts to repro the issue ?


> > Also, remember that multilib is not entirely about skype or slack,
> > this was made with multibin in mind too: for example an ABI may
> > perform better than another one on specific workflows (x32) and it
> > may make sense to use this abi for a specific binary (which would
> > be manually built for now).
> 
> No, it weren't.  Just because someone found it convenient to use the
> design beyond its original purpose doesn't mean it had a different
> purpose.  Besides, how many 'multibin' packages do we have right now?


It was, because portage multilib was doing this and claimed to have a
use for this. Its original purpose was generic enough to allow multibin
to be added easily. The number of multibin packages is only
relevant to show that this is not important enough for someone
to step up and do it: Everybody can easily build anything themselves
with the current implementation.


> That said, yes, I am also concerned about people proactively adding
> multilib to all random libraries which probably will never have any
> real multilib use case and only waste time of people who have
> abi_x86_32 enabled globally.

That last part is easily fixable: Have a useflag config option that
says "default off unless some package wants it on". This can be either
in the ebuild themselves (alike +/- iuse defaults) or, more simply, in
the package manager. None ever saw the light.

Re: [gentoo-dev] Rethinking multilib flags in Gentoo

[Alexis Ballier]

On Wed, 08 May 2019 12:31:47 +0200
Michał Górny  wrote:

> On Wed, 2019-05-08 at 12:19 +0200, Alexis Ballier wrote:
> > On Wed, 08 May 2019 12:01:21 +0200
> > Michał Górny  wrote:
> > 
> > > On Wed, 2019-05-08 at 11:54 +0200, Alexis Ballier wrote:
> > > > On Wed, 08 May 2019 11:41:41 +0200
> > > > Michał Górny  wrote:
> > > > 
> > > > > > There's multilib that adds a lot of flags with a single
> > > > > > eclass change, but I'd guess the number of packages and
> > > > > > flags is constantly growing, so sooner or later you'll be
> > > > > > hit by this again and no multilib killing will help you
> > > > > > then.
> > > > > > 
> > > > > > I think it is more future proof to use the addition of
> > > > > > multilib flags to fix pkgcheck rather than actively
> > > > > > reducing the number of multilib flags to cope with its
> > > > > > limitations.
> > > > > 
> > > > > Then please do it, by all means.  The reality is simple.  If
> > > > > the tool is broken, you either fix it or stop doing what you
> > > > > know that breaks it. Being unable to do the former, and
> > > > > having no good replacement, I'd go for the latter.
> > > > 
> > > > Well, why is it slow ? IO ? CPU ? Did you collect profiling
> > > > data ?
> > > 
> > > CPU definitely.  More detail than that, I don't and I don't have
> > > time to investigate.
> > 
> > So you don't have time to change 3 lines to add cProfile but do have
> > time to send various emails and rework the entire multilib system ?
> > weird.
> 
> Instead of being so smug, you could have provided helpful instructions
> on how to do it.  Or did it yourself.

;)

I wasn't being smug, rather the contrary by assuming you would know
better than me.

So here we go:
https://docs.python.org/2/library/profile.html

$ python -m cProfile -o /full/path/to/file.log /usr/bin/command args

This will give you a profile log.

Then you need something to visualize it, I like runsnakerun.
http://www.vrplumber.com/programming/runsnakerun/


> Removing unused flags from multilib-build.eclass is certainly less
> work than that.

As a temporarily solution yes.

Re: [gentoo-dev] Rethinking multilib flags in Gentoo

[Alexis Ballier]

On Wed, 08 May 2019 12:01:21 +0200
Michał Górny  wrote:

> On Wed, 2019-05-08 at 11:54 +0200, Alexis Ballier wrote:
> > On Wed, 08 May 2019 11:41:41 +0200
> > Michał Górny  wrote:
> > 
> > > > There's multilib that adds a lot of flags with a single eclass
> > > > change, but I'd guess the number of packages and flags is
> > > > constantly growing, so sooner or later you'll be hit by this
> > > > again and no multilib killing will help you then.
> > > > 
> > > > I think it is more future proof to use the addition of multilib
> > > > flags to fix pkgcheck rather than actively reducing the number
> > > > of multilib flags to cope with its limitations.
> > > 
> > > Then please do it, by all means.  The reality is simple.  If the
> > > tool is broken, you either fix it or stop doing what you know
> > > that breaks it. Being unable to do the former, and having no good
> > > replacement, I'd go for the latter.
> > 
> > Well, why is it slow ? IO ? CPU ? Did you collect profiling data ?
> 
> CPU definitely.  More detail than that, I don't and I don't have time
> to investigate.

So you don't have time to change 3 lines to add cProfile but do have
time to send various emails and rework the entire multilib system ?
weird.


> > > > Also, remember that multilib is not entirely about skype or
> > > > slack, this was made with multibin in mind too: for example an
> > > > ABI may perform better than another one on specific workflows
> > > > (x32) and it may make sense to use this abi for a specific
> > > > binary (which would be manually built for now).
> > > 
> > > No, it weren't.  Just because someone found it convenient to use
> > > the design beyond its original purpose doesn't mean it had a
> > > different purpose.  Besides, how many 'multibin' packages do we
> > > have right now?
> > 
> > It was, because portage multilib was doing this and claimed to have
> > a use for this.
> 
> I am not talking of 'portage multilib'.  I am talking of
> multilib-build implementation.  If you are only concerned about the
> former, we can surely drop those abi_* flags that are irrelevant to
> it, can't we?

I am also talking about multilib-build and its way of doing cleanly and
replacing portage multilib.


> >  Its original purpose was generic enough to allow multibin
> > to be added easily. The number of multibin packages is only
> > relevant to show that this is not important enough for someone
> > to step up and do it: Everybody can easily build anything themselves
> > with the current implementation.
> 
> Exactly.  And that's why I believe having fast CI is more important
> than DIY multibin.  If someone can 'easily build anything
> themselves', they can also add needed flags locally.


It's a "little" more complex than just adding one flag to an eclass and
cannot really be done with an overlay; in the end, this requires to fork
gentoo.git more or less.

Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable

[Alexis Ballier]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Tue, 30 Jul 2019 23:26:27 +0100
James Le Cuirot  wrote:

> > Admittedly without a full understanding of the problem, but this
> > looks wrong to me: SYSROOT, EPREFIX and BROOT are only relevant in
> > build phases (src_*); (EPREFIX is a little special here but mostly
> > for convenience). ROOT is only relevant in pkg_* phases. I don't
> > see how this can work. Say I build a binpkg with ROOT=/ then use
> > that binpkg with ROOT=/somewhere, you can't go back and change
> > SYSROOT.
> 
> ROOT is used to determine ESYSROOT, not the other way around. As you
> say, (E)SYSROOT is only relevant in src phases so it doesn't matter if
> ROOT has changed when installing a binpkg.

I am missing something here: You are making ESYSROOT depend on the
value of ROOT, so how can it not matter ?

> I take your point that setting a src phase variable based on a pkg
> phase variable seems odd but we're only using ROOT to determine the
> applicable prefix. We're not taking the actual value of ROOT. When
> Portage works all this out, it has access to all the necessary
> variables. It only filters the variables based on the phase function
> later on.

The value does not really matter, the dependency of a variable used in
src_* from a variable that can change when installing a binpkg is what
worries me here.


Alexis.
-BEGIN PGP SIGNATURE-

iHUEAREIAB0WIQSpOxaxaZikKNVNlsYOJUi7xgflrgUCXUGc/gAKCRAOJUi7xgfl
rrlNAP9AEX0enc5Tzh++N6I+P4ZKp0hBdljlYBteYTuEipZLEAD/a3fT/pgOR3HJ
LyGvu98wHn6sCldtBMjoV/RgrxfaKO8=
=L9Hp
-END PGP SIGNATURE-

Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable

[Alexis Ballier]

On Sun, 28 Jul 2019 22:37:35 +0100
James Le Cuirot  wrote:

[...]
> -& \t{BDEPEND} &
> \t{DEPEND} & \t{RDEPEND}, \t{PDEPEND} \\
> +& \t{BDEPEND} &
> \t{DEPEND}& \t{RDEPEND}, \t{PDEPEND} \\
> \midrule


There seems to be unneeded whitespace only changes here that make the
diff less readable

[...]
>  \t{ESYSROOT} &
>  Ditto &
>  No &
> -Contains the concatenation of the paths in the \t{SYSROOT} and
> \t{EPREFIX} variables,
> -for convenience. See also the \t{EPREFIX} variable. Only for
> EAPIs listed
> -in table~\ref{tab:offset-env-vars-table} as supporting
> \t{ESYSROOT}. \\
> +Contains the concatenation of the \t{SYSROOT} path and
> applicable prefix value. The prefix value
> +is \t{EPREFIX}, \t{BROOT}, or blank if \t{SYSROOT} is equal to
> \t{ROOT}, \t{/}, or something
> +else respectively. Only for EAPIs listed in
> table~\ref{tab:offset-env-vars-table} as supporting
> +\t{ESYSROOT}. \\
>  \t{BROOT} &
>  Ditto &
>  No &

Admittedly without a full understanding of the problem, but this looks
wrong to me: SYSROOT, EPREFIX and BROOT are only relevant in build
phases (src_*); (EPREFIX is a little special here but mostly for
convenience). ROOT is only relevant in pkg_* phases. I don't see how
this can work. Say I build a binpkg with ROOT=/ then use that binpkg
with ROOT=/somewhere, you can't go back and change SYSROOT.

Also, I think the wording could be more "programmatic" (if ... then
ESYSROOT is ... else ... ) to avoid confusion.


Alexis.

Re: [gentoo-dev] [PMS] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable

[Alexis Ballier]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Wed, 31 Jul 2019 21:40:19 +0100
James Le Cuirot  wrote:

> On Wed, 31 Jul 2019 15:51:58 +0200
> Alexis Ballier  wrote:
> 
> > On Tue, 30 Jul 2019 23:26:27 +0100
> > James Le Cuirot  wrote:
> > 
> > > > Admittedly without a full understanding of the problem, but this
> > > > looks wrong to me: SYSROOT, EPREFIX and BROOT are only relevant
> > > > in build phases (src_*); (EPREFIX is a little special here but
> > > > mostly for convenience). ROOT is only relevant in pkg_* phases.
> > > > I don't see how this can work. Say I build a binpkg with ROOT=/
> > > > then use that binpkg with ROOT=/somewhere, you can't go back
> > > > and change SYSROOT.  
> > > 
> > > ROOT is used to determine ESYSROOT, not the other way around. As
> > > you say, (E)SYSROOT is only relevant in src phases so it doesn't
> > > matter if ROOT has changed when installing a binpkg.  
> > 
> > I am missing something here: You are making ESYSROOT depend on the
> > value of ROOT, so how can it not matter ?
> 
> What I am trying to say (somewhat unsuccessfully!) is that the value
> of (E)SYSROOT only changes how the package is built, not what the
> resulting package looks like. It's where all the headers and libraries
> are sourced from at build time, which is irrelevant at runtime.

err, SYSROOT does change what the resulting package looks like: it
defines ABI (headers and needed entries for example).

> So why does ROOT affect it? Normally you install the packages for
> BDEPEND, DEPEND, and RDEPEND to the same location. If BDEPEND and
> RDEPEND are installed to different locations (ROOT!=/) then DEPEND
> will almost always be installed to one of the other two. If either of
> those two locations is prefixed then we need the prefix for DEPEND's
> location to match, otherwise it wouldn't actually be the same
> location. Using ROOT allows us to figure this out automatically in a
> way that covers all sensible use cases and avoids accidentally
> falling into an unsupported case.


So, now let's simplify this a bit and forget about prefix and crossdev
for a moment. Say I am building an atom chroot (for nfs root) on an
haswell desktop. I set ROOT=SYSROOT=/atomchroot. My desktop has CFLAGS
for haswell, and I have atom CFLAGS in
/atomchroot/etc/portage/make.conf. When I build something and portage
installs a BDEPEND to /, I want it to use my / configuration, and when
it is in /atomchroot I want it to use the configuration from there.
emerge has command line options to do that but I am not sure if this is
properly specified in PMS.


Now, say I am doing the same on a prefix haswell desktop. With your
algorithm, we have SYSROOT==ROOT so ESYSROOT is EPREFIX/SYSROOT.
However, what I want is a normal chroot with EPREFIX=/ here, so when
should one use ESYSROOT in an ebuild in that case ? where does this
EPREFIX comes from by the way ?


To me, this looks more of a general problem of defining where the
configuration is taken from, so that we can set EPREFIX independently
if it is SYSROOT or BROOT.



> I hope that's clearer.

Sorry :-(
-BEGIN PGP SIGNATURE-

iHUEAREIAB0WIQSpOxaxaZikKNVNlsYOJUi7xgflrgUCXUL2ygAKCRAOJUi7xgfl
rkwRAP9LDImZBCYxsWKMjT/ckCeK0hOLtctdpZuBwzjv5RIuiAD/cTUj7P7h9rBd
gYaEEI+pqdN25ZNbjao/8w/j7SsXUMo=
=WYef
-END PGP SIGNATURE-

Re: [gentoo-dev] [PATCH] virtualx.eclass: Append RESTRICT="!test? ( test )" by default

[Alexis Ballier]

On Thu, 2019-12-12 at 12:14 -0500, Mike Gilbert wrote:
> On Thu, Dec 12, 2019 at 12:02 PM NP-Hardass 
> wrote:
> > On 12/11/19 9:58 AM, Michał Górny wrote:
> > > Append RESTRICT="!test? ( test )" in the default case when
> > > virtualx
> > > is conditional to USE=test.  This fixes 440 MissingTestRestrict
> > > warnings.
> > > 
> > > Signed-off-by: Michał Górny 
> > > ---
> > >  eclass/virtualx.eclass | 2 ++
> > >  1 file changed, 2 insertions(+)
> > > 
> > > diff --git a/eclass/virtualx.eclass b/eclass/virtualx.eclass
> > > index 40eeea5463bc..6aba6bf488dd 100644
> > > --- a/eclass/virtualx.eclass
> > > +++ b/eclass/virtualx.eclass
> > > @@ -89,6 +89,8 @@ case ${VIRTUALX_REQUIRED} in
> > >   fi
> > >   RDEPEND=""
> > >   IUSE="${VIRTUALX_REQUIRED}"
> > > + [[ ${VIRTUALX_REQUIRED} == test ]] &&
> > > + RESTRICT+=" !test? ( test )"
> > >   ;;
> > >  esac
> > > 
> > > 
> > 
> > Is there a better way to address this than editing a ton of
> > eclasses
> > independently?
> 
> Not really.
> 

Seems a good candidate for a future EAPI

Re: [gentoo-dev] [PATCH] package.deprecated: Create initial template

[Alexis Ballier]

On Thu, 2019-12-05 at 17:09 +0100, Michał Górny wrote:
> +
> +#
> +# This file specifies packages that are considered deprecated (but
> not
> +# masked yet).  It will trigger pkgcheck warnings whenever other
> +# packages depend on them.
> 


repoman would be more useful for this

Re: [gentoo-dev] [PATCH] package.deprecated: Create initial template

[Alexis Ballier]

On Thu, 2019-12-05 at 10:53 -0600, William Hubbs wrote:
> On Thu, Dec 05, 2019 at 05:36:58PM +0100, Alexis Ballier wrote:
> > On Thu, 2019-12-05 at 17:09 +0100, Michał Górny wrote:
> > > +
> > > 
> > > +#
> > > +# This file specifies packages that are considered deprecated
> > > (but
> > > not
> > > +# masked yet).  It will trigger pkgcheck warnings whenever other
> > > +# packages depend on them.
> > > 
> > 
> > repoman would be more useful for this
> 
> I wouldn't stop pkgcheck from supporting this, but repoman should as
> well.
> 
> 
> 

of course, that's what i meant ;)

Re: [gentoo-dev] [PATCH] package.deprecated: Create initial template

[Alexis Ballier]

On Thu, 2019-12-05 at 18:39 +0100, Michał Górny wrote:
> On Thu, 2019-12-05 at 17:36 +0100, Alexis Ballier wrote:
> > On Thu, 2019-12-05 at 17:09 +0100, Michał Górny wrote:
> > > +
> > > 
> > > +#
> > > +# This file specifies packages that are considered deprecated
> > > (but
> > > not
> > > +# masked yet).  It will trigger pkgcheck warnings whenever other
> > > +# packages depend on them.
> > > 
> > 
> > repoman would be more useful for this
> > 
> 
> Then feel free to take repoman over, and start maintaining it.  I've
> lost interest in contributing to the project after the last pointless
> refactoring made adding anything even more effort, and it doesn't
> seem
> that anyone else has.
> 
> Given that pkgcheck is a. faster by design, b. running checks
> in parallel, c. has sane API making contributing a pleasure, I don't
> really see a point in putting any more effort to support a dead
> repoman.
> 

it's not about who's maintaining what here...
just s/pkgcheck/QA tools/ and be done with it


unless i missed something, repoman is still the standard for pre-commit 
checks and raising everyone's attention on potential
improvements/issues; pkgcheck is mostly used by your CI checks for
producing huge reports, which is nice but addresses a different problem

i could see this file being useful for auto-generating lists on qa-
reports like for eapis too


as for your current views on repoman vs pkgcheck, i have nothing
against stopping using repoman and switching to pkgcheck, but AFAIK
this has yet to happen at a policy level

Re: [gentoo-dev] [PATCH] package.deprecated: Create initial template

[Alexis Ballier]

On Thu, 2019-12-05 at 19:04 +0100, Michał Górny wrote:
> On Thu, 2019-12-05 at 18:59 +0100, Alexis Ballier wrote:
> > On Thu, 2019-12-05 at 18:39 +0100, Michał Górny wrote:
> > > On Thu, 2019-12-05 at 17:36 +0100, Alexis Ballier wrote:
> > > > On Thu, 2019-12-05 at 17:09 +0100, Michał Górny wrote:
> > > > > +
> > > > > 
> > > > > 
> > > > > +#
> > > > > +# This file specifies packages that are considered
> > > > > deprecated
> > > > > (but
> > > > > not
> > > > > +# masked yet).  It will trigger pkgcheck warnings whenever
> > > > > other
> > > > > +# packages depend on them.
> > > > > 
> > > > 
> > > > repoman would be more useful for this
> > > > 
> > > 
> > > Then feel free to take repoman over, and start maintaining
> > > it.  I've
> > > lost interest in contributing to the project after the last
> > > pointless
> > > refactoring made adding anything even more effort, and it doesn't
> > > seem
> > > that anyone else has.
> > > 
> > > Given that pkgcheck is a. faster by design, b. running checks
> > > in parallel, c. has sane API making contributing a pleasure, I
> > > don't
> > > really see a point in putting any more effort to support a dead
> > > repoman.
> > > 
> > 
> > it's not about who's maintaining what here...
> > just s/pkgcheck/QA tools/ and be done with it
> 
> Oh, I've listed pkgcheck there because it's the only tool
> implementing
> the file at the moment.  I'm happy to replace it with larger list or
> something more generic once there are other tools.  However, I
> believe
> that saying 'pkgcheck' right now has the advantage that devs know
> which
> tool to use to see the result.

IMHO maintaining such a list is better suited for devmanual or wiki;
just like skel.ebuild could be improved by removing portage references
and refer to PMS


> > pkgcheck is mostly used by your CI checks for
> > producing huge reports, which is nice but addresses a different
> > problem
> 
> There is nothing stopping you from running pkgcheck locally.  In
> fact,
> it should work out of the box these days.  If you have any problems,
> please report them and I'm sure they will be addressed promptly.

Sure I did that to get reports like what CI does for me now but that's
always been a different usecase; I wasn't aware pkgcheck had the
equivalent of repoman commit


> > i could see this file being useful for auto-generating lists on qa-
> > reports like for eapis too
> 
> I don't think there's really a point in duplicating this.

For now certainly not. Once someone wants to wipe a deprecated package
this could come in handy instead of searching a huge html page, but
sure this could be fixed the other way by having this in the per-check
reports like what is on the left side of the current CI reports

Re: [gentoo-dev] [PATCH] package.deprecated: Create initial template

[Alexis Ballier]

On Fri, 2019-12-06 at 03:23 -0500, Tim Harder wrote:
> On 2019-12-05 Thu 17:00, Alexis Ballier wrote:
> > > > pkgcheck is mostly used by your CI checks for
> > > > producing huge reports, which is nice but addresses a different
> > > > problem
> > > There is nothing stopping you from running pkgcheck locally.  In
> > > fact,
> > > it should work out of the box these days.  If you have any
> > > problems,
> > > please report them and I'm sure they will be addressed promptly.
> > Sure I did that to get reports like what CI does for me now but
> > that's
> > always been a different usecase; I wasn't aware pkgcheck had the
> > equivalent of repoman commit
> 
> While I dislike contributing more to this off-topic tangent, since
> I've
> fielded this question/request in IRC a few times in the past I figure
> I
> might as well address it again here for the IRC-averse.
> 
> Personally I use pkgcheck as a QA tool and *git* (or another vcs
> tool)
> as a commit tool, just like how I used to use repoman and cvs a long
> time ago. I generally dislike when cli tools amalgamate disparate
> features that they weren't designed for so no one has been able to
> convince me why a tool designed to verify ebuilds and their related
> repos should support commit capabilities internally.

it's not just like repoman and cvs since repoman commit did push ;)
it will never be perfect but i really like repoman commit to refuse to
even commit if there's something obviously wrong

as you write below, it's just a matter of checking exit status and
using git, which can be done by scripting, but the script is standard
(*) and mandated to be part of the workflow

it also allows to check or templatize commit messages to follow policy

(*) and force the use of some handy git options like only commit paths
starting from cwd even if other files had been git added, which i never
remember what is the git cli option for this

Alexis.

Re: [gentoo-dev] [PATCH] package.deprecated: Create initial template

[Alexis Ballier]

On Fri, 6 Dec 2019 04:33:36 -0500
Tim Harder  wrote:

> On 2019-12-06 Fri 04:03, Alexis Ballier wrote:
> > it's not just like repoman and cvs since repoman commit did push ;)
> > it will never be perfect but i really like repoman commit to refuse
> > to even commit if there's something obviously wrong
> 
> I'm more of the opinion (and am working towards that practicality in
> terms of runtime speed) that a subset of QA checks should be run as a
> git hook which would cause push failures on certain classes of bad
> commits.


There should be both. Amending the 23rd commit message because one
mistyped a semicolon is kind of a PITA.

> > as you write below, it's just a matter of checking exit status and
> > using git, which can be done by scripting, but the script is
> > standard (*) and mandated to be part of the workflow
> 
> > it also allows to check or templatize commit messages to follow
> > policy
> 
> Technically pkgcheck supports more git-related checks than repoman
> last I checked, i.e. result keywords including BadCommitSummary,
> DirectStableKeywords, DroppedUnstableKeywords, DroppedStableKeywords,
> DirectNoMaintainer, and MissingSignOff; with possible future additions
> such as warning when modifying deps in an ebuild without revbumping.
> 
> Futhermore, one can scan against all commits in parallel via `pkgcheck
> scan --commits` which will enable potential commit results that are
> otherwise skipped.

All this seems post-commit, not pre-commit.

> Anyway, my main point is that if someone really wants commit
> functionality it's semi-trivial to script something similar to what
> repoman does (assuming exit status/api support exists) and if it's
> decent enough quality (including tests) I'd probably consider adding
> it to the pkgcheck repo.

It doesn't necessarily have to live in the pkgcheck repo, but it should
definitely not be "meh, script it yourself, it's trivial" since that
will probably lead to several scripts with varying degrees of quality
and brokeness.

Re: [gentoo-dev] [PATCH] package.deprecated: Create initial template

[Alexis Ballier]

On Fri, 6 Dec 2019 10:41:56 -0500
Matt Turner  wrote:

> On Fri, Dec 6, 2019 at 5:51 AM Alexis Ballier 
> wrote:
> >
> > On Fri, 6 Dec 2019 04:33:36 -0500
> > Tim Harder  wrote:
> >
> > > On 2019-12-06 Fri 04:03, Alexis Ballier wrote:
> > > > it's not just like repoman and cvs since repoman commit did
> > > > push ;) it will never be perfect but i really like repoman
> > > > commit to refuse to even commit if there's something obviously
> > > > wrong
> > >
> > > I'm more of the opinion (and am working towards that practicality
> > > in terms of runtime speed) that a subset of QA checks should be
> > > run as a git hook which would cause push failures on certain
> > > classes of bad commits.
> >
> >
> > There should be both. Amending the 23rd commit message because one
> > mistyped a semicolon is kind of a PITA.
> 
> It is?
> 
> git rebase -i HEAD~23
> 
> Is that what you think is a pain in the ass, or do you not know about
> interactive rebase? :)


You made me look at the doc and I indeed had never used the reword
option ;) got stuck at pick/squash/edit somehow and that's the edit I
did consider a PITA yes

Without good integration from the checker it is probably a PITA to
figure out that 23 too and also still doesn't help for broken commits
(not messages) that may or may not trigger later conflicts (unless we
decide we don't care about working commits, just working pushes, which
WFM)

Re: [gentoo-dev] unsanctioned python 2.7 crusade

[Alexis Ballier]

On Fri, 2019-12-06 at 21:28 +0100, Thomas Deutschmann wrote:
> On 2019-12-06 21:10, Andreas Sturmlechner wrote:
> > Just so we're on the same page, a recent example of what some
> > people 
> > suggesting to keep py27 ad nauseam are asking users to deal with:
> > [...]
> > WARNING: One or more updates/rebuilds have been skipped due to a
> > dependency 
> > conflict:
> 
> Yes, like said, at some point you cannot prevent that. Remember when
> I
> bumped libvpx to v1.8.0 and people yelled at me because they are now
> seeing that message all the time (If you are using gnome you probably
> know the same msg which triggers for unicode stuff which I am also
> responsible for) because I bumped that package but not everything
> supports that version yet?


For having maintained packages that often have this issue for years, I
must say this is a very bad idea: You are asking (or doing yourself)
consumer packages to have < deps on your package. This falsely gives
the impression that the non-latest version is still maintained. This
also makes removing this old version very error prone (we do have tools
to check for that but those are not in the standard workflow). Not sure
how portage handles this, but negative deps (<, =, ! & co) are much
harder to solve than purely positive ones -- PM probably uses some
heuristics but then this has some limitations and if the number of such
deps grows too much it may fail to solve them or do the right thing.

I think it is a much better way to package.mask the newest version of
your lib until all consumers work with it, or those that don't are
masked. This is how we handled such transitions before portage improved
its handling of negative deps and is IMHO still better.


Alexis.

Re: [gentoo-dev] Last rites: dev-python/* leaf packages

[Alexis Ballier]

On Wed, 2019-12-04 at 19:15 -0500, Aaron Bauman wrote:
> * Removal in 30 days
> 


IMHO masking with unfixed, or much later, removal date will better help
achieve your goal: You are making your point by having them masked so
that it will make enough noise for interested people to understand py2
only is not a thing anymore.

We already see a lot of "false positives", there are probably packages
that just work but are lacking attention. If after a longer time those
packages are still not fixed, you can probably safely remove them with
the "meh, nobody cares anyway" reason and not even bother having to
check hundreds (?) of packages yourself. The 30 days is usually a
guideline for packages that have known issues but seems a bit short for
checking if someone cares about a package using a deprecated but
working python.


Also, your list is missing dev-ros/* which is py2 only. I hope I'll be
able to update them soon, last time I tried I failed miserably though
since the whole stack is really python-single style so that one broken
package with py3 causes the whole stack to be py2...


Alexis.

Re: [gentoo-dev] [EAPI 8 RFC] Install-time dependencies

[Alexis Ballier]

On Thu, 2019-12-19 at 20:40 +0100, Michał Górny wrote:
> Hello,
> 
> Here's another potential EAPI 8 feature I'd like to discuss.  Please
> note that this is about *new dependency type*, so please don't hijack
> it
> into the big 'let's steal Exherbo syntax' debate.
> 
> Bug: https://bugs.gentoo.org/660306
> 
> 
> The problem
> ===
> 
> Right now we don't really have a clean way of specifying dependencies
> that are used during pkg_*inst (and pkg_*rm?) phases.  So far RDEPEND
> was used as a 'close enough' alternative (except for a few developers
> who rejected it as 'invalid' and used DEPEND which is even more
> wrong). 
> However, this is no longer sufficient with EAPI 7 cross support.
> 
> By design, pkg_*inst phases are run in build host's environment when
> cross is used (because obviously you can't run target host
> executables).
> Therefore, the relevant dependencies need to be installed into CBUILD
> root, while RDEPEND is installed into CHOST root.
> 
> 
> The proposed solution
> =
> 
> The proposal is to add a new dependency type (codename: IDEPEND)
> which
> indicates dependencies used for pkg_*inst (and pkg_*rm?)
> phases.  Those
> dependencies would be installed into CBUILD root (like BDEPEND), and
> therefore would be runnable from build host.  Similarly to RDEPEND,
> they
> would be installed for binary package installs but not for pure
> binpkg
> builds (without install).
> 
> Example:
> 
>   inherit xdg-utils
> 
>   IDEPEND="dev-util/desktop-file-utils"
> 
>   pkg_postinst() {
> xdg_desktop_database_update
>   }
> 
> 
> WDYT?
> 


Should we use this to drop RDEPEND from pkg_*inst/rm phases ?
PMS states "RDEPEND (unless the particular dependency results in a
circular dependency, in which case it may be installed later)" which is
kind of "maybe maybe not" and I'm not sure how one can rely on RDEPEND
for those phases in their ebuilds if another random ebuild can trigger
a case where it's not satisfied.

Re: [gentoo-dev] [EAPI 8 RFC] Install-time dependencies

[Alexis Ballier]

On Fri, 2019-12-20 at 18:55 +0100, Ulrich Mueller wrote:
> > > > > > On Fri, 20 Dec 2019, Alexis Ballier wrote:
> > Should we use this to drop RDEPEND from pkg_*inst/rm phases ?
> > PMS states "RDEPEND (unless the particular dependency results in a
> > circular dependency, in which case it may be installed later)"
> > which is
> > kind of "maybe maybe not" and I'm not sure how one can rely on
> > RDEPEND
> > for those phases in their ebuilds if another random ebuild can
> > trigger
> > a case where it's not satisfied.
> 
> This wording has already been updated in git. It just says "RDEPEND"
> now
> for these four phases:
> 
> https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-720008.1
> 
> 

Ok.
My question still stands though, so that we can avoid mutual RDEPEND
cycles and also, as you noted on the bug, in a x-compile setting, CHOST
deps are not really needed at installation.

Re: [gentoo-dev] [PATCH 0/4] elt-patches: support wrapped Win32 MSVC toolchain

[Alexis Ballier]

On Thu, 2020-03-12 at 09:06 +0100, ha...@gentoo.org wrote:
> As this native Win32 support is considered highly experimental still,
> I
> would like to apply the libtool patches for parity via elibtoolize
> only,
> without applying them in sys-devel/libtool itself yet.
> 

IIRC you need to do it this way, experimental or not: elibtoolize is
needed for packages whose autotools have been generated with an old
libtool (ie all of them for now). eautoreconf should call elibtoolize,
so, after having this in elt-patches, better focus on upstreaming this
in libtool itself so that the need for elibtoolize fades away with
time.

You will probably run into the same issues as in the old days with BSD:
not all packages run elibtoolize and you do not have a sane way to
force this besides editing ebuilds.


Alexis.

Re: [gentoo-dev] [PATCH 0/4] elt-patches: support wrapped Win32 MSVC toolchain

[Alexis Ballier]

On Thu, 2020-03-12 at 20:59 +, James Le Cuirot wrote:
> On Thu, 12 Mar 2020 11:23:04 +0100
> Alexis Ballier  wrote:
> 
> > On Thu, 2020-03-12 at 09:06 +0100, ha...@gentoo.org wrote:
> > > As this native Win32 support is considered highly experimental
> > > still,
> > > I
> > > would like to apply the libtool patches for parity via
> > > elibtoolize
> > > only,
> > > without applying them in sys-devel/libtool itself yet.
> > >   
> > 
> > IIRC you need to do it this way, experimental or not: elibtoolize
> > is
> > needed for packages whose autotools have been generated with an old
> > libtool (ie all of them for now). eautoreconf should call
> > elibtoolize,
> > so, after having this in elt-patches, better focus on upstreaming
> > this
> > in libtool itself so that the need for elibtoolize fades away with
> > time.
> > 
> > You will probably run into the same issues as in the old days with
> > BSD:
> > not all packages run elibtoolize and you do not have a sane way to
> > force this besides editing ebuilds.
> 
> I've long wanted to automatically apply elibtoolize to fix other
> cross-compile issues. I did come up with a rough prototype and it did
> work though I imagine it might break some packages. Maybe it should
> be
> opt-out rather than opt-in?

If a patch in elibtoolize might break something then it should not be
there in the first place: the function is called by a lot of packages.
However, we can assume that such packages have been tested whereas by
magically calling elibtoolize they have not. A good solution to avoid
this could be to modify the default src_prepare in EAPI8 to call it.

I think some hacks were implemented for fbsd via profile.bashrc because
the pain caused by *not* calling elibtoolize (soname changes) was worse
than having untested packages that might break under a red moon.


Alexis.

Re: [gentoo-dev] musl doesn't provide execinfo.h

[Alexis Ballier]

On Fri, 2020-03-27 at 08:03 -0400, Anthony G. Basile wrote:
> On 3/26/20 9:25 PM, Joshua Kinard wrote:
> > On 3/23/2020 04:21, Jaco Kroon wrote:
> > > Hi,
> > > 
> > > https://bugs.gentoo.org/713668 relates.
> > > 
> > >  * Searching for /usr/include/execinfo.h ...
> > > sys-libs/glibc-2.29-r7 (/usr/include/execinfo.h)
> > > 
> > > As I see I can either add an explicit depend on glibc which I'd
> > > prefer
> > > not to.  Or someone from the musl team could possibly assist on
> > > how to
> > > get the backtrace() set of calls on musl please?
> > > 
> > > Alternatively I need to add a test and simply path debug.c to
> > > only
> > > provide stub function for print_backtrace(FILE *fp) that just
> > > does
> > > fprintf(fp, "No backtrace() available to print a backtrace.\n");
> > > 
> > > Suggestions?
> > > 
> > > Kind Regards,
> > > Jaco
> > 
> > Some quick searching on google, it looks like the cleanest fix for
> > that bug
> > is dahdi-tools needs to be patched to only include execinfo.h or
> > only use
> > backtrace() on glibc-based systems, and that patch then sent to the
> > dahdi-tools upstream developers for inclusion in a future
> > release.  That
> > way, we're not dragging that patch around forever in the tree or in
> > the musl
> > overlay.
> > 
> > It also doesn't look like musl itself will ever implement
> > execinfo.h or
> > backtrace(), per this message in 2015 from the lead musl developer:
> > https://www.openwall.com/lists/musl/2015/04/09/3
> > 
> 
> Correct.  I've been adding -standalone packages to provide for
> features
> like fts, obstack, argp,etc. which are bundled into glibc but not
> really
> under the POSIX standard.
> 
> So either we patch packages to turn off backtrace() or we add
> libunwind-standalone to the tree.
> 


BTW, we had libexecinfo for fbsd, which seems also present in alpine:
https://pkgs.alpinelinux.org/package/edge/main/x86/libexecinfo

Re: [gentoo-dev] musl doesn't provide execinfo.h

[Alexis Ballier]

On Fri, 2020-03-27 at 19:07 -0400, Anthony G. Basile wrote:
> On 3/27/20 3:17 PM, Alexis Ballier wrote:
> > On Fri, 2020-03-27 at 08:03 -0400, Anthony G. Basile wrote:
> > > On 3/26/20 9:25 PM, Joshua Kinard wrote:
> > > > On 3/23/2020 04:21, Jaco Kroon wrote:
> > > > > Hi,
> > > > > 
> > > > > https://bugs.gentoo.org/713668 relates.
> > > > > 
> > > > >  * Searching for /usr/include/execinfo.h ...
> > > > > sys-libs/glibc-2.29-r7 (/usr/include/execinfo.h)
> > > > > 
> > > > > As I see I can either add an explicit depend on glibc which
> > > > > I'd
> > > > > prefer
> > > > > not to.  Or someone from the musl team could possibly assist
> > > > > on
> > > > > how to
> > > > > get the backtrace() set of calls on musl please?
> > > > > 
> > > > > Alternatively I need to add a test and simply path debug.c to
> > > > > only
> > > > > provide stub function for print_backtrace(FILE *fp) that just
> > > > > does
> > > > > fprintf(fp, "No backtrace() available to print a
> > > > > backtrace.\n");
> > > > > 
> > > > > Suggestions?
> > > > > 
> > > > > Kind Regards,
> > > > > Jaco
> > > > 
> > > > Some quick searching on google, it looks like the cleanest fix
> > > > for
> > > > that bug
> > > > is dahdi-tools needs to be patched to only include execinfo.h
> > > > or
> > > > only use
> > > > backtrace() on glibc-based systems, and that patch then sent to
> > > > the
> > > > dahdi-tools upstream developers for inclusion in a future
> > > > release.  That
> > > > way, we're not dragging that patch around forever in the tree
> > > > or in
> > > > the musl
> > > > overlay.
> > > > 
> > > > It also doesn't look like musl itself will ever implement
> > > > execinfo.h or
> > > > backtrace(), per this message in 2015 from the lead musl
> > > > developer:
> > > > https://www.openwall.com/lists/musl/2015/04/09/3
> > > > 
> > > 
> > > Correct.  I've been adding -standalone packages to provide for
> > > features
> > > like fts, obstack, argp,etc. which are bundled into glibc but not
> > > really
> > > under the POSIX standard.
> > > 
> > > So either we patch packages to turn off backtrace() or we add
> > > libunwind-standalone to the tree.
> > > 
> > 
> > BTW, we had libexecinfo for fbsd, which seems also present in
> > alpine:
> > https://pkgs.alpinelinux.org/package/edge/main/x86/libexecinfo
> > 
> > 
> 
> Had?  Is it in the tree now or should I look into adding it?

Restoring it: https://bugs.gentoo.org/683284

Re: [gentoo-dev] [RFC] Should NATTkA reject keywordreqs for packages with -arch (-*) keywords?

[Alexis Ballier]

On Wed, 2020-05-06 at 03:41 +0200, Thomas Deutschmann wrote:
> On 2020-05-06 00:52, James Le Cuirot wrote:
> > On Tue, 05 May 2020 22:19:59 +0200
> > Michał Górny  wrote:
> > > WDYT?
> > 
> > Play it safe. -* is frequently used for binary packages where an
> > arch
> > will simply either work or it won't, with little likelihood of the
> > situation changing. -arch is so rare that I don't recall ever
> > seeing
> > it. In either case, restoring an arch should be an explicit action.
> 
> +1
> 


+1

with the addition that -arch (by opposition to -*) is usually used for
specifying "this has been tested and is broken, don't waste your time
on it". Or at least used to be used that way.

If a package relies on arch specific support, then we could make a case
that it should be '-*' + a whitelist of arches having said support.
So, IMHO, -arch is quite version specific: package being broken is
likely due to a bug that may or may not be fixed in later versions,
hence, to me it makes total sense to have keyword reqs even for -arch
if this is a newer version that the one that initially had its -arch
added.

I also believe tools like ekeyword or repoman should reset -arch to
nothing, or at least issue a warning when adding new ebuilds with it,
which would make this simpler for nattka.


Alexis.

Re: [gentoo-dev] Please port your packages to Python 3.8

[Alexis Ballier]

On Wed, 2 Sep 2020 15:00:27 -0400
Michael Orlitzky  wrote:

> On 2020-09-02 14:08, Andreas Sturmlechner wrote:
> > On Wednesday, 2 September 2020 19:42:33 CEST Michael Orlitzky wrote:
> >> New USE flags generally change dependencies (as is the case here),
> >> so a new revision ensures that people are forced to install the
> >> ebuild that supports python-3.8. Otherwise, you will eventually
> >> find a lot of people stuck unable to upgrade because they have an
> >> ebuild installed that only supports <=python-3.7, and were never
> >> prompted to install the copy that supports python-3.8.
> > 
> > Python target changes must be done with -U, also documented by the 
> > accompanying repository news item, not really a problem.
> > 
> 
> If you want to write the GLEP that obsoletes the PMS, I might even
> support it at this point. But until then, requiring --changed-use to
> have a functional system is not allowed. Any PMS-compliant package
> manager must be able to use ::gentoo, including one that does not
> implement portage-only heuristics.
> 

?

if some upgrade wants a package with unmatched deps (e.g. not installed
at all or py38 usedep not satisfied), $PM will surely try to satisfy
it by installing an ebuild. I don't think PMS specifies this, nor should
it.

Re: [gentoo-dev] Re: [PATCH 1/2] acct-group.eclass: declare the missing dependency on shadow

[Alexis Ballier]

On Tue, 8 Sep 2020 15:54:14 -0400
David Michael  wrote:

> Hi,
> 
> This fix might not be so straightforward.  A configuration I tested
> hit a dependency loop with shadow -> pambase -> systemd -> a bunch of
> groups -> shadow.  It is possible to bootstrap around by emerging
> shadow with no USE flags first, but I don't know how acceptable it is
> to introduce new dep loops like this.


what happens without your change ?

my understanding is that this will be merged in that order:

1. a bunch of groups
2. systemd
3. pambase
4. shadow


in which case, the groups from 1. will fail if shadow is not present at
that point


PS: we have the 'build' useflag for this when building stage1's

Re: [gentoo-dev] Re: [PATCH 1/2] acct-group.eclass: declare the missing dependency on shadow

[Alexis Ballier]

On Wed, 9 Sep 2020 09:48:04 -0400
David Michael  wrote:

> On Wed, Sep 9, 2020 at 5:37 AM Alexis Ballier 
> wrote:
> > On Tue, 8 Sep 2020 15:54:14 -0400
> > David Michael  wrote:
> >
> > > Hi,
> > >
> > > This fix might not be so straightforward.  A configuration I
> > > tested hit a dependency loop with shadow -> pambase -> systemd ->
> > > a bunch of groups -> shadow.  It is possible to bootstrap around
> > > by emerging shadow with no USE flags first, but I don't know how
> > > acceptable it is to introduce new dep loops like this.
> >
> > what happens without your change ?
> 
> Someone reported an issue in https://bugs.gentoo.org/720948 that
> showed shadow and groups are not ordered during installation.  I am
> not sure what environment produced those symptoms since I never
> encountered it, but you can rage-clean shadow and a group, delete the
> group, then reinstall it to reproduce the problem.
> 

Yep, that's exactly why one needs the change you are proposing.
The dependency loop needs to be resolved, but introducing it like that
is IMHO better than failing like in the above bug because it is not
resolved properly.

Re: [gentoo-dev] [PATCH] ebuild-maintenance/removal: Process for virtual removal

[Alexis Ballier]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Mon, 7 Sep 2020 21:39:54 +1200
Kent Fredric  wrote:

> On Mon,  7 Sep 2020 08:14:52 +0200
> Michał Górny  wrote:
> 
> > However, please
> > +do not include it in the package.mask entry as users do
> > not need
> > +to be forced to proactively unmerge it.
> 
> I can think of a utilitarian value of doing this anyway.
> 
> Namely, it gives a window during `emerge -uD @world` where portage
> tells you that they have a masked package installed, and the reason.

I think portage also warns for installed packages with no corresponding
ebuild; the reason is somewhat generic though (sth like 'it is gone')

IMHO last rites windows have only one purpose: give time to people to
step up and fix the reason why it is going away
-BEGIN PGP SIGNATURE-

iHUEAREIAB0WIQSpOxaxaZikKNVNlsYOJUi7xgflrgUCX1YK/gAKCRAOJUi7xgfl
rrooAQCBK/WXgwMl1wm8jQh+e2oyD8WIAzSrzFFBCE7xWL1WGgEAgWSGBmG9ug8/
ZNfrHnOhBL2o6hZupzMy/84dX7DFBvk=
=pwzN
-END PGP SIGNATURE-

Re: [gentoo-dev] Please port your packages to Python 3.8

[Alexis Ballier]

On Fri, 4 Sep 2020 09:06:46 -0400
Michael Orlitzky  wrote:

> On 2020-09-04 08:54, Alexis Ballier wrote:
> > 
> > py37 will (*) still be installed as it cannot be depcleaned because
> > of 1. emerge won't fail since deps are satisfied.
> > 
> > 
> > (*) or rather should, but I think the only case that matters is a
> > valid system state where noone forced uninstall of a needed package
> > or manually rm'ed some random files
> > 
> 
> There's no need to speculate; use pkgcore for a month and come back
> and tell me how much comfort these hypotheticals were.

there's no speculation in assuming a consistent set of installed
packages (consistent as specified in... PMS ;) ); there is, however,
speculation in the hypothetical error messages when the installed set
of packages is inconsistent


> >> or..
> >>
> >>   3b. Some reverse dependency of foo-1.2.3 gets python-3.8 support.
> >>   4b. A user tries to install that revdep, but the PM sees that
> >>   the latest version of foo is already installed, and it (the
> >>   installed version) doesn't support python-3.8. Mysterious
> >>   error messages and manual intervention ensue.
> > 
> > 
> > precisely the case I wrote above: unsatisfied dep -> pull ebuild.
> > non-issue too.
> 
> It's easy to say "well this is not an issue because it can be solved
> by ..."

are you kidding ? are you seriously suggesting adding to PMS that PM
needs to pull ebuilds to install packages ? good luck with that ;)

Re: [gentoo-dev] Please port your packages to Python 3.8

[Alexis Ballier]

On Thu, 3 Sep 2020 14:17:06 -0400
Michael Orlitzky  wrote:

> On 2020-09-03 12:38, Alexis Ballier wrote:
> > 
> > if some upgrade wants a package with unmatched deps (e.g. not
> > installed at all or py38 usedep not satisfied), $PM will surely try
> > to satisfy it by installing an ebuild. I don't think PMS specifies
> > this, nor should it.
> > 
> 
> It's not an upgrade per se. The situation is roughly this:
> 
>   1. User installs foo-1.2.3.ebuild, which supports python-3.7.
>   2. Developer ninja-edits the ebuild to support python-3.8.
>   3a. (crickets)
>   4a. Developer removes python-3.7 support from foo-1.2.3.ebuild.
>   5a. The next time something pulls foo-1.2.3 into the depgraph,
>   the PM will see that the installed version of foo-1.2.3 depends
> on python-3.7, but that there is no python-3.7 in the tree or that
>   it's masked. Now emerge always fails.


py37 will (*) still be installed as it cannot be depcleaned because of
1. emerge won't fail since deps are satisfied.


(*) or rather should, but I think the only case that matters is a valid
system state where noone forced uninstall of a needed package or
manually rm'ed some random files

> 
> or..
> 
>   3b. Some reverse dependency of foo-1.2.3 gets python-3.8 support.
>   4b. A user tries to install that revdep, but the PM sees that
>   the latest version of foo is already installed, and it (the
>   installed version) doesn't support python-3.8. Mysterious
>   error messages and manual intervention ensue.


precisely the case I wrote above: unsatisfied dep -> pull ebuild.
non-issue too.


> What's happening is that the PM is using the metadata from the
> installed version of the package, rather than the ninja-edited
> metadata in the repo (how would it know which ebuilds were edited
> meaningfully?). That's completely legal according to the PMS, and
> also the smart thing to do: sourcing a few thousand lines of bash
> *just in case* there was an important change in some ebuild is a huge
> waste of users' time.

you seem to be confusing dynamic deps and unsatisfied deps here. A
package installed with py38 disabled (e.g. after a revbump) or no py38
support at all (e.g. without revbump) will not satisfy a [py38] usedep
in any case so will work just the same


> Developers have an easy way to signal that there was an important
> change: to bump the "r" number at the end of the ebuild. This forces
> any upgrade involving e.g. foo-1.2.3 to pull in foo-1.2.3-r1, and the
> fact that an upgrade is available is evident from `ls`, rather than
> from sourcing a mountain of bash. One developer makes a change, and
> it saves thousands of users each a lot of time. That's what we're all
> here for.

fixing non-issues does not seem so important to me :/

[...]

Re: [gentoo-dev] Improving warnings on packages.g.o

[Alexis Ballier]

On Wed, 2020-08-26 at 18:57 +, Max Magorsch wrote:
> Hi all,
> 
> Good news regarding packages.g.o!
> 
> While the new packages.g.o version went into production some time
> ago,
> this also led to some false warnings about outdated package versions.
> This is because we currently take information about outdated package
> versions from repology.org, which are not always accurate.
> 


Any plans on using the remoteid from metadata.xml ?
It's likely to be much more accurate since people have been filling
those manually for some time now ;)

Alexis

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Tue, 2020-05-26 at 14:14 -0400, Mike Gilbert wrote:
[...]
> 
> > 
> > Assuming that the pic performance penalty is really only relevant
> > on
> > legacy arches (like x86), here are a couple of options:
> > 
> > 1. Disable pic on arches where tie performance penalty is small.
> > 2. Force pic everywhere, performance be damned.
> 
> Option 1 should say "Disable pic on arches where the performance
> penalty is large."

Performance penalty is not really a matter of arch. Of course -fPIC
from C/C++ code has a small performance impact on x86 (bigger than on
amd64), but the gain is worth it. Here it is a matter of hand written
assembly that is not relocatable. The performance impact thus depends
on the gain of said assembly, which in the case of multimedia apps/libs
is huge because that's usually about vectorizing inner loops bodys
(somewhere between x4 or x8 speedups is not unusual for the part in
question).

That's why the policy has always been to have PIC everywhere with
exceptions/workarounds tied to the pic useflag, giving users and
profiles the choice.

Again, the real fix is to rewrite the assembly in a PIC way, or at
least have ifdefs to allow it, but that is hard. Packages that have the
option to have PIC or slightly faster non-PIC assembly do/should not
have a pic useflag and always use the PIC variant.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Tue, 26 May 2020 23:41:28 +0200
zo...@gentoo.org wrote:

> tisdag 26 maj 2020 kl. 19:54:51 CEST skrev  Alexis Ballier:
> > On Tue, 26 May 2020 10:45:39 -0400
> > 
> > Mike Gilbert  wrote:
> > > > Note that having the 'pic' useflag should be considered
> > > > something to be fixed: rewrite the asm in a PIC way. But these
> > > > days nobody has the will to do it since this is mostly an issue
> > > > on x86+pax, both being slowly decreasing.
> > > 
> > > Given that PaX has been stripped out of official Gentoo kernels
> > > due to the grsecurity licensing issue, I wonder if there is any
> > > other good reason to keep the "pic" USE flag today. Surely this
> > > affects a very small population of users.
> > 
> > Yeah that was my thought when I saw pax/grsec beginning to be more
> > hostile to open source. That's not my call but the hardened team's,
> > however I'm all for removing these workarounds entirely if there's
> > no point in having them anymore.
> Is not only PaX/Grsec that don't allow textrel. SELinux do it to and
> that is manline kernel.


k so that means there's no dropping the pic useflag then i guess

[gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Sun, 24 May 2020 20:25:11 + (UTC)
"Thomas Deutschmann"  wrote:

> commit: eba596db8a926adb18595549c89294ed0a1e929e
> Author: Thomas Deutschmann  gentoo  org>
> AuthorDate: Sun May 24 15:07:04 2020 +
> Commit: Thomas Deutschmann  gentoo  org>
> CommitDate: Sun May 24 20:23:50 2020 +
> URL:
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=eba596db
> 
> media-libs/x265: rework assembly support
> 
> Closes: https://bugs.gentoo.org/681878
> Package-Manager: Portage-2.3.99, Repoman-2.3.22
> Signed-off-by: Thomas Deutschmann  gentoo.org>
> 
>  media-libs/x265/metadata.xml|  1 +
>  media-libs/x265/x265-3.3.ebuild | 66
> - 2 files changed, 34
> insertions(+), 33 deletions(-)
> 
> diff --git a/media-libs/x265/metadata.xml
> b/media-libs/x265/metadata.xml index 22a07293b83..c585d553631 100644
> --- a/media-libs/x265/metadata.xml
> +++ b/media-libs/x265/metadata.xml
> @@ -5,6 +5,7 @@
>  media-vi...@gentoo.org
>
>
> +Enable x86_64 assembly optimizations.



This should not even be an useflag.
Either it works or does not. Individual features support is controlled
by cpu_flags_* (or built-in and autodetected at runtime).
Please fix.



>  Add support for producing 10bits HEVC.
>  Add support for producing 12bits HEVC.
>  Build with support for NUMA nodes.
> 
> diff --git a/media-libs/x265/x265-3.3.ebuild
> b/media-libs/x265/x265-3.3.ebuild index 9fc0159bc00..f5c4fee6d97
> 100644 --- a/media-libs/x265/x265-3.3.ebuild
> +++ b/media-libs/x265/x265-3.3.ebuild
> @@ -19,15 +19,17 @@ HOMEPAGE="http://x265.org/
> https://bitbucket.org/multicoreware/x265/wiki/Home; LICENSE="GPL-2"
>  # subslot = libx265 soname
>  SLOT="0/188"
> -IUSE="+10bit +12bit cpu_flags_arm_neon numa pic power8 test"
> +IUSE="+asm +10bit +12bit cpu_flags_arm_neon numa pic power8 test"
>  
>  # Test suite requires assembly support and is known to be broken
>  RESTRICT="test"
>  
>  ASM_DEPEND=">=dev-lang/yasm-1.2.0"
>  
> -BDEPEND="abi_x86_32? ( ${ASM_DEPEND} )
> - abi_x86_64? ( ${ASM_DEPEND} )"
> +BDEPEND="asm? (
> + abi_x86_32? ( ${ASM_DEPEND} )
> + abi_x86_64? ( ${ASM_DEPEND} )
> + )"
>  
>  RDEPEND="numa? ( >=sys-process/numactl-2.0.10-r1[${MULTILIB_USEDEP}]
> )" 
> @@ -85,17 +87,6 @@ x265_variant_src_configure() {
>   -DENABLE_CLI=OFF
>   -DMAIN12=ON
>   )
> - if [[ ${ABI} = x86 ]] ; then
> - mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> )
> - fi
> - if [[ ${ABI} = arm ]] ; then
> - # 589674
> - mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> )
> - fi
> - if [[ ${ABI} = ppc64 ]] ; then
> - #
> https://bugs.gentoo.org/show_bug.cgi?id=607802#c5
> - mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> -DENABLE_ALTIVEC=OFF )
> - fi
>   ;;
>   "main10")
>   mycmakeargs+=(
> @@ -104,17 +95,6 @@ x265_variant_src_configure() {
>   -DENABLE_SHARED=OFF
>   -DENABLE_CLI=OFF
>   )
> - if [[ ${ABI} = x86 ]] ; then
> - mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> )
> - fi
> - if [[ ${ABI} = arm ]] ; then
> - # 589674
> - mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> )
> - fi
> - if [[ ${ABI} = ppc64 ]] ; then
> - #
> https://bugs.gentoo.org/show_bug.cgi?id=607802#c5
> - mycmakeargs+=( -DENABLE_ASSEMBLY=OFF
> -DENABLE_ALTIVEC=OFF )
> - fi
>   ;;


What are you trying to fix here ?
This sounds like a regression to me: some asm will not work for
10/12bits variants while 8bit is fine. You are removing this work here.



>   "main")
>   if (( "${#MULTIBUILD_VARIANTS[@]}" > 1 )) ;
> then @@ -146,7 +126,6 @@ multilib_src_configure() {
>   append-cxxflags -fPIC
>  
>   local myabicmakeargs=(
> - -DENABLE_TESTS=$(usex test ON OFF)
>   $(multilib_is_native_abi || echo "-DENABLE_CLI=OFF")
>   -DENABLE_LIBNUMA=$(usex numa ON OFF)
>   -DCPU_POWER8=$(usex power8 ON OFF)
> @@ -154,18 +133,39 @@ multilib_src_configure() {
>   -DLIB_INSTALL_DIR="$(get_libdir)"
>   )
>  
> + local supports_asm=yes
> +
>   if [[ ${ABI} = x86 ]] ; then
> - # Bug #528202
> - if use pic ; then
> + if use asm && use pic ; then
> + # Bug #528202
>   ewarn "PIC has been requested but x86 asm is
> not 

[gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Sun, 24 May 2020 20:25:11 + (UTC)
"Thomas Deutschmann"  wrote:

> commit: 6e149596cc76f1bbcee6720828c8c8c92420f2a3
> Author: Thomas Deutschmann  gentoo  org>
> AuthorDate: Sun May 24 19:47:08 2020 +
> Commit: Thomas Deutschmann  gentoo  org>
> CommitDate: Sun May 24 20:23:53 2020 +
> URL:
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> 
> media-libs/x265: drop USE=pic
> 
> Gentoo's toolchain uses PIC by default. Since USE=asm was added,
> we no longer need a USE flag to control that behavior.


You got it wrong here it seems: USE=pic does not control whether
the toolchain produces PIC or not. Shared libs always are, and have
always been, built that way on Gentoo.
In this case, USE=pic means "no matter what it costs, I do not want
textrels", for the cases of hand written assembly that has to be
rewritten to support PIC. And, still in this case, this costs a lot of
performance, so it is enabled by default on hardened profiles and not
others.
Textrels work fine (on some architectures), they disallow W^X and force
each process using the shared lib to make a "copy" at runtime in order
to resolve relocations, so are not desirable but sometimes the cost
outweights the gain.

Plus, profiles/features/hardened enables pic by default but knows
nothing about USE=asm so this is a regression for them.


Alexis.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Mon, 25 May 2020 11:26:26 -0400
Mike Gilbert  wrote:

> On Mon, May 25, 2020 at 9:13 AM Alexis Ballier 
> wrote:
> >
> > On Sun, 24 May 2020 20:25:11 + (UTC)
> > "Thomas Deutschmann"  wrote:
> >
> > > commit: 6e149596cc76f1bbcee6720828c8c8c92420f2a3
> > > Author: Thomas Deutschmann  gentoo  org>
> > > AuthorDate: Sun May 24 19:47:08 2020 +
> > > Commit: Thomas Deutschmann  gentoo  org>
> > > CommitDate: Sun May 24 20:23:53 2020 +
> > > URL:
> > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> > >
> > > media-libs/x265: drop USE=pic
> > >
> > > Gentoo's toolchain uses PIC by default. Since USE=asm was added,
> > > we no longer need a USE flag to control that behavior.
> >
> >
> > You got it wrong here it seems: USE=pic does not control whether
> > the toolchain produces PIC or not. Shared libs always are, and have
> > always been, built that way on Gentoo.
> > In this case, USE=pic means "no matter what it costs, I do not want
> > textrels", for the cases of hand written assembly that has to be
> > rewritten to support PIC. And, still in this case, this costs a lot
> > of performance, so it is enabled by default on hardened profiles
> > and not others.
> > Textrels work fine (on some architectures), they disallow W^X and
> > force each process using the shared lib to make a "copy" at runtime
> > in order to resolve relocations, so are not desirable but sometimes
> > the cost outweights the gain.
> >
> > Plus, profiles/features/hardened enables pic by default but knows
> > nothing about USE=asm so this is a regression for them.
> 
> The USE flag toggles use of assembly, not use of PIC. The default USE
> value in the hardened profile should not drive decisions on what we
> name USE flags.

... but using a global well documented useflag instead of a local
invention should drive such decisions.

> 
> You can add the flag to package.use or package.use.mask in the
> hardened profile if necessary.


Sure but it was not added there and it's not really relevant here since
 USE=asm does not make any sense either. (aka: this was better
 before this series of commits)

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Mon, 2020-05-25 at 17:04 -0400, Mike Gilbert wrote:
> On Mon, May 25, 2020 at 3:18 PM Michał Górny 
> wrote:
> > On Mon, 2020-05-25 at 19:49 +0200, Alexis Ballier wrote:
> > > On Mon, 25 May 2020 11:26:26 -0400
> > > Mike Gilbert  wrote:
> > > 
> > > > On Mon, May 25, 2020 at 9:13 AM Alexis Ballier <
> > > > aball...@gentoo.org>
> > > > wrote:
> > > > > On Sun, 24 May 2020 20:25:11 + (UTC)
> > > > > "Thomas Deutschmann"  wrote:
> > > > > 
> > > > > > commit: 6e149596cc76f1bbcee6720828c8c8c92420f2a3
> > > > > > Author: Thomas Deutschmann  gentoo 
> > > > > > org>
> > > > > > AuthorDate: Sun May 24 19:47:08 2020 +
> > > > > > Commit: Thomas Deutschmann  gentoo 
> > > > > > org>
> > > > > > CommitDate: Sun May 24 20:23:53 2020 +
> > > > > > URL:
> > > > > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> > > > > > 
> > > > > > media-libs/x265: drop USE=pic
> > > > > > 
> > > > > > Gentoo's toolchain uses PIC by default. Since USE=asm was
> > > > > > added,
> > > > > > we no longer need a USE flag to control that behavior.
> > > > > 
> > > > > You got it wrong here it seems: USE=pic does not control
> > > > > whether
> > > > > the toolchain produces PIC or not. Shared libs always are,
> > > > > and have
> > > > > always been, built that way on Gentoo.
> > > > > In this case, USE=pic means "no matter what it costs, I do
> > > > > not want
> > > > > textrels", for the cases of hand written assembly that has to
> > > > > be
> > > > > rewritten to support PIC. And, still in this case, this costs
> > > > > a lot
> > > > > of performance, so it is enabled by default on hardened
> > > > > profiles
> > > > > and not others.
> > > > > Textrels work fine (on some architectures), they disallow W^X
> > > > > and
> > > > > force each process using the shared lib to make a "copy" at
> > > > > runtime
> > > > > in order to resolve relocations, so are not desirable but
> > > > > sometimes
> > > > > the cost outweights the gain.
> > > > > 
> > > > > Plus, profiles/features/hardened enables pic by default but
> > > > > knows
> > > > > nothing about USE=asm so this is a regression for them.
> > > > 
> > > > The USE flag toggles use of assembly, not use of PIC. The
> > > > default USE
> > > > value in the hardened profile should not drive decisions on
> > > > what we
> > > > name USE flags.
> > > 
> > > ... but using a global well documented useflag instead of a local
> > > invention should drive such decisions.
> > 
> > What 'global well documented useflag'?
> 
> It's neither global, nor well-documented, but several packages do
> define it locally.
> 

https://gitweb.gentoo.org/repo/gentoo/historical.git/commit/profiles/use.desc?id=103236c295aa30e5e42cfc8a7429e4eea5f0d680

https://gitweb.gentoo.org/repo/gentoo/historical.git/commit/profiles/use.desc?id=784deb7134b9d430546557a8f8a0877bf35c02ba

I guess this hasn't been really discussed back then.

It is also used in a global way in profiles (make.defaults).

> Personally, I think it should be renamed to "asm" or something
> similar
> in the majority of cases where it actually disables all use of
> assembly code.

Thankfully these days there's usually no need to disable asm to have
pic. hardened has no mention of that flag, and I think that e.g. for
openssl they would have noticed long ago.
And again, 'asm' as a useflag makes no sense: if it works and simply
replaces a C function by a faster one then it shouldn't even be an
useflag. 'pic' on the other hand conveys the tradeoff idea.

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: media-libs/x265/

[Alexis Ballier]

On Mon, 2020-05-25 at 21:09 -0400, Mike Gilbert wrote:
> On Mon, May 25, 2020 at 7:35 PM Alexis Ballier 
> wrote:
> > On Mon, 2020-05-25 at 17:04 -0400, Mike Gilbert wrote:
> > > On Mon, May 25, 2020 at 3:18 PM Michał Górny 
> > > wrote:
> > > > On Mon, 2020-05-25 at 19:49 +0200, Alexis Ballier wrote:
> > > > > On Mon, 25 May 2020 11:26:26 -0400
> > > > > Mike Gilbert  wrote:
> > > > > 
> > > > > > On Mon, May 25, 2020 at 9:13 AM Alexis Ballier <
> > > > > > aball...@gentoo.org>
> > > > > > wrote:
> > > > > > > On Sun, 24 May 2020 20:25:11 + (UTC)
> > > > > > > "Thomas Deutschmann"  wrote:
> > > > > > > 
> > > > > > > > commit: 6e149596cc76f1bbcee6720828c8c8c92420f2a3
> > > > > > > > Author: Thomas Deutschmann  gentoo
> > > > > > > > 
> > > > > > > > org>
> > > > > > > > AuthorDate: Sun May 24 19:47:08 2020 +
> > > > > > > > Commit: Thomas Deutschmann  gentoo
> > > > > > > > 
> > > > > > > > org>
> > > > > > > > CommitDate: Sun May 24 20:23:53 2020 +
> > > > > > > > URL:
> > > > > > > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6e149596
> > > > > > > > 
> > > > > > > > media-libs/x265: drop USE=pic
> > > > > > > > 
> > > > > > > > Gentoo's toolchain uses PIC by default. Since USE=asm
> > > > > > > > was
> > > > > > > > added,
> > > > > > > > we no longer need a USE flag to control that behavior.
> > > > > > > 
> > > > > > > You got it wrong here it seems: USE=pic does not control
> > > > > > > whether
> > > > > > > the toolchain produces PIC or not. Shared libs always
> > > > > > > are,
> > > > > > > and have
> > > > > > > always been, built that way on Gentoo.
> > > > > > > In this case, USE=pic means "no matter what it costs, I
> > > > > > > do
> > > > > > > not want
> > > > > > > textrels", for the cases of hand written assembly that
> > > > > > > has to
> > > > > > > be
> > > > > > > rewritten to support PIC. And, still in this case, this
> > > > > > > costs
> > > > > > > a lot
> > > > > > > of performance, so it is enabled by default on hardened
> > > > > > > profiles
> > > > > > > and not others.
> > > > > > > Textrels work fine (on some architectures), they disallow
> > > > > > > W^X
> > > > > > > and
> > > > > > > force each process using the shared lib to make a "copy"
> > > > > > > at
> > > > > > > runtime
> > > > > > > in order to resolve relocations, so are not desirable but
> > > > > > > sometimes
> > > > > > > the cost outweights the gain.
> > > > > > > 
> > > > > > > Plus, profiles/features/hardened enables pic by default
> > > > > > > but
> > > > > > > knows
> > > > > > > nothing about USE=asm so this is a regression for them.
> > > > > > 
> > > > > > The USE flag toggles use of assembly, not use of PIC. The
> > > > > > default USE
> > > > > > value in the hardened profile should not drive decisions on
> > > > > > what we
> > > > > > name USE flags.
> > > > > 
> > > > > ... but using a global well documented useflag instead of a
> > > > > local
> > > > > invention should drive such decisions.
> > > > 
> > > > What 'global well documented useflag'?
> > > 
> > > It's neither global, nor well-documented, but several packages do
> > > define it locally.
> > > 
> > 
> > https://gitweb.gentoo.org/repo/gentoo/historical.git/commit/profiles/use.desc?id=103236c295aa30e5e42cfc8a7429e4eea5f0d680
> > 
> > https://gitweb.gentoo.org/repo/gentoo/historical.git/commit/profiles/use.desc?id=784deb7134b9d430546557a8f8a0877bf35c02ba
> > 
> > I guess this hasn't been really discussed back then.
> > 
> > It is also used in a global way in profiles (make.defaults).
> > 
> > > Personally, I think it should be renamed to "asm" or something
> > > similar
> > > in the majority of cases where it actually disables all use of
> > > assembly code.
> > 
> > Thankfully these days there's usually no need to disable asm to
> > have
> > pic. hardened has no mention of that flag, and I think that e.g.
> > for
> > openssl they would have noticed long ago.
> > And again, 'asm' as a useflag makes no sense: if it works and
> > simply
> > replaces a C function by a faster one then it shouldn't even be an
> > useflag. 'pic' on the other hand conveys the tradeoff idea.
> 
> If I understand you correctly, we should just drop the USE="pic"
> logic
> from the remaining packages that have it? Or are you trying to say
> something else?


Drop USE=asm unless there's a real reason to it: Such a useflag is,
IMHO, at the same level of a useflag on dev-lang/python that would
toggle dict's underlying implementations but not the semantics of the
language.
Have USE=pic for its historical meaning, aka, sacrificing everything to
have PIC shared libs because your system enforces this (pax).

Note that having the 'pic' useflag should be considered something to be
fixed: rewrite the asm in a PIC way. But these days nobody has the will
to do it since this is mostly an issue on x86+pax, both being slowly
decreasing.