[gentoo-portage-dev] Re: [PATCH v2] emerge: auto-enable --with-bdeps if --usepkg is not enabled (bug 598444)

[Duncan]

Michael Orlitzky posted on Sun, 05 Mar 2017 14:44:58 -0500 as excerpted:

> On 03/05/2017 02:12 PM, Zac Medico wrote:
>> 
>> Incorrect.
>> 
>> ...
>> 
>> Incorrect.
>> 
>> 
> I see my mistakes, but maintain that this is confusing =)

For the record, I think it's /somewhat/ confusing too, and would prefer 
your two options... except for the backward compatibility thing, which 
throws a serious wrench into things if we end up keeping the existing --
with-bdeps option even for a deprecation period, and throws an entirely 
different wrench into things if we simply ignore backward compatibility 
and remove it without a deprecation period.

Which leaves me at a loss as to which option would be better, killing 
backward compatibility for an arguably clearer pair of options, or 
staying with backward compatibility and Zac's confusing, but perhaps the 
best that can be done given the existing option and backward 
compatibility, pair of options.

Tho for me personally, I've been --with-bdeps=y all the way, since 
original introduction, and Zac's changes would affect that at all, tho of 
course I'd have to adjust due to loss of backward compatibility if mjo's 
options were taken.

And there's already portage precedent for "I don't want to have to care, 
just make it do the right thing unless I tell it otherwise", in other 
areas, so I think --with-bdeps-auto= seems to be most consistent with the 
existing pattern, which means, confusing tho it may be, it /does/ result 
in most people not needing to care, so in the end, even tho I agree it's 
definitely more complex than I'd wish, I'll have to lean Zac's way on 
this one.

Which effectively surprises even me.  I started this post expecting I was 
going to agree with mjo, but ended up talking myself out of it. 

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-portage-dev] [PATCH v2] emerge: auto-enable --with-bdeps if --usepkg is not enabled (bug 598444)

[Zac Medico]

On 03/05/2017 11:44 AM, Michael Orlitzky wrote:
> On 03/05/2017 02:12 PM, Zac Medico wrote:
>>
>> Incorrect.
>>
>> ...
>>
>> Incorrect.
>>
> 
> I see my mistakes, but maintain that this is confusing =)

It could be worse, and I think it's questionable that we could do any
better, given the nature of the problem.

>> The --with-bdeps-auto option results in desirable behavior by default,
>> and it's also backward compatible with existing --with-bdeps and
>> --usepkg usage. It just does the right thing, minimizing the impact to
>> existing emerge usage.
> 
> I was hoping that since nothing breaks with --update-bdeps=y and
> --clean-bdeps=n (the new defaults), we could just disable --with-bdeps
> immediately and emit a warning when it's given.

Breaking backward compatibility is too disruptive, unless we allow for a
transition period where --with-bdeps is deprecated.

>> There some problems with --update-bdeps/--clean-bdeps:
>>
>> * The program logic will be more complicated, since --with-bdeps will
>> have to override both options for backward-compatibility with existing
>> --with-bdeps usage.
> 
> Not applicable if --with-bdeps goes away.

We don't have a justification to break compatibility without a
transition period.

>> * The meaning of --clean-bdeps is confusing. Does --clean-bdeps=y mean
>> to clean build time deps, or does it mean to pull build time deps into
>> the dependency graph for removal operations?
> 
> --clean-bdeps means clean the bdeps.

It's confusing to have an option with inverted meaning relative to
--with-bdeps. Cleaning the bdeps is a consequence of excluding them from
the dependency graph, not an action in itself. --clean-bdeps sounds like
an action.

> I totally agree that the worst option of all is to keep --with-bdeps AND
> introduce the other two.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH v2] emerge: auto-enable --with-bdeps if --usepkg is not enabled (bug 598444)

[Michael Orlitzky]

On 03/05/2017 02:12 PM, Zac Medico wrote:
> 
> Incorrect.
> 
> ...
> 
> Incorrect.
> 

I see my mistakes, but maintain that this is confusing =)


> 
> The --with-bdeps-auto option results in desirable behavior by default,
> and it's also backward compatible with existing --with-bdeps and
> --usepkg usage. It just does the right thing, minimizing the impact to
> existing emerge usage.

I was hoping that since nothing breaks with --update-bdeps=y and
--clean-bdeps=n (the new defaults), we could just disable --with-bdeps
immediately and emit a warning when it's given.


> 
> There some problems with --update-bdeps/--clean-bdeps:
> 
> * The program logic will be more complicated, since --with-bdeps will
> have to override both options for backward-compatibility with existing
> --with-bdeps usage.

Not applicable if --with-bdeps goes away.


> * The meaning of --clean-bdeps is confusing. Does --clean-bdeps=y mean
> to clean build time deps, or does it mean to pull build time deps into
> the dependency graph for removal operations?

--clean-bdeps means clean the bdeps.


I totally agree that the worst option of all is to keep --with-bdeps AND
introduce the other two.

Re: [gentoo-portage-dev] [PATCH v2] emerge: auto-enable --with-bdeps if --usepkg is not enabled (bug 598444)

[Zac Medico]

On 03/05/2017 07:08 AM, Michael Orlitzky wrote:
> On 03/05/2017 03:40 AM, Zac Medico wrote:
>>
>> A new --with-bdeps-auto= option is provided, making it possible to
>> enable or disable the program logic that causes --with-bdeps to be
>> automatically enabled. Use --with-bdeps-auto=n to prevent --with-bdeps
>> from being automatically enabled for installation actions. This is useful
>> for some rare cases in which --with-bdeps triggers unsolvable dependency
>> conflicts (and putting --with-bdeps=n in EMERGE_DEFAULT_OPTS would cause
>> undesirable --depclean behavior).
>>
> 
> If I understand correctly, the end result of this is two --flags that
> combine in a (rather complicated) way to let the user control the
> default bdeps behavior of both "emerge --update" and "emerge --depclean"
> separately. I'll try to summarize my understanding:
> 
>I. emerge --update
> 
>  I.a. Some people want to --update the build dependencies they have
>   installed for e.g. security purposes.
> 
>  I.b. Others don't want build deps pulled in by "emerge --update",
>   because they're using binary packages or because it causes
>   conflicts in the resolver (llvm).
> 
>   II. emerge --depclean
> 
> II.a. The default behavior is to not remove build-only dependencies
>   because, generally, they will just have to rebuilt again.
> 
> II.b. However, some people want to remove the build-only deps that
>   aren't strictly in use -- particularly if those build-only
>   deps are not being updated by emerge --update.
> 
> 
> To choose between those four options...
> 
>   i.   --with-bdeps=n and --with-bdeps-auto=y gives you I.a. + II.b.

Incorrect.

These options give you I.b. + II.b. because --with-bdeps=n overrides
--with-bdeps-auto settings (the override is backward-compatible with
existing --with-bdeps usage).

Since I.a. + II.b. is a contradictory combination, there is no
combination of EMERGE_DEFAULT_OPTS that will give you I.a. + II.b.

Specifying --with-bdeps-auto=y has no effect, since that's the default.

>   ii.  --with-bdeps=n and --with-bdeps-auto=n gives you I.b. + II.b.

Correct.

Specifying --with-bdeps-auto=n has no effect, since --with-bdeps=n
overrides --with-bdeps-auto settings (the override is
backward-compatible with existing --with-bdeps usage).

>   iii. --with-bdeps=y and --with-bdeps-auto=y gives you I.a. + II.a.

Correct.

Specifying --with-bdeps-auto=y has no effect for 2 reasons:

* --with-bdeps-auto=y is the default
* --with-bdeps=y overrides --with-bdeps-auto settings (the override is
backward-compatible with existing --with-bdeps usage).

>   iv.  --with-bdeps=y and --with-bdeps-auto=n gives you I.a. + II.a.

Correct.

Specifying --with-bdeps-auto=n has no effect, since --with-bdeps=y
overrides --with-bdeps-auto settings (the override is
backward-compatible with existing --with-bdeps usage).

> That only gets you three out of the four options. You have to read the
> fine print to get the other:
> 
>   v.   passing no flags explicitly gives you I.b. and II.a.

Incorrect.

Since --with-bdeps-auto=y is the the default setting, you get I.a. +
II.a. unless --usepkg is enabled. If --usepkg is enabled, you get I.b. +
II.a., which is backward-compatible with existing --usepkg usage.

> If there's going to be two flags, can't we do better? Why not just name
> the flags after what we want them to do:
> 
>   --update-bdeps= (default: y)
> 
>   --clean-bdeps=  (default: n)

The --with-bdeps-auto option results in desirable behavior by default,
and it's also backward compatible with existing --with-bdeps and
--usepkg usage. It just does the right thing, minimizing the impact to
existing emerge usage.

There some problems with --update-bdeps/--clean-bdeps:

* The program logic will be more complicated, since --with-bdeps will
have to override both options for backward-compatibility with existing
--with-bdeps usage.

* The meaning of --clean-bdeps is confusing. Does --clean-bdeps=y mean
to clean build time deps, or does it mean to pull build time deps into
the dependency graph for removal operations?

> With those two options, it's easy to tell portage exactly what you want.
> If I don't like the defaults, I can turn one of them off without
> affecting the other.

No, there are problems, as described above.

> But what about the --usepkg magic? I think the workaround can be left
> as-is. When --usepkg is enabled, switch the default for --update-bdeps
> to "n" unless it is explicitly set.

That would work, but introducing 2 new options is no really appealing,
because it will lead to more complicated logic that will be more
difficult to explain and for people to comprehend. I really don't want
to spend time documenting the interactions between --with-bdeps,
--update-bdes, and --clean-bdeps, and then get a ton of frequently asked
questions from people having difficulty understanding it.

Overall, I think the --with-bdeps-auto approach seems 

Re: [gentoo-portage-dev] [PATCH v2] emerge: auto-enable --with-bdeps if --usepkg is not enabled (bug 598444)

[Michael Orlitzky]

On 03/05/2017 03:40 AM, Zac Medico wrote:
> 
> A new --with-bdeps-auto= option is provided, making it possible to
> enable or disable the program logic that causes --with-bdeps to be
> automatically enabled. Use --with-bdeps-auto=n to prevent --with-bdeps
> from being automatically enabled for installation actions. This is useful
> for some rare cases in which --with-bdeps triggers unsolvable dependency
> conflicts (and putting --with-bdeps=n in EMERGE_DEFAULT_OPTS would cause
> undesirable --depclean behavior).
> 

If I understand correctly, the end result of this is two --flags that
combine in a (rather complicated) way to let the user control the
default bdeps behavior of both "emerge --update" and "emerge --depclean"
separately. I'll try to summarize my understanding:

   I. emerge --update

 I.a. Some people want to --update the build dependencies they have
  installed for e.g. security purposes.

 I.b. Others don't want build deps pulled in by "emerge --update",
  because they're using binary packages or because it causes
  conflicts in the resolver (llvm).

  II. emerge --depclean

II.a. The default behavior is to not remove build-only dependencies
  because, generally, they will just have to rebuilt again.

II.b. However, some people want to remove the build-only deps that
  aren't strictly in use -- particularly if those build-only
  deps are not being updated by emerge --update.


To choose between those four options...

  i.   --with-bdeps=n and --with-bdeps-auto=y gives you I.a. + II.b.

  ii.  --with-bdeps=n and --with-bdeps-auto=n gives you I.b. + II.b.

  iii. --with-bdeps=y and --with-bdeps-auto=y gives you I.a. + II.a.

  iv.  --with-bdeps=y and --with-bdeps-auto=n gives you I.a. + II.a.

That only gets you three out of the four options. You have to read the
fine print to get the other:

  v.   passing no flags explicitly gives you I.b. and II.a.

If there's going to be two flags, can't we do better? Why not just name
the flags after what we want them to do:

  --update-bdeps= (default: y)

  --clean-bdeps=  (default: n)

With those two options, it's easy to tell portage exactly what you want.
If I don't like the defaults, I can turn one of them off without
affecting the other.

But what about the --usepkg magic? I think the workaround can be left
as-is. When --usepkg is enabled, switch the default for --update-bdeps
to "n" unless it is explicitly set.

Food for thought. Anyway, thanks for working on this.

[gentoo-portage-dev] [PATCH v2] emerge: auto-enable --with-bdeps if --usepkg is not enabled (bug 598444)

[Zac Medico]

It's useful to automatically enable --with-bdeps so that @world updates
will update all packages that are not eligible for removal by
emerge --depclean. However, many users of binary packages do not want
unnecessary build time dependencies installed, therefore do not
auto-enable --with-bdeps for installation actions when the --usepkg
option is enabled.

A new --with-bdeps-auto= option is provided, making it possible to
enable or disable the program logic that causes --with-bdeps to be
automatically enabled. Use --with-bdeps-auto=n to prevent --with-bdeps
from being automatically enabled for installation actions. This is useful
for some rare cases in which --with-bdeps triggers unsolvable dependency
conflicts (and putting --with-bdeps=n in EMERGE_DEFAULT_OPTS would cause
undesirable --depclean behavior).

X-Gentoo-bug: 598444
X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=598444
---
[PATCH v2] adds a --with-bdeps-auto= option which is useful for some
rare cases in which --with-bdeps triggers unsolvable dependency conflicts

 man/emerge.1 |  37 +++-
 pym/_emerge/create_depgraph_params.py|   5 +
 pym/_emerge/depgraph.py  |   4 +-
 pym/_emerge/main.py  |   5 +
 pym/portage/tests/resolver/ResolverPlayground.py |   5 +
 pym/portage/tests/resolver/test_bdeps.py | 215 +++
 6 files changed, 266 insertions(+), 5 deletions(-)
 create mode 100644 pym/portage/tests/resolver/test_bdeps.py

diff --git a/man/emerge.1 b/man/emerge.1
index 5b61220..7db4271 100644
--- a/man/emerge.1
+++ b/man/emerge.1
@@ -986,13 +986,44 @@ The default is set to "y" (on).
 .TP
 .BR "\-\-with\-bdeps < y | n >"
 In dependency calculations, pull in build time dependencies
-that are not strictly required. This defaults to \'n\' for
-installation actions, meaning they will not be installed, and
-\'y\' for the \fB\-\-depclean\fR action, meaning they will not be removed.
+that are not strictly required. This option is automatically enabled for
+installation actions, meaning they will be installed, and defaults to
+\(aqy\(aq for the \fB\-\-depclean\fR action, meaning they will not be
+removed. In order to prevent the \fB\-\-with\-bdeps\fR option from being
+automatically enabled for installation actions, specify
+\fB\-\-with\-bdeps\-auto=n\fR in either the command line or
+\fBEMERGE_DEFAULT_OPTS\fR.
+
+Since many users of binary packages do not want unnecessary build time
+dependencies installed, this option is not automatically enabled for
+installation actions when the \fB\-\-usepkg\fR option is enabled. In
+order to pull in build time dependencies for binary packages with
+\fB\-\-usepkg\fR, \fB\-\-with\-bdeps=y\fR must be specified explicitly.
+This also applies to options that enable the \fB\-\-usepkg\fR option
+implicitly, such as \fB\-\-getbinpkg\fR.
+
 This setting can be added to
 \fBEMERGE_DEFAULT_OPTS\fR (see make.conf(5)) and later overridden via the
 command line.
 .TP
+.BR "\-\-with\-bdeps\-auto < y | n >"
+This option is used to enable or disable the program logic that causes
+\fB\-\-with\-bdeps\fR is to be automatically enabled for installation
+actions. This option is enabled by default. Use
+\fB\-\-with\-bdeps\-auto=n\fR to prevent \fB\-\-with\-bdeps\fR from
+being automatically enabled for installation actions. This setting can
+be added to \fBEMERGE_DEFAULT_OPTS\fR (see make.conf(5)) and later
+overridden via the command line.
+
+\fBNOTE:\fR The program logic that causes \fB\-\-with\-bdeps\fR to be
+automatically enabled for installation actions does not affect removal
+actions such as the \fB\-\-depclean\fR action. Therefore, when
+\fB\-\-with\-bdeps\-auto=n\fR is specified in \fBEMERGE_DEFAULT_OPTS\fR,
+it does not affect the default \fB\-\-with\-bdeps=y\fR setting that
+applies to the \fB\-\-depclean\fR action. The default
+\fB\-\-with\-bdeps=y\fR setting that applies to the \fB\-\-depclean\fR
+action can be overridden only by specifying \fB\-\-with\-bdeps=n\fR.
+.TP
 .BR "\-\-with\-test\-deps [ y | n ]"
 For packages matched by arguments, this option will pull in dependencies
 that are conditional on the "test" USE flag, even if "test" is not
diff --git a/pym/_emerge/create_depgraph_params.py 
b/pym/_emerge/create_depgraph_params.py
index 2c64928..cdea029 100644
--- a/pym/_emerge/create_depgraph_params.py
+++ b/pym/_emerge/create_depgraph_params.py
@@ -13,6 +13,8 @@ def create_depgraph_params(myopts, myaction):
# deep:  go into the dependencies of already merged packages
# empty: pretend nothing is merged
# complete:  completely account for all known dependencies
+   # bdeps: satisfy build time dependencies of packages that are
+   #   already built, even though they are not strictly required
# remove:build graph for use in removing packages
# rebuilt_binaries: replace installed packages with rebuilt binaries