Hello, For a long time we seem to be missing appropriate tools to handle USE flag constraints efficiently. EAPI 4 brought REQUIRED_USE but all things considered, it has proven to be far from an optimal solution. I would therefore like to discuss adding a better tool to amend or replace it, to allow for automated handling of USE flag constraints.
1. Motivation ============= [[This section is purely informational. It is not up for discussion. If you really can't help yourself and need to discuss superiority of example A over example B, then please have mercy and wait for a few days to give developers a chance to stay focused on the proposal at hand.]] EAPI 4 provides REQUIRED_USE as a simple tool to enforce USE flag constraints for packages. It can be used to express dependencies between flags (e.g. A requires B). However, it poses a few problems which results in developers being reluctant to use them and which limit their use cases. Most notably: 1. They require an explicit action from the user. A large number of USE flag constraints can be frustrating, especially when most of them are trivial to solve. 2. They can cause a significant clutter in package.use. As a result, changing past decisions becomes hard as the user may need to find past USE flag changes and revert them. 3. The constraint output is sometimes considered hard to comprehend. Those issues have resulted in developers sometimes avoiding REQUIRED_USE in favor of other solutions such as implicit behavior (ignoring USE flags), additional control flags (think PYTHON_SINGLE_TARGET) or pkg_pretend() (which is horribly slow). I think that the problems are most visible when dealing with 'provider flags', that is USE flags that are used to choose between different libraries providing the same feature. There are three basic approaches to the problem: A. Binary provider flags (+ feature flags), e.g.: USE='[ssl] libressl' -> use LibreSSL USE='[ssl] -libressl' -> use OpenSSL USE='-ssl' -> disable SSL/TLS (USE=libressl ignored) B. Unary provider flags with REQUIRED_USE: USE='libressl -openssl' -> use LibreSSL USE='-libressl openssl' -> use OpenSSL USE='libressl openssl' -> invalid USE='-libressl -openssl' -> disable SSL/TLS or invalid C. Unary provider flags without REQUIRED_USE: USE='libressl -openssl' -> use LibreSSL USE='-libressl openssl' -> use OpenSSL USE='libressl openssl' -> use LibreSSL or OpenSSL, depending on pkg USE='-libressl -openssl' -> disable SSL/TLS or as above Sometimes solutions B and C are amended with additional feature flags to reduce the confusion. Each of these solutions has its disadvantages. What's worse, they are sometimes not even used consistently within a single ecosystem -- going as far as using the same USE flag with slightly different meaning. The Gentoo developer attitude 'my package, my way' does not help here. Solving all of the listed problems is not possible with the existing tools as each of the possible solutions has its own issues, and it is impossible to get developers to agree on one of them. 2. Specification ================ 2.1. Forced USE flag constraints ================================ This proposal introduces USE flag constraints that are meant to be solved automatically by the package manager. This is somewhat similar to REQUIRED_USE, except that it does not require the user to take any explicit action. In that regard, it is closer to conditional package.use.force/mask. In the basic form, it can be used to conditionally force a specific flag to be enabled or disabled. For example: foo? ( bar ) would mean that the bar flag is implicitly enabled (forced) if foo is enabled as well. Appropriately: foo? ( !bar ) would mean that the bar flag is implicitly disabled (masked) in that case. It can also be used with multi-flag ??, ^^ and || constraints, i.e.: - ?? means that at most one of the flags can be enabled. If user configuration causes more than one of the flags to be enabled, additional flags are implicitly disabled (masked) to satisfy the constraint. - || means that at least one of the flags must be enabled. If user configuration causes none of the flags to be enabled, one of them is enabled implicitly (forced). - ^^ means that exactly one of the flags must be enabled. The behavior is a combination of both above constraints. The automated solving of USE constraints would require the developers to consider the implicit effect of the constraints they are writing. 2.2. UI implications ==================== The automated flag handling would require an appropriate UI changes, to clearly indicate whenever the flags are being altered automatically. Otherwise, users might be confused of their requested USE flags not being applied correctly. In the most basic form, a solution similar to use.force/use.mask can be used, e.g.: USE="[bar] [-baz] foo" with square brackets indicating that the flags have been forced/masked by other USE flags on the list. It would also be necessary for the package manager to provide more verbose information on USE flag constraints being applied, either implicitly or on request. For simple constraints, the visual presentation can be even included in place, e.g.: USE="foo -> [bar -baz]" indicating that both constraints were forced by the flag 'foo' (i.e. that disabling 'foo' would loosen them). For multi-flag constraints (??, ^^, ||), the package manager could even go as far to group the flags together with explicit constraint notation, e.g.: USE="^^ [ foo -bar -baz ]" The exact UI design is left to the package manager developers. Those are merely proof-of-concept ideas. 2.3. Configuration bits ======================= In its most basic form, this proposal does not require explicit configuration -- all flags are applied automatically. However, a few bits might be useful. Firstly, the package manager might provide a way to explicitly forbid switching specific flags or automatically applying the constraints completely. If the user uses that, it will just fail in the same way as REQUIRED_USE fails now. Secondly, it might be reasonable to provide configurable priorities for solving multi-flag constraints. For example, we could use rightmost- preferred logic for package.use, e.g.: */* PROVIDER_SSL: openssl gnutls dev-util/foo PROVIDER_SSL: polarssl which would mean that for all packages, gnutls is preferred over openssl (i.e. if ?? or ^^ applies, openssl will be disabled and gnutls will be used), and polarssl is additionally preferred over everything else for dev-util/foo. 2.4. Backwards compatibility ============================ Before choosing the exact way of implementing this, we need to answer how far we want to provide backwards compatibility. In particular: a. Do we need the REQUIRED_USE requiring explicit user selection? Or can we rely completely on other solutions (automatic solving, pkg_pretend())? b. Would changing REQUIRED_USE behavior in place cause unintended side effects? Are we going to accept those effects until the packages are fixed? (e.g. selecting less preferred solution of constraint) If we can agree on doing the change in place (retroactively), I think we can start solving a few major issues. For example, I would be really happy to kill PYTHON_SINGLE_TARGET flags in favor of constraints on PYTHON_TARGETS being solved automatically. We could also look into unifying provider flags into PROVIDER_SSL, PROVIDER_AV etc. I should point out that strictly speaking, PMS does not define any specific handling of REQUIRED_USE. It only defines that the ebuild must not be used if the constraints are not met: | If the package manager encounters a package version where REQUIRED_USE | assertions are not met, it must treat this package version as if it | was masked. No phase functions must be called. In other words, it does not define whether the package manager should request the user to explicitly solve the problem or whether it should be solved automatically. However, from past discussions I recall that the original intent of REQUIRED_USE was for machine processing, and so I would consider it reasonable to start using it. 2.5. Ebuild interface ===================== This section is most flexible as I don't really care how it's done, as long as it's done. Depending on our preferences, and the answers to questions in the preceding section, the options include: 1. changing behavior of REQUIRED_USE retroactively -- i.e. making Portage start solving USE constraints automatically, and being able to rely on that a few months from now, 2. changing behavior of REQUIRED_USE in a future EAPI, 3. adding ENFORCED_USE with the new behavior in a future EAPI (and either keeping or removing REQUIRED_USE), 4. adding special syntax to REQUIRED_USE to indicate which constraints can be solved automatically, 5. adding special syntax to IUSE to indicate which flags can be enabled or disabled automatically via REQUIRED_USE (a little different from the exact proposal), 6. adding a dedicated variable to indicate USE_EXPAND sets that can be solved automatically via REQUIRED_USE (partial solution). 3. Rationale ============ I believe this solution to be quite optimal. It solves most of the issues with the REQUIRED_USE, making it a feasible solution to many of the existing problems, including those that were not even worth considering with the current REQUIRED_USE semantics. Reusing the existing syntax has the advantage of lowering the learning curve and simplifying the specification. We can even enable the new semantics without changing the PMS (provided that variant 1. is chosen), or with as little duplication as possible. However, the developers will be required to start considering the implicit effects of their constraints (as a matter of policy). Automatically solving USE constraints solve all three fore-mentioned issues with REQUIRED_USE. By default, no user intervention is required to solve USE constraints and package.use needs to be modified only to enforce a non-standard solutons. While the 'readability of output' is not really changed per se, most of the time users wouldn't have to read it. As pointed out throughout the text, automatically solved USE constraints can be used to replace existing hacks for different kinds of problems: A. Packages supporting selecting a subset of possible targets, e.g. Python implementations. Currently we have packages supporting 'any number of implementations', 'one implementation only' and 'one python2* + one python3*'. For user convenience, we hacked the second into separate PYTHON_SINGLE_TARGET flags which encumbered the implementation seriously, and we discouraged the third case completely. With automatically solved USE constraints, we'd just use PYTHON_TARGETS for everything, and let the constraints handle invalid combinations, e.g.: PYTHON_TARGETS="python2_7 python3_5 {-python3_4} {-python3_6} ..." B. Provider flags. With automatically solved constraints, we can easily move them all into unary USE_EXPAND sets with USE constraints, and let the PM handle selecting the one most preferred automatically, e.g.: USE="[ssl]" PROVIDER_SSL="gnutls {-openssl} {-libressl} ..." What's important, the automatic flags have the advantage of being both clearer in meaning than implicit flags, and more convenient than REQUIRED_USE. In other words, the user clearly sees which implementation (provider) is used, and can change that in a relatively easy way. Furthermore, the explicit visibility of selection reduces the number of USE flag combinations in binary packages. That is, we can easily reduce the number of different apparent sets of USE flags that build the same package, making it easier for binary packages to match. 4. Comments =========== What do you think? -- Best regards, Michał Górny
signature.asc
Description: This is a digitally signed message part