Re: [gentoo-dev] sys-devel/autoconf wrong dependency?

[Zac Medico]

On 1/31/24 04:56, Andreas Fink wrote:

With the move of sys-devel/autoconf to dev-build/autoconf the ebuild
has some inconsistency, namely in the RDEPEND section, it is saying:

RDEPEND="
 ${BDEPEND}
 >=dev-build/autoconf-wrapper-20231224
 sys-devel/gnuconfig
 !~sys-devel/${P}:2.5
"

This should probably be `!~dev-build/${P}:2.5`. It shows up in the
following as a problem:
emerge -auvDN --changed-deps=y --with-bdeps=y @world # --> merges 
autoconf-2.72-r1

Now:
emaint --fix all # --> fixes dependency from sys-devel/autoconf to 
dev-build/autoconf

and again a full system update:
emerge -auvDN --changed-deps=y --with-bdeps=y @world # --> merges 
autoconf-2.72-r1

This would want to again install autoconf, due to the changed-deps flag.

I am quite sure that this is not the intended behaviour, or is it?

Best
Andreas



Yeah, that's not intended. Reported as https://bugs.gentoo.org/923424.
--
Thanks,
Zac



OpenPGP_signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Last rites: net-misc/drive

[Zac Medico]

commit ba6f1c6fd9b9434bd2c07cf7233ee38cb6ab430a
Author: Brian Harring 
AuthorDate: 2023-11-09 20:51:11 -0800
Commit: Zac Medico 
CommitDate: 2023-11-09 21:59:23 -0800

net-misc/drive: treeclean

Dead upstream and fully broken since 2023-02 due to google
auth changes.

Closes: https://bugs.gentoo.org/658028
Closes: https://bugs.gentoo.org/903862
Signed-off-by: Brian Harring 
Closes: https://github.com/gentoo/gentoo/pull/33751
Signed-off-by: Zac Medico 


OpenPGP_signature
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] go-module.eclass: Add GO_OPTIONAL flag

[Zac Medico]

On 11/21/21 02:57, Florian Schmaus wrote:

Following the pattern found in other eclasses, add GO_OPTIONAL to the
go-module eclass. This allows to inherit the eclass without pulling
its dependencies. See, e.g., bug #775779 for the motivation.

Signed-off-by: Florian Schmaus 
---
  eclass/go-module.eclass | 31 ++-
  1 file changed, 22 insertions(+), 9 deletions(-)

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
index 3ad8542a28ae..c9eb90ac62ea 100644
--- a/eclass/go-module.eclass
+++ b/eclass/go-module.eclass
@@ -1,4 +1,4 @@
-# Copyright 2019-2020 Gentoo Authors
+# Copyright 2019-2021 Gentoo Authors
  # Distributed under the terms of the GNU General Public License v2
  
  # @ECLASS: go-module.eclass

@@ -55,13 +55,17 @@ if [[ -z ${_GO_MODULE} ]]; then
  
  _GO_MODULE=1
  
-BDEPEND=">=dev-lang/go-1.12"

+if [[ ! ${GO_OPTIONAL} ]]; then
+   BDEPEND=">=dev-lang/go-1.12"
  
-# Workaround for pkgcheck false positive: https://github.com/pkgcore/pkgcheck/issues/214

-# MissingUnpackerDep: version ...: missing BDEPEND="app-arch/unzip"
-# Added here rather than to each affected package, so it can be cleaned up just
-# once when pkgcheck is improved.
-BDEPEND+=" app-arch/unzip"
+   # Workaround for pkgcheck false positive: 
https://github.com/pkgcore/pkgcheck/issues/214
+   # MissingUnpackerDep: version ...: missing BDEPEND="app-arch/unzip"
+   # Added here rather than to each affected package, so it can be cleaned 
up just
+   # once when pkgcheck is improved.
+   BDEPEND+=" app-arch/unzip"
+
+   EXPORT_FUNCTIONS src_unpack
+fi
  
  # Force go to build in module mode.

  # In this mode the GOPATH environment variable is ignored.
@@ -83,8 +87,6 @@ QA_FLAGS_IGNORED='.*'
  # Go packages should not be stripped with strip(1).
  RESTRICT+=" strip"
  
-EXPORT_FUNCTIONS src_unpack

-
  # @ECLASS-VARIABLE: EGO_SUM
  # @DESCRIPTION:
  # This is an array based on the go.sum content from inside the target package.
@@ -147,6 +149,17 @@ EXPORT_FUNCTIONS src_unpack
  # directory structure.
  declare -A -g _GOMODULE_GOSUM_REVERSE_MAP
  
+# @ECLASS-VARIABLE: GO_OPTIONAL

+# @DEFAULT_UNSET
+# @PRE_INHERIT
+# @DESCRIPTION:
+# If set to a non-null value before inherit, then the Go part of the
+# ebuild will be considered optional. No dependencies will be added and
+# no phase functions will be exported.
+#
+# If you enable GO_OPTIONAL, you have to set BDEPEND on >=dev-lang/go-1.12
+# for your package and call go-module_src_unpack manually.
+
  # @FUNCTION: go-module_set_globals
  # @DESCRIPTION:
  # Convert the information in EGO_SUM for other usage in the ebuild.



How about if we also add a GO_DEPEND variable, so that eclasshi 
consumers can do something like BDEPEND="go? ( ${GO_DEPEND} )" ?

--
Thanks,
Zac


OpenPGP_signature
Description: OpenPGP digital signature

[gentoo-dev] Re: [PATCH 0/2] go-module.eclass cleanups

[Zac Medico]

On 5/21/21 8:45 AM, William Hubbs wrote:
> This is an improvement to my previous patch. It is a patch series now
> because there are two separate changes:
> 
> - GOPROXY is exported in go-module_set_globals since it is not needed if
>   EGO_SUM is not set in the ebuild.
> 
> - go-module_setup_proxy is added. This function sets up the go proxy
>   directory so that dependencies can be read from it.
> 
> William Hubbs (2):
>   go-module.eclass: fix GOPROXY export
>   go-module.eclass: add go-module_setup_proxy function
> 
>  eclass/go-module.eclass | 47 +++--
>  1 file changed, 45 insertions(+), 2 deletions(-)
> 

This series looks great to me! Thank you!
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

[Zac Medico]

On 5/20/21 9:03 AM, William Hubbs wrote:
> On Wed, May 19, 2021 at 01:57:38PM -0700, Zac Medico wrote:
>> On 5/19/21 1:45 PM, Zac Medico wrote:
>>>> +# @FUNCTION: go-module_setup_proxy
>>>> +# @DESCRIPTION:
>>>> +# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
>>>> +# this function in src_unpack.
>>>> +# It sets up the go module proxy in the appropriate location and exports
>>>> +# the GOPROXY environment variable so that go calls will be able to
>>>> +# locate the proxy directory.
>>>> +go-module_setup_proxy() {
>>>> +  # shellcheck disable=SC2120
>>>> +  debug-print-function "${FUNCNAME}" "$@"
>>>> +
>>>> +  if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
>>>> +  die "go-module_set_globals must be called in global scope"
>>>> +  fi
>>>> +
>>>> +  local goproxy_dir="${T}/go-proxy"
>>>> +  mkdir -p "${goproxy_dir}" || die
>>>> +
>>>> +  # For each Golang module distfile, look up where it's supposed to go and
>>>> +  # symlink it into place.
>>>> +  local f
>>>> +  local goproxy_mod_dir
>>>> +  for f in ${A}; do
>>>> +  goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
>>>> +  if [[ -n "${goproxy_mod_path}" ]]; then
>>>> +  debug-print-function "Populating go proxy for 
>>>> ${goproxy_mod_path}"
>>>> +  # Build symlink hierarchy
>>>> +  goproxy_mod_dir=$( dirname 
>>>> "${goproxy_dir}"/"${goproxy_mod_path}" )
>>>> +  mkdir -p "${goproxy_mod_dir}" || die
>>>> +  ln -sf "${DISTDIR}"/"${f}" 
>>>> "${goproxy_dir}/${goproxy_mod_path}" ||
>>>> +  die "Failed to ln"
>>>> +  local v=${goproxy_mod_path}
>>>> +  v="${v%.mod}"
>>>> +  v="${v%.zip}"
>>>> +  v="${v//*\/}"
>>>> +  _go-module_gosum_synthesize_files "${goproxy_mod_dir}" 
>>>> "${v}"
>>>> +  fi
>>>> +  done
>>>> +  export GOPROXY="file://${goproxy_dir}"
>>>> +
>>>> +  # Validate the gosum now
>>>> +  _go-module_src_unpack_verify_gosum
>>>> +}
>>>> +
>>>>  # @FUNCTION: go-module_src_unpack
>>>>  # @DESCRIPTION:
>>>>  # If EGO_SUM is set, unpack the base tarball(s) and set up the
>>>>
>>>
>>> The go-module_setup_proxy function solves bug 790851 nicely, since
>>> sys-cluster/k3s ebuilds can call that instead of go-module_src_unpack.
>>
>> I do have one criticism of the go-module_setup_proxy, which is that it
>> relies on the side-effect of the GOPROXY export for its operation. We
>> can instead echo the GOPROXY value to stdout and force the caller to
>> export it themselves, and provide a convenience wrapper function which
>> works based on side-effects.
> 
> You really only need GOPROXY if you use EGO_SUM in your ebuild, so I
> could probably export GOPROXY in go-module_set_globals.
> 
> William
> 

Yes, go-module_set_globals is a nice place to consolidate all of the
exports. Great idea!
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

[Zac Medico]

On 5/19/21 1:45 PM, Zac Medico wrote:
>> +# @FUNCTION: go-module_setup_proxy
>> +# @DESCRIPTION:
>> +# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
>> +# this function in src_unpack.
>> +# It sets up the go module proxy in the appropriate location and exports
>> +# the GOPROXY environment variable so that go calls will be able to
>> +# locate the proxy directory.
>> +go-module_setup_proxy() {
>> +# shellcheck disable=SC2120
>> +debug-print-function "${FUNCNAME}" "$@"
>> +
>> +if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
>> +die "go-module_set_globals must be called in global scope"
>> +fi
>> +
>> +local goproxy_dir="${T}/go-proxy"
>> +mkdir -p "${goproxy_dir}" || die
>> +
>> +# For each Golang module distfile, look up where it's supposed to go and
>> +# symlink it into place.
>> +local f
>> +local goproxy_mod_dir
>> +for f in ${A}; do
>> +goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
>> +if [[ -n "${goproxy_mod_path}" ]]; then
>> +debug-print-function "Populating go proxy for 
>> ${goproxy_mod_path}"
>> +# Build symlink hierarchy
>> +goproxy_mod_dir=$( dirname 
>> "${goproxy_dir}"/"${goproxy_mod_path}" )
>> +mkdir -p "${goproxy_mod_dir}" || die
>> +ln -sf "${DISTDIR}"/"${f}" 
>> "${goproxy_dir}/${goproxy_mod_path}" ||
>> +die "Failed to ln"
>> +local v=${goproxy_mod_path}
>> +v="${v%.mod}"
>> +v="${v%.zip}"
>> +v="${v//*\/}"
>> +_go-module_gosum_synthesize_files "${goproxy_mod_dir}" 
>> "${v}"
>> +fi
>> +done
>> +export GOPROXY="file://${goproxy_dir}"
>> +
>> +# Validate the gosum now
>> +_go-module_src_unpack_verify_gosum
>> +}
>> +
>>  # @FUNCTION: go-module_src_unpack
>>  # @DESCRIPTION:
>>  # If EGO_SUM is set, unpack the base tarball(s) and set up the
>>
> 
> The go-module_setup_proxy function solves bug 790851 nicely, since
> sys-cluster/k3s ebuilds can call that instead of go-module_src_unpack.

I do have one criticism of the go-module_setup_proxy, which is that it
relies on the side-effect of the GOPROXY export for its operation. We
can instead echo the GOPROXY value to stdout and force the caller to
export it themselves, and provide a convenience wrapper function which
works based on side-effects.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

[Zac Medico]

On 5/19/21 12:48 PM, William Hubbs wrote:
> If an ebuild uses EGO_SUM and needs to define a custom src_unpack phase,
> these functions will make that easier.
> 
> go-module_setup_proxy is used to create a local file proxy of the
> dependencies listed in EGO_SUM and go-module_filter_proxy is used to
> create a new ${A} with the EGO_SUM_SRC_URI values removed.
> 
> Signed-off-by: William Hubbs 
> ---
>  eclass/go-module.eclass | 69 +
>  1 file changed, 69 insertions(+)
> 
> diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> index c9a7ab12eaf..80e1f711215 100644
> --- a/eclass/go-module.eclass
> +++ b/eclass/go-module.eclass
> @@ -236,6 +236,75 @@ go-module_set_globals() {
>   _GO_MODULE_SET_GLOBALS_CALLED=1
>  }
>  
> +# @FUNCTION: go-module_filter_proxy
> +# @DESCRIPTION:
> +# If your ebuild redefines src_unpack and uses EGO_SUM, use the return
> +# value of this function in place of ${A} in your src_unpack function.
> +# It filters the EGO_SUM_SRC_URI values out of SRC_URI.
> +go-module_filter_proxy() {
> + # shellcheck disable=SC2120
> + debug-print-function "${FUNCNAME}" "$@"
> +
> + if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
> + die "go-module_set_globals must be called in global scope"
> + fi
> +
> + # Skip golang modules and add the rest of the entries to a local
> + # array.
> + local f
> + local -a src
> + for f in ${A}; do
> + if [[ -z ${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]} ]]; then
> + src+=("${f}")
> + fi
> + done
> + echo "${src[@]}"
> +}

While this go-module_filter_proxy might seem like a convenient way for
an ebuild to list distfiles which are not derived from EGO_SUM, I think
it's not the responsibility of go-module.eclass to provide this
information, therefore it's probably better to force the ebuild
developer to provide their own mechanism to track those files if needed,
rather than add a superfluous function to the eclass.

> +# @FUNCTION: go-module_setup_proxy
> +# @DESCRIPTION:
> +# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
> +# this function in src_unpack.
> +# It sets up the go module proxy in the appropriate location and exports
> +# the GOPROXY environment variable so that go calls will be able to
> +# locate the proxy directory.
> +go-module_setup_proxy() {
> + # shellcheck disable=SC2120
> + debug-print-function "${FUNCNAME}" "$@"
> +
> + if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
> + die "go-module_set_globals must be called in global scope"
> + fi
> +
> + local goproxy_dir="${T}/go-proxy"
> + mkdir -p "${goproxy_dir}" || die
> +
> + # For each Golang module distfile, look up where it's supposed to go and
> + # symlink it into place.
> + local f
> + local goproxy_mod_dir
> + for f in ${A}; do
> + goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
> + if [[ -n "${goproxy_mod_path}" ]]; then
> + debug-print-function "Populating go proxy for 
> ${goproxy_mod_path}"
> + # Build symlink hierarchy
> + goproxy_mod_dir=$( dirname 
> "${goproxy_dir}"/"${goproxy_mod_path}" )
> + mkdir -p "${goproxy_mod_dir}" || die
> + ln -sf "${DISTDIR}"/"${f}" 
> "${goproxy_dir}/${goproxy_mod_path}" ||
> + die "Failed to ln"
> + local v=${goproxy_mod_path}
> + v="${v%.mod}"
> + v="${v%.zip}"
> + v="${v//*\/}"
> + _go-module_gosum_synthesize_files "${goproxy_mod_dir}" 
> "${v}"
> + fi
> + done
> + export GOPROXY="file://${goproxy_dir}"
> +
> + # Validate the gosum now
> + _go-module_src_unpack_verify_gosum
> +}
> +
>  # @FUNCTION: go-module_src_unpack
>  # @DESCRIPTION:
>  # If EGO_SUM is set, unpack the base tarball(s) and set up the
> 

The go-module_setup_proxy function solves bug 790851 nicely, since
sys-cluster/k3s ebuilds can call that instead of go-module_src_unpack.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Package up for grabs: dev-python/python-backoff

[Zac Medico]

We have and ebuild for this package up for grabs:

https://pypi.org/project/backoff/

No open bugs and no reverse dependencies:

https://bugs.gentoo.org/buglist.cgi?quicksearch=python-backoff
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/24/21 2:29 AM, Zac Medico wrote:
> For example, for 3 USE flags, up to 8 combinations will be indexed:
> 
> IUSE="a b c installsources splitdebug"
> SRC_URI="
>   !a? !b? !c? ( mirror://binhost/24fe6bd377 )
>   !a? !b?  c? ( mirror://binhost/fbe14cbb02 )
>   !a?  b? !c? ( mirror://binhost/1dfff1f2ac )
>   !a?  b?  c? ( mirror://binhost/ae60f2940d )
>a? !b? !c? ( mirror://binhost/2976e1acc0 )
>a? !b?  c? ( mirror://binhost/f4809db70c )
>a?  b? !c? ( mirror://binhost/ecd08466cf )
>a?  b?  c? ( mirror://binhost/0c00f33b2e )
>   installsources? (
> !a? !b? !c? ( mirror://binhost/063a14d6c7 )
> !a? !b?  c? ( mirror://binhost/f67c311625 )
> !a?  b? !c? ( mirror://binhost/1dfff1f2ac )
> !a?  b?  c? ( mirror://binhost/17a673e16a )
>  a? !b? !c? ( mirror://binhost/914d1cecfe )
>  a? !b?  c? ( mirror://binhost/ca18d86a2b )
>  a?  b? !c? ( mirror://binhost/6bce13471a )
>  a?  b?  c? ( mirror://binhost/3a6bdcd228 )
>   )
>   splitdebug? (
> !a? !b? !c? ( mirror://binhost/29b2f38c41 )
> !a? !b?  c? ( mirror://binhost/8adc9bef51 )
> !a?  b? !c? ( mirror://binhost/954d2ce484 )
> !a?  b?  c? ( mirror://binhost/32a614aaca )
>  a? !b? !c? ( mirror://binhost/3548a2302d )
>  a? !b?  c? ( mirror://binhost/e0c02cdc88 )
>  a?  b? !c? ( mirror://binhost/f9cbd3c181 )
>  a?  b?  c? ( mirror://binhost/31d4c03474 )
>   )
> "
> 
> For installsources, we can automate deduplication, so that we can
> distribute the same file content for multiple USE combinations when
> appropriate. If all of the combinations have identical content, then it
> will look like this:
> 
>   installsources? (
> !a? !b? !c? ( mirror://binhost/063a14d6c7 )
> !a? !b?  c? ( mirror://binhost/063a14d6c7 )
> !a?  b? !c? ( mirror://binhost/063a14d6c7 )
> !a?  b?  c? ( mirror://binhost/063a14d6c7 )
>  a? !b? !c? ( mirror://binhost/063a14d6c7 )
>  a? !b?  c? ( mirror://binhost/063a14d6c7 )
>  a?  b? !c? ( mirror://binhost/063a14d6c7 )
>  a?  b?  c? ( mirror://binhost/063a14d6c7 )
>   )
> 
> In order to ensure that an ebin is not selected for a USE combination
> that has not been built yet, combinations for existing builds will be
> listed in REQUIRED_USE, like this:
> 
> REQUIRED_USE="
> || (
>   ( !a !b !c )
>   ( !a !b  c )
>   ( !a  b !c )
>   ( !a  b  c )
>   (  a !b !c )
>   (  a !b  c )
>   (  a  b !c )
>   (  a  b  c )
> )
> "

In https://bugs.gentoo.org/772380 I'm planning to implement a script
that will take an existing $PKGDIR as input, and generate an "ebin"
binhost as output. It will automatically split out pre-built content
bundles for installsources and splitdebug as shown above. If there is
more than one build for a particular package version and USE
combination, then the script will choose the package instance with
latest BUILD_TIME metadata (in alignment with
FEATURES=binpkg-multi-instance).
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/23/21 12:33 PM, Zac Medico wrote:
> On 2/23/21 12:05 PM, Zac Medico wrote:
>> On 2/23/21 11:46 AM, Zac Medico wrote:
>>> On 2/20/21 8:17 PM, Zac Medico wrote:
>>>> IUSE_RUNTIME will obviously introduce conditionals in binary package
>>>> dependencies, but we should welcome these conditionals because they will
>>>> provide useful flexibility.
>>>>
>>>> I think IUSE_RUNTIME will be a very nice feature to have for project
>>>> binhost, since it will allow binary package dependencies to behave with
>>>> flexibility that more closely resembles the flexibility of ebuild
>>>> dependencies.
>>>
>>> We can borrow paludis's notion of pbins [1] to generate ebuilds which
>>> install pre-built content, and the generated ebuilds could have USE
>>> flags that behave similarly to IUSE_RUNTIME in the sense that changes to
>>> USE flags will result in different builds of pre-built content being
>>> installed. A content-hash distfiles layout [2] could serve as a
>>> convenient way to store separate builds of pre-built content for
>>> multiple combinations of USE flags, and a generated ebuild would index
>>> the build by USE flag combination.
>>>
>>> Also, for the generated ebuilds, we can generate USE flags to include
>>> separate SRC_URI downloads for pre-built content to support things like
>>> FEATURES=split-debug and FEATURES=install-sources.
>>
>> Note that all of this can use existing EAPI features, since everything new
>> would be implemented in an ebuild generator that generates a single
>> ebuild to index pre-built content from multiple binary package builds.
>>
>>> [1] https://paludis.exherbo.org/overview/pbins.html
>>> [2] https://bugs.gentoo.org/756778
>>
> 
> For generated ebuilds, we'll have a choice to model things as USE flags
> or sub-packages. For example, content from FEATURES=split-debug and
> FEATURES=install-sources content might make more sense to model as
> sub-packages than USE flags. It makes more sense to generate a
> sub-package when there is a need for the sub-package to have USE flags.
> For example, a split-debug sub-package can have USE flags which index
> pre-built content from builds for multiple USE flag combinations.
> Similar USE flags could be useful for an install-sources sub-package if
> the source code has patches which are conditional on USE flags.

Since the generated ebuilds are inspired by pbins, we might call them
"ebins". Once we have designed an ebin generation process that we're
happy with, we should consider making it part of an EAPI, so that
package managers can generate "ebins" on request. The EAPI should
include ways to split out files and distribute them separately based on
USE flags and/or sub-packages, so that binhost users can easily filter
which files are installed based on USE configuration.

We can automatically map user's splitdebug and installsources FEATURES
settings into USE settings, in the same way that FEATURES=test
automatically maps to USE=test.

Each generated ebuild (ebin) will use its SRC_URI metadata to index
builds for each USE flag combination for which a build exists. For
example, for 3 USE flags, up to 8 combinations will be indexed:

IUSE="a b c installsources splitdebug"
SRC_URI="
  !a? !b? !c? ( mirror://binhost/24fe6bd377 )
  !a? !b?  c? ( mirror://binhost/fbe14cbb02 )
  !a?  b? !c? ( mirror://binhost/1dfff1f2ac )
  !a?  b?  c? ( mirror://binhost/ae60f2940d )
   a? !b? !c? ( mirror://binhost/2976e1acc0 )
   a? !b?  c? ( mirror://binhost/f4809db70c )
   a?  b? !c? ( mirror://binhost/ecd08466cf )
   a?  b?  c? ( mirror://binhost/0c00f33b2e )
  installsources? (
!a? !b? !c? ( mirror://binhost/063a14d6c7 )
!a? !b?  c? ( mirror://binhost/f67c311625 )
!a?  b? !c? ( mirror://binhost/1dfff1f2ac )
!a?  b?  c? ( mirror://binhost/17a673e16a )
 a? !b? !c? ( mirror://binhost/914d1cecfe )
 a? !b?  c? ( mirror://binhost/ca18d86a2b )
 a?  b? !c? ( mirror://binhost/6bce13471a )
 a?  b?  c? ( mirror://binhost/3a6bdcd228 )
  )
  splitdebug? (
!a? !b? !c? ( mirror://binhost/29b2f38c41 )
!a? !b?  c? ( mirror://binhost/8adc9bef51 )
!a?  b? !c? ( mirror://binhost/954d2ce484 )
!a?  b?  c? ( mirror://binhost/32a614aaca )
 a? !b? !c? ( mirror://binhost/3548a2302d )
 a? !b?  c? ( mirror://binhost/e0c02cdc88 )
 a?  b? !c? ( mirror://binhost/f9cbd3c181 )
 a?  b?  c? ( mirror://binhost/31d4c03474 )
  )
"

For installsources, we can automate deduplication, so that we can
distribute the same file content for multiple USE combinations when
appropriate. If all of the combinations have identical content, then it
will look like this:

  ins

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/23/21 12:05 PM, Zac Medico wrote:
> On 2/23/21 11:46 AM, Zac Medico wrote:
>> On 2/20/21 8:17 PM, Zac Medico wrote:
>>> IUSE_RUNTIME will obviously introduce conditionals in binary package
>>> dependencies, but we should welcome these conditionals because they will
>>> provide useful flexibility.
>>>
>>> I think IUSE_RUNTIME will be a very nice feature to have for project
>>> binhost, since it will allow binary package dependencies to behave with
>>> flexibility that more closely resembles the flexibility of ebuild
>>> dependencies.
>>
>> We can borrow paludis's notion of pbins [1] to generate ebuilds which
>> install pre-built content, and the generated ebuilds could have USE
>> flags that behave similarly to IUSE_RUNTIME in the sense that changes to
>> USE flags will result in different builds of pre-built content being
>> installed. A content-hash distfiles layout [2] could serve as a
>> convenient way to store separate builds of pre-built content for
>> multiple combinations of USE flags, and a generated ebuild would index
>> the build by USE flag combination.
>>
>> Also, for the generated ebuilds, we can generate USE flags to include
>> separate SRC_URI downloads for pre-built content to support things like
>> FEATURES=split-debug and FEATURES=install-sources.
> 
> Note that all of this can use existing EAPI features, since everything new
> would be implemented in an ebuild generator that generates a single
> ebuild to index pre-built content from multiple binary package builds.
> 
>> [1] https://paludis.exherbo.org/overview/pbins.html
>> [2] https://bugs.gentoo.org/756778
> 

For generated ebuilds, we'll have a choice to model things as USE flags
or sub-packages. For example, content from FEATURES=split-debug and
FEATURES=install-sources content might make more sense to model as
sub-packages than USE flags. It makes more sense to generate a
sub-package when there is a need for the sub-package to have USE flags.
For example, a split-debug sub-package can have USE flags which index
pre-built content from builds for multiple USE flag combinations.
Similar USE flags could be useful for an install-sources sub-package if
the source code has patches which are conditional on USE flags.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/23/21 11:46 AM, Zac Medico wrote:
> On 2/20/21 8:17 PM, Zac Medico wrote:
>> On 2/13/21 4:53 PM, Zac Medico wrote:
>>> On 2/13/21 4:37 PM, Zac Medico wrote:
>>>> On 2/11/21 1:17 AM, Michał Górny wrote:
>>>>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>>>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>>>>
>>>>>>> Hi all, 
>>>>>>>
>>>>>>> I'm announcing a new project here - "binhost"
>>>>>>>
>>>>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>>>>> precompiled packages for a subset of configurations, via central
>>>>>>> binary package hosting. Currently we are still in the conceptual
>>>>>>> planning stage. "
>>>>>>>
>>>>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>>>>
>>>>>>> If you're interested in helping out, feel free to add yourself on the
>>>>>>> wiki page.
>>>>>>>
>>>>>>> Note that I see actually *building* the packages not as the central
>>>>>>> point of the project (that could be e.g. a side effect of a
>>>>>>> tinderbox). I'm more concerned about
>>>>>>> * what configurations should we use
>>>>>>> * what portage features are still needed or need improvements (e.g.
>>>>>>> binpkg signing and verification)
>>>>>>> * how should hosting look like
>>>>>>> * and how we can test this on a limited scale before it goes "into
>>>>>>> production"
>>>>>>> * ...
>>>>>>>
>>>>>>> Comments, ideas, flamebaits? :D
>>>>>>>
>>>>>>> Cheers, 
>>>>>>> Andreas
>>>>>>>
>>>>>>
>>>>>> It would be great to improve portage speed with handling binpkgs. I
>>>>>> already have my own binhost for a couple of Gentoo systems and even
>>>>>> though these systems don't have to compile anything themselves,
>>>>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>>>>> installation time. Arch Linux' pacman only takes a fraction of this
>>>>>> time for the very same task.
>>>>>> I know that I compare apples with pears here but even reducing the
>>>>>> current portage time by 50% would be a huge improvement.
>>>>>
>>>>> Is that really a problem?  For me, Portage takes about an hour just to
>>>>> do the dependency processing these days.  In fact, building from sources
>>>>> is now faster than dependency calculations.
>>>>
>>>> The ratio of these times is dependent on the complexity of the
>>>> dependencies involved, and so is the answer to your question.
>>>
>>> Also, in the context of binary packages, dependencies calculations are
>>> generally simpler for binary packages because their USE conditionals and
>>> slot-operator := dependencies are frozen in a particular state. This
>>> dramatically reduces the search space involved in dependency calculation.
>>
>> IUSE_RUNTIME will obviously introduce conditionals in binary package
>> dependencies, but we should welcome these conditionals because they will
>> provide useful flexibility.
>>
>> I think IUSE_RUNTIME will be a very nice feature to have for project
>> binhost, since it will allow binary package dependencies to behave with
>> flexibility that more closely resembles the flexibility of ebuild
>> dependencies.
> 
> We can borrow paludis's notion of pbins [1] to generate ebuilds which
> install pre-built content, and the generated ebuilds could have USE
> flags that behave similarly to IUSE_RUNTIME in the sense that changes to
> USE flags will result in different builds of pre-built content being
> installed. A content-hash distfiles layout [2] could serve as a
> convenient way to store separate builds of pre-built content for
> multiple combinations of USE flags, and a generated ebuild would index
> the build by USE flag combination.
> 
> Also, for the generated ebuilds, we can generate USE flags to include
> separate SRC_URI downloads for pre-built content to support things like
> FEATURES=split-debug and FEATURES=install-sources.

Note that all of this can existing EAPI features, since everything new
would be implemented in an ebuild generator that generates a single
ebuild to index pre-built content from multiple binary package builds.

> [1] https://paludis.exherbo.org/overview/pbins.html
> [2] https://bugs.gentoo.org/756778

-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/13/21 5:51 PM, Zac Medico wrote:
> On 2/10/21 11:11 AM, Rich Freeman wrote:
>> On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel  
>> wrote:
>>>
>>> * what portage features are still needed or need improvements (e.g. binpkg
>>> signing and verification)
>>> * how should hosting look like
>>
>> Some ideas for portage enhancements:
>>
>> 1.  Ability to fetch binary packages from some kind of repo.
> 
> The old PORTAGE_BINHOST functionality has been replaced with a
> binrepos.conf file that's very similar to repos.conf:
> 
> https://bugs.gentoo.org/668334
> 
> It doesn't have explicit support for multiple local binary package
> repositories yet, but somebody got it working with src-uri set to a
> file:/ uri as described in comments on this bug:
> 
> https://bugs.gentoo.org/768957
> 
>> 2.  Ability to have multiple binary packages co-exist in a repo (local
>> or remote) with different build attributes (arch, USE, CFLAGS,
>> DEPENDS, whatever).
> 
> We can now enable FEATURES=binpkg-multi-instance by default now that
> this bug is fixed:
> 
> https://bugs.gentoo.org/571126
> 
>> 3.  Ability to pick the most appropriate binary packages to use based
>> on user preferences (with a mix of hard and soft preferences).
> 
> Current package selection logic for binary packages is basically the
> same as for ebuilds. These are the notable differences:
> 
> 1) Binary packages are sorted in descending order by (version, mtime),
> so then most recent builds are preferred when the versions are identical.
> 
> 2) The --binpkg-respect-use option rejects binary packages what would
> need to be rebuilt in order to match local USE settings.
> 
>> One idea I've had around how #2-3 might be implemented is:
>> 1.  Binary packages already contain data on how they were built (USE
>> flags, dependencies, etc).  Place this in a file using a deterministic
>> sorting/etc order so that two builds with the same settings will have
>> the same results.
> 
> This would only be needed to multi-profile binhosts that provide a
> variety of configurations for the same package.
> 
> Features like this are not necessary if the binhost only intends to
> provide packages for a single profile.
> 
>> 2.  Generate a hash of the file contents - this can go in the filename
>> so that the file can co-exist with other files, and be located
>> assuming you have a full matching set of metadata.
> 
> For FEATURES=binpkg-multi-instance we currently use an integer BUILD_ID
> ensure that file names are unique.
> 
>> 3.  Start dropping attributes from the file based on a list of
>> priorities and generate additional hashes.  Create symlinked files to
>> the original file using these hashes (overwriting or not existing
>> symlinks based on policy).  This allows the binary package to be found
>> using either an exact set of attributes or a subset of higher-priority
>> attributes.  This is analogous to shared object symlinking.
>> 4.  The package manager will look for a binary package first using the
>> user's full config, and then by dropping optional elements of the
>> config (so maybe it does the search without CFLAGs, then without USE
>> flags).  Eventually it aborts based on user prefs (maybe the user only
>> wants an exact match, or is willing to accept alternate CFLAGs but not
>> USE flags, or maybe anything for the arch is selected> 5.  As always the 
>> final selected binary package still gets evaluated
>> like any other binary package to ensure it is usable.
>>
>> Such a system can identify whether a potentially usable file exists
>> using only filename, cutting down on fetching.  In the interests of
>> avoiding useless fetches we would only carry step 3 reasonably far -
>> packages would have to match based on architecture and any dynamic
>> linking requirements.  So we wouldn't generate hashes that didn't
>> include at least those minimums, and the package manager wouldn't
>> search for them.
>>
>> Obviously you could do more (if you have 5 combinations of use flags,
>> look for the set that matches most closely).  That couldn't be done
>> using hashes alone in an efficient way.  You could have a small
>> manifest file alongside the binary package that could be fetched
>> separately if the package manager wants to narrow things down and
>> fetch a few of those to narrow it down further.
> 
> All of the above is oriented toward multi-profile binhosts, so we'll
> have to do a cost/benefit analysis to determine whether it's worth the
> effort to introduce the complexity that multi-profile binhosts add.

This idea to borrow paludis's notion of pbins could work well for a
multi-profile binhost:

https://archives.gentoo.org/gentoo-dev/message/1aea0de0ff588c67bf652f4c1f8ef304
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/20/21 8:17 PM, Zac Medico wrote:
> On 2/13/21 4:53 PM, Zac Medico wrote:
>> On 2/13/21 4:37 PM, Zac Medico wrote:
>>> On 2/11/21 1:17 AM, Michał Górny wrote:
>>>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>>>
>>>>>> Hi all, 
>>>>>>
>>>>>> I'm announcing a new project here - "binhost"
>>>>>>
>>>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>>>> precompiled packages for a subset of configurations, via central
>>>>>> binary package hosting. Currently we are still in the conceptual
>>>>>> planning stage. "
>>>>>>
>>>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>>>
>>>>>> If you're interested in helping out, feel free to add yourself on the
>>>>>> wiki page.
>>>>>>
>>>>>> Note that I see actually *building* the packages not as the central
>>>>>> point of the project (that could be e.g. a side effect of a
>>>>>> tinderbox). I'm more concerned about
>>>>>> * what configurations should we use
>>>>>> * what portage features are still needed or need improvements (e.g.
>>>>>> binpkg signing and verification)
>>>>>> * how should hosting look like
>>>>>> * and how we can test this on a limited scale before it goes "into
>>>>>> production"
>>>>>> * ...
>>>>>>
>>>>>> Comments, ideas, flamebaits? :D
>>>>>>
>>>>>> Cheers, 
>>>>>> Andreas
>>>>>>
>>>>>
>>>>> It would be great to improve portage speed with handling binpkgs. I
>>>>> already have my own binhost for a couple of Gentoo systems and even
>>>>> though these systems don't have to compile anything themselves,
>>>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>>>> installation time. Arch Linux' pacman only takes a fraction of this
>>>>> time for the very same task.
>>>>> I know that I compare apples with pears here but even reducing the
>>>>> current portage time by 50% would be a huge improvement.
>>>>
>>>> Is that really a problem?  For me, Portage takes about an hour just to
>>>> do the dependency processing these days.  In fact, building from sources
>>>> is now faster than dependency calculations.
>>>
>>> The ratio of these times is dependent on the complexity of the
>>> dependencies involved, and so is the answer to your question.
>>
>> Also, in the context of binary packages, dependencies calculations are
>> generally simpler for binary packages because their USE conditionals and
>> slot-operator := dependencies are frozen in a particular state. This
>> dramatically reduces the search space involved in dependency calculation.
> 
> IUSE_RUNTIME will obviously introduce conditionals in binary package
> dependencies, but we should welcome these conditionals because they will
> provide useful flexibility.
> 
> I think IUSE_RUNTIME will be a very nice feature to have for project
> binhost, since it will allow binary package dependencies to behave with
> flexibility that more closely resembles the flexibility of ebuild
> dependencies.

We can borrow paludis's notion of pbins [1] to generate ebuilds which
install pre-built content, and the generated ebuilds could have USE
flags that behave similarly to IUSE_RUNTIME in the sense that changes to
USE flags will result in different builds of pre-built content being
installed. A content-hash distfiles layout [2] could serve as a
convenient way to store separate builds of pre-built content for
multiple combinations of USE flags, and a generated ebuild would index
the build by USE flag combination.

Also, for the generated ebuilds, we can generate USE flags to include
separate SRC_URI downloads for pre-built content to support things like
FEATURES=split-debug and FEATURES=install-sources.

[1] https://paludis.exherbo.org/overview/pbins.html
[2] https://bugs.gentoo.org/756778
-- 
Thanks,
Zac





signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/13/21 4:53 PM, Zac Medico wrote:
> On 2/13/21 4:37 PM, Zac Medico wrote:
>> On 2/11/21 1:17 AM, Michał Górny wrote:
>>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>>
>>>>> Hi all, 
>>>>>
>>>>> I'm announcing a new project here - "binhost"
>>>>>
>>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>>> precompiled packages for a subset of configurations, via central
>>>>> binary package hosting. Currently we are still in the conceptual
>>>>> planning stage. "
>>>>>
>>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>>
>>>>> If you're interested in helping out, feel free to add yourself on the
>>>>> wiki page.
>>>>>
>>>>> Note that I see actually *building* the packages not as the central
>>>>> point of the project (that could be e.g. a side effect of a
>>>>> tinderbox). I'm more concerned about
>>>>> * what configurations should we use
>>>>> * what portage features are still needed or need improvements (e.g.
>>>>> binpkg signing and verification)
>>>>> * how should hosting look like
>>>>> * and how we can test this on a limited scale before it goes "into
>>>>> production"
>>>>> * ...
>>>>>
>>>>> Comments, ideas, flamebaits? :D
>>>>>
>>>>> Cheers, 
>>>>> Andreas
>>>>>
>>>>
>>>> It would be great to improve portage speed with handling binpkgs. I
>>>> already have my own binhost for a couple of Gentoo systems and even
>>>> though these systems don't have to compile anything themselves,
>>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>>> installation time. Arch Linux' pacman only takes a fraction of this
>>>> time for the very same task.
>>>> I know that I compare apples with pears here but even reducing the
>>>> current portage time by 50% would be a huge improvement.
>>>
>>> Is that really a problem?  For me, Portage takes about an hour just to
>>> do the dependency processing these days.  In fact, building from sources
>>> is now faster than dependency calculations.
>>
>> The ratio of these times is dependent on the complexity of the
>> dependencies involved, and so is the answer to your question.
> 
> Also, in the context of binary packages, dependencies calculations are
> generally simpler for binary packages because their USE conditionals and
> slot-operator := dependencies are frozen in a particular state. This
> dramatically reduces the search space involved in dependency calculation.

IUSE_RUNTIME will obviously introduce conditionals in binary package
dependencies, but we should welcome these conditionals because they will
provide useful flexibility.

I think IUSE_RUNTIME will be a very nice feature to have for project
binhost, since it will allow binary package dependencies to behave with
flexibility that more closely resembles the flexibility of ebuild
dependencies.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/10/21 11:11 AM, Rich Freeman wrote:
> On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel  
> wrote:
>>
>> * what portage features are still needed or need improvements (e.g. binpkg
>> signing and verification)
>> * how should hosting look like
> 
> Some ideas for portage enhancements:
> 
> 1.  Ability to fetch binary packages from some kind of repo.

The old PORTAGE_BINHOST functionality has been replaced with a
binrepos.conf file that's very similar to repos.conf:

https://bugs.gentoo.org/668334

It doesn't have explicit support for multiple local binary package
repositories yet, but somebody got it working with src-uri set to a
file:/ uri as described in comments on this bug:

https://bugs.gentoo.org/768957

> 2.  Ability to have multiple binary packages co-exist in a repo (local
> or remote) with different build attributes (arch, USE, CFLAGS,
> DEPENDS, whatever).

We can now enable FEATURES=binpkg-multi-instance by default now that
this bug is fixed:

https://bugs.gentoo.org/571126

> 3.  Ability to pick the most appropriate binary packages to use based
> on user preferences (with a mix of hard and soft preferences).

Current package selection logic for binary packages is basically the
same as for ebuilds. These are the notable differences:

1) Binary packages are sorted in descending order by (version, mtime),
so then most recent builds are preferred when the versions are identical.

2) The --binpkg-respect-use option rejects binary packages what would
need to be rebuilt in order to match local USE settings.

> One idea I've had around how #2-3 might be implemented is:
> 1.  Binary packages already contain data on how they were built (USE
> flags, dependencies, etc).  Place this in a file using a deterministic
> sorting/etc order so that two builds with the same settings will have
> the same results.

This would only be needed to multi-profile binhosts that provide a
variety of configurations for the same package.

Features like this are not necessary if the binhost only intends to
provide packages for a single profile.

> 2.  Generate a hash of the file contents - this can go in the filename
> so that the file can co-exist with other files, and be located
> assuming you have a full matching set of metadata.

For FEATURES=binpkg-multi-instance we currently use an integer BUILD_ID
ensure that file names are unique.

> 3.  Start dropping attributes from the file based on a list of
> priorities and generate additional hashes.  Create symlinked files to
> the original file using these hashes (overwriting or not existing
> symlinks based on policy).  This allows the binary package to be found
> using either an exact set of attributes or a subset of higher-priority
> attributes.  This is analogous to shared object symlinking.
> 4.  The package manager will look for a binary package first using the
> user's full config, and then by dropping optional elements of the
> config (so maybe it does the search without CFLAGs, then without USE
> flags).  Eventually it aborts based on user prefs (maybe the user only
> wants an exact match, or is willing to accept alternate CFLAGs but not
> USE flags, or maybe anything for the arch is selected> 5.  As always the 
> final selected binary package still gets evaluated
> like any other binary package to ensure it is usable.
> 
> Such a system can identify whether a potentially usable file exists
> using only filename, cutting down on fetching.  In the interests of
> avoiding useless fetches we would only carry step 3 reasonably far -
> packages would have to match based on architecture and any dynamic
> linking requirements.  So we wouldn't generate hashes that didn't
> include at least those minimums, and the package manager wouldn't
> search for them.
> 
> Obviously you could do more (if you have 5 combinations of use flags,
> look for the set that matches most closely).  That couldn't be done
> using hashes alone in an efficient way.  You could have a small
> manifest file alongside the binary package that could be fetched
> separately if the package manager wants to narrow things down and
> fetch a few of those to narrow it down further.

All of the above is oriented toward multi-profile binhosts, so we'll
have to do a cost/benefit analysis to determine whether it's worth the
effort to introduce the complexity that multi-profile binhosts add.

> Or you could skip the hash searching and just fetch all the manifests
> for a particular package/arch and just search all of those, but that
> is more data to transfer just to do a query.  A metadata cache of some
> kind of might be another solution.  Content hashes would probably
> still be useful just to allow co-existence of alternate builds.

This also relates to the centralized Packages file that's currently used
to distribute the package metadata for all packages in a binhost. We can
make it scale better if we split out a separate index per package, not
unlike a pypi simple index:

https://pypi.org/simple/
-- 
Thanks

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/13/21 4:37 PM, Zac Medico wrote:
> On 2/11/21 1:17 AM, Michał Górny wrote:
>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>>
>>>> Hi all, 
>>>>
>>>> I'm announcing a new project here - "binhost"
>>>>
>>>> "The Gentoo Binhost project aims to provide readily installable,
>>>> precompiled packages for a subset of configurations, via central
>>>> binary package hosting. Currently we are still in the conceptual
>>>> planning stage. "
>>>>
>>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>>
>>>> If you're interested in helping out, feel free to add yourself on the
>>>> wiki page.
>>>>
>>>> Note that I see actually *building* the packages not as the central
>>>> point of the project (that could be e.g. a side effect of a
>>>> tinderbox). I'm more concerned about
>>>> * what configurations should we use
>>>> * what portage features are still needed or need improvements (e.g.
>>>> binpkg signing and verification)
>>>> * how should hosting look like
>>>> * and how we can test this on a limited scale before it goes "into
>>>> production"
>>>> * ...
>>>>
>>>> Comments, ideas, flamebaits? :D
>>>>
>>>> Cheers, 
>>>> Andreas
>>>>
>>>
>>> It would be great to improve portage speed with handling binpkgs. I
>>> already have my own binhost for a couple of Gentoo systems and even
>>> though these systems don't have to compile anything themselves,
>>> installing ~100 to ~200 binpkgs takes way more than an hour of
>>> installation time. Arch Linux' pacman only takes a fraction of this
>>> time for the very same task.
>>> I know that I compare apples with pears here but even reducing the
>>> current portage time by 50% would be a huge improvement.
>>
>> Is that really a problem?  For me, Portage takes about an hour just to
>> do the dependency processing these days.  In fact, building from sources
>> is now faster than dependency calculations.
> 
> The ratio of these times is dependent on the complexity of the
> dependencies involved, and so is the answer to your question.

Also, in the context of binary packages, dependencies calculations are
generally simpler for binary packages because their USE conditionals and
slot-operator := dependencies are frozen in a particular state. This
dramatically reduces the search space involved in dependency calculation.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/11/21 1:17 AM, Michał Górny wrote:
> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote:
>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
>>
>>> Hi all, 
>>>
>>> I'm announcing a new project here - "binhost"
>>>
>>> "The Gentoo Binhost project aims to provide readily installable,
>>> precompiled packages for a subset of configurations, via central
>>> binary package hosting. Currently we are still in the conceptual
>>> planning stage. "
>>>
>>> https://wiki.gentoo.org/wiki/Project:Binhost
>>>
>>> If you're interested in helping out, feel free to add yourself on the
>>> wiki page.
>>>
>>> Note that I see actually *building* the packages not as the central
>>> point of the project (that could be e.g. a side effect of a
>>> tinderbox). I'm more concerned about
>>> * what configurations should we use
>>> * what portage features are still needed or need improvements (e.g.
>>> binpkg signing and verification)
>>> * how should hosting look like
>>> * and how we can test this on a limited scale before it goes "into
>>> production"
>>> * ...
>>>
>>> Comments, ideas, flamebaits? :D
>>>
>>> Cheers, 
>>> Andreas
>>>
>>
>> It would be great to improve portage speed with handling binpkgs. I
>> already have my own binhost for a couple of Gentoo systems and even
>> though these systems don't have to compile anything themselves,
>> installing ~100 to ~200 binpkgs takes way more than an hour of
>> installation time. Arch Linux' pacman only takes a fraction of this
>> time for the very same task.
>> I know that I compare apples with pears here but even reducing the
>> current portage time by 50% would be a huge improvement.
> 
> Is that really a problem?  For me, Portage takes about an hour just to
> do the dependency processing these days.  In fact, building from sources
> is now faster than dependency calculations.

The ratio of these times is dependent on the complexity of the
dependencies involved, and so is the answer to your question.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New project: binhost

[Zac Medico]

On 2/10/21 10:51 AM, Lars Wendler wrote:
> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote:
> 
>> Hi all, 
>>
>> I'm announcing a new project here - "binhost"
>>
>> "The Gentoo Binhost project aims to provide readily installable,
>> precompiled packages for a subset of configurations, via central
>> binary package hosting. Currently we are still in the conceptual
>> planning stage. "
>>
>> https://wiki.gentoo.org/wiki/Project:Binhost
>>
>> If you're interested in helping out, feel free to add yourself on the
>> wiki page.
>>
>> Note that I see actually *building* the packages not as the central
>> point of the project (that could be e.g. a side effect of a
>> tinderbox). I'm more concerned about
>> * what configurations should we use
>> * what portage features are still needed or need improvements (e.g.
>> binpkg signing and verification)
>> * how should hosting look like
>> * and how we can test this on a limited scale before it goes "into
>> production"
>> * ...
>>
>> Comments, ideas, flamebaits? :D
>>
>> Cheers, 
>> Andreas
>>
> 
> It would be great to improve portage speed with handling binpkgs. I
> already have my own binhost for a couple of Gentoo systems and even
> though these systems don't have to compile anything themselves,
> installing ~100 to ~200 binpkgs takes way more than an hour of
> installation time. Arch Linux' pacman only takes a fraction of this
> time for the very same task.
> I know that I compare apples with pears here but even reducing the
> current portage time by 50% would be a huge improvement.

In order to maximize throughput, we have FEATURES="parallel-install"
that for example allows one package's files to be merged while a
pkg_{pre,post}{inst,rm} phase is running for an unrelated package that
is in the merge queue.

There's also FEATURES="-ebuild-locks" which allows privileged
pkg_{pre,post}{inst,rm} phases to run concurrently with privileged
phases of unrelated packages in the merge queue.

Ultimately, pkg_{pre,post}{inst,rm} phases could be the limiting factor
here. I portage, we should eliminate calls to these phases when
DEFINED_PHASES metadata indicated that they're not defined.

Also, FEATURES=preserve-libs introduces considerable overhead because it
regenerates LinkageMapELF data structures for every merge. It would be
much more efficient if it could incrementally update these data structures.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] speeding up usage of portage in e-file / portage file list

[Zac Medico]

On 5/24/20 9:40 AM, Daniel Buschke wrote:
> Am 24.05.2020 um 14:10 schrieb Daniel Pielmeier:
>> Here the bash version takes around 2.9 seconds while the python version
>> takes 3.2 seconds. Excluding the portage API it takes 2.8 seconds and
>> also excluding the data query it takes 0.3 seconds. So in the python
>> version the data query takes 2.5 seconds (probably this is similar for
>> the bash version) while all the rest takes 0.7 seconds
>>
>> My initial tests showed that the bash version is a lot quicker than the
>> python version. Somehow I can not reproduce this any more. As mentioned
>> previously the data query is the most time consuming part in both the
>> bash and the python version.
> 
> Oh dear! I readded the database index for file names. Now the data query
> takes ~0.3 seconds *insert self slapping image here*
> 
> Anyway, for some strange reason I cannot reproduce the slothy behaviour
> of portage, too. I'm 100% sure the bash version took 1 second while the
> python version took 3 seconds. Strange.
> 
> @Zac: Did you add some performance optimizations in the last 30 days?
> Maybe Caching? No? Then you fixed this by pure imagination :)

No portage changes, but it looks like whatever "big difference" you've
observed was probably related to the slowest step which is the remote
data query.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] speeding up usage of portage in e-file / portage file list

[Zac Medico]

On 5/23/20 1:41 PM, magic-gen...@damage.devloop.de wrote:
> Am 23.05.2020 um 22:20 schrieb Zac Medico:
>> On 5/23/20 1:02 PM, magic-gen...@damage.devloop.de wrote:
>>> I rewrote e-file in python by using the portage API [1]. But loading the
>>> API slows down the whole script. Is there any way to speed up my
>>> implementation? Have I done something fundamentally wrong?
>>
>> When I patched the portage API out of your script, I saw the run time
>> drop from 4.2 seconds to 3.2 seconds with this patch:
>> ...
>>
>> Are your results worse than mine?
> 
> Nope, but maybe the phrase "loading the API" was misleading. I'd like to
> replace it with: "But using the API slows down the whole script.". This
> means it is much slower to get the additional informations by portage
> API than just grep'ing throught the ebuild files. If I run the python
> e-file on my machine it takes 3.2 seconds for a single file. The bash
> e-file show the same result within a second or so.
> 
> regards
> Daniel
> 
> 

Since the portage API only added about 1 second to the python script
time, I guess it's on par with your bash implementation. ;-P
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] speeding up usage of portage in e-file / portage file list

[Zac Medico]

On 5/23/20 1:02 PM, magic-gen...@damage.devloop.de wrote:
> Hi,
> the current e-file tool of app-portage/pfl is written in bash. e-file is
> a CLI tool which searches portagefilelist.de for a given file and list
> additional informations based on the local portage. Latter is done by
> grep'ing through the ebuild files which is fast but IMHO not very smart.
> 
> I rewrote e-file in python by using the portage API [1]. But loading the
> API slows down the whole script. Is there any way to speed up my
> implementation? Have I done something fundamentally wrong?
> 
> Thanks in advance
> Daniel
> 
> [1] https://github.com/portagefilelist/client/blob/e-file-python/bin/e-file
> 

When I patched the portage API out of your script, I saw the run time
drop from 4.2 seconds to 3.2 seconds with this patch:

$ diff -U1 e-file.py.orig e-file.py
--- e-file.py.orig  2020-05-23 13:16:26.873861849 -0700
+++ e-file.py   2020-05-23 13:16:42.076861548 -0700
@@ -6,3 +6,3 @@
 import json
-import portage
+#import portage

@@ -35,2 +35,3 @@
 files = json.loads(r.text)
+sys.exit(0)
 if len(files['result']) > 0:

Are your results worse than mine?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: "emerge --sync" vs "emaint sync"

[Zac Medico]

On 5/6/20 3:02 PM, William Hubbs wrote:
> All,
> 
> I know that most of our documentation tells people to use "emerge --sync";
> however, today I heard about "emaint sync" for the first time.  ;-)
> 
> Which one should we use? Will there be a phase-out for "emerge --sync" or
> "emaint sync"? Are the plans to keep both available?

I'm strongly against removal of emerge --sync, since I don't want to
deal with inevitable backlash from trying to force users to change their
habits for no apparent reason.

> 
> Thanks,
> 
> William
> 


-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Zac Medico]

On 3/27/20 4:12 PM, Alec Warner wrote:
> On Fri, Mar 27, 2020 at 3:54 PM Patrick McLean  <mailto:chutz...@gentoo.org>> wrote:
> 
> On Fri, 27 Mar 2020 15:51:35 -0700
> Alec Warner mailto:anta...@gentoo.org>> wrote:
> 
> > On Fri, Mar 27, 2020 at 3:11 PM Patrick McLean
> mailto:chutz...@gentoo.org>> wrote:
> >
> > > On Fri, 27 Mar 2020 14:48:53 -0700
> > > Matt Turner mailto:matts...@gentoo.org>>
> wrote:
> > > 
> > > > On Thu, Mar 26, 2020 at 2:03 PM Patrick McLean
> mailto:chutz...@gentoo.org>> 
> > > wrote: 
> > > > >
> > > > > This patch splits the definition of _PYTHON_ALL_IMPLS and
> > > > > _python_impl_supported to a separate eclass, this allows
> overlays
> > > > > to easily support a different set of python implementations than
> > > > > ::gentoo without having to fork the entire suite of eclasses. 
> > > >
> > > > (I think the issue is that you have some packages that still need
> > > > Python 3.4. Correct me if I'm wrong)
> > > >
> > > > How many packages do you need to work with Python 3.4?
> Presumably just
> > > > a couple and then a pile of dependencies in ::gentoo?
> > > > 
> > >
> > > For our particular purpose, it's more complicated than that. We
> do not
> > > expect or want ::gentoo to try support Python 3.4 in any way. We
> have an
> > > overlay that is shared on a lot of machines, some of those
> machines are
> > > older than others. As such we still have ebuilds that only support
> > > python3_4 that still exist in the overlay. We would like it if the
> > > existence of these packages in the overlay, do not cause metadata
> > > generation or dependency calculation to explode on the more
> up-to-date
> > > machines.
> > > 
> >
> > > We do not need Python 3.4 packages to be installable on the newer
> > > machines, we just need them not to explode.
> > > 
> >
> > Couldn't you simply remove the ebuilds from the overlay entirely
> in this
> > case? It's my understanding that on the machines with the packages
> > installed, the merged package metadata is being used (which is why
> those
> > machines work) and since the newer machines don't have this merged
> package
> > metadata, they don't work properly.
> >
> 
> Sometimes we have to fix the older machines, so we have to keep
> everything around until none of our machines are using it any more.
> 
> 
> +Zac Medico <mailto:zmed...@gentoo.org> 
> 
> I'm not following something then. One of the proposals earlier was "do
> not generate metadata for these ebuilds" which to me reads as "keep
> these ebuilds in the repo, but the ebuilds themselves will not be
> usable[0]." Then you go on to say that old machines need to be fixed
> occasionally, so either you need to reinstall a package or update it.
> The reinstall might be strictly possible from the vdb metadata, but the
> upgrade would require working repository metadata, which we said earlier
> we didn't want to generate.

The ebuilds may or may not be usable, depending on the snapshot of
::gentoo eclasses that they're paired with.

> (There is a different question of whether you could use a binpkg binhost
> to basically build versions of these packages and use those for
> reinstalls, because at least the binpkg metadata *is* nominally fixed in
> time and can be re-used easily and doesn't require eclass magic in
> theory, as the eclasses are bound in the environment.tar?) I suspect in
> essence it might be easier to just do what Robin suggested and use a
> frozen ::gentoo on the older machines.

We do use a frozen snapshot of ::gentoo on older machines. The issue is
that we've got a repository of ebuilds that can be used with many
different snapshots of ::gentoo, but we'd like to be able to support
python implementations that may not be officially supported by a
particular snapshot of ::gentoo eclasses.

> -A
> 
> [0] Perhaps the earlier proposals were ..slightly off. With more
> information it seems like what you want is to be able to easily maintain
> some python-3.4 ebuilds in your overlay, even though Gentoo is on to 3.6
> (and soon 3.7). I personally think the conversation would have gone much
> better had you just come out and said "

Re: [gentoo-dev] [RFC] Discontinuing (more-than-absolutely-minimal) Python support for non-x86 arches

[Zac Medico]

On 3/14/20 1:15 PM, James Le Cuirot wrote:
> On Sat, 14 Mar 2020 19:13:58 +0100
> Michał Górny  wrote:
> I've never joined an arch team as I'm not really interested in stable.
> I understand what you're saying about keyword requests but I didn't
> realise they were also a big issue.
> 
> I'm not even keeping up with the work I'm supposed to be doing already
> so I don't exactly want to sign up for more. However, I do have a
> couple of (unstable) arm systems that are important to me so I need to
> keep them running and up to date. Perhaps we could write a tool that
> looks up newer versions of installed ebuilds that have dropped keywords
> for a given arch and report which dependencies also need keywording. I
> guess we don't have something like that?

Automation like that sounds good.

Also, I wonder if we should introduce an arch-independent keyword which
would probably make sense for a lot of python packages.

Also, I've got this patch from drobbins that allows one architecture to
sort of inherit keywords from another architecture:

https://github.com/zmedico/portage/compare/master...zmedico:drobbins-track-keywords?expand=1
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Inconsistent use of || preferences for www-client/elinks, links, lynx, w3m, and w3mmee

[Zac Medico]

On 2/9/20 10:44 PM, Michał Górny wrote:
> On Sun, 2020-02-09 at 22:30 -0800, Zac Medico wrote:
>> Hi all (especially package owners in CC),
>>
>> In various packages we have inconsistent use of || preferences for
>> www-client/elinks, links, lynx, w3m, and w3mmee. This means that the
>> default preference depends on the combination of packages that one has
>> installed and the order that one has installed them, leading to
>> unpredictable results.
>>
>> Here is a list of relevant packages and their dependencies:
>>
>> app-text/docbook-sgml-utils: || ( www-client/lynx www-client/links
>> www-client/elinks virtual/w3m )
>> app-text/sgmltools-lite: || ( www-client/w3m www-client/lynx )
>> app-text/xmlto: || ( virtual/w3m www-client/lynx www-client/elinks )
>> dev-lang/mono: || ( www-client/links www-client/lynx )
>> mail-client/mutt: || ( www-client/lynx www-client/w3m www-client/elinks )
>> mail-client/neomutt: || ( www-client/lynx www-client/w3m www-client/elinks )
>> net-irc/irssi: || ( www-client/lynx www-client/elinks )
>> sys-fs/gt5: || ( www-client/links www-client/elinks www-client/lynx )
>> x11-base/xorg-server: || ( www-client/links www-client/lynx www-client/w3m )
>>
>> How about if we create some more virtuals to cover all of the relevant
>> cases?
> 
> I don't think that's a valid case for a virtual since those tools do not
> provide a consistent API for other packages.  It just happens that some
> packages explicitly support multiple choices, and this is exactly what
> || indicates.
> 
> The virtuals would really be arbitrary here.  Developers would
> repeatedly fail to use them because they wouldn't naturally expect
> the virtual to exist.

In that case, I suppose we'll have to apply consistency manually? Can we
all agree on a global order of preference for the relevant packages?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Inconsistent use of || preferences for www-client/elinks, links, lynx, w3m, and w3mmee

[Zac Medico]

Hi all (especially package owners in CC),

In various packages we have inconsistent use of || preferences for
www-client/elinks, links, lynx, w3m, and w3mmee. This means that the
default preference depends on the combination of packages that one has
installed and the order that one has installed them, leading to
unpredictable results.

Here is a list of relevant packages and their dependencies:

app-text/docbook-sgml-utils: || ( www-client/lynx www-client/links
www-client/elinks virtual/w3m )
app-text/sgmltools-lite: || ( www-client/w3m www-client/lynx )
app-text/xmlto: || ( virtual/w3m www-client/lynx www-client/elinks )
dev-lang/mono: || ( www-client/links www-client/lynx )
mail-client/mutt: || ( www-client/lynx www-client/w3m www-client/elinks )
mail-client/neomutt: || ( www-client/lynx www-client/w3m www-client/elinks )
net-irc/irssi: || ( www-client/lynx www-client/elinks )
sys-fs/gt5: || ( www-client/links www-client/elinks www-client/lynx )
x11-base/xorg-server: || ( www-client/links www-client/lynx www-client/w3m )

How about if we create some more virtuals to cover all of the relevant
cases?

We've already got virtual/w3m to cover this particular case:

|| ( www-client/w3m www-client/w3mmee )

We might want to use nested virtuals, in the same way that virtual/jdk
nests under virtual/jre.
-- 
Thanks,
Zac





signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: virtual/libcrypt for libcrypt.so implementation

[Zac Medico]

On 11/7/19 11:52 AM, Patrick McLean wrote:
> DEPEND="
>   elibc_glibc? ( || (
>   sys-libs/glibc[crypt(+)]

If a new version of glibc will remove crypt then it needs to be
something like:

|| ( >=sys-libs/glibc-2.30-r2[crypt(-)] 

signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [RFC] Disable --autounmask auto-unmasking keywords / masks by default

[Zac Medico]

On 10/9/19 7:57 PM, Kent Fredric wrote:
> On Wed, 9 Oct 2019 19:45:34 -0700
> Zac Medico  wrote:
> 
>> I'd prefer to disable --autounmask by default and include warnings about
>> harmful behavior in the documentation.
> 
> I think autounmasks behaviour with regards to USE flags is useful,
> (heh), its just everything else it does tends to violate stability
> assumptions.
> 
> For instance, I can see why automatically escalating "arch" to "~arch"
> may be useful in some cases, but its a very dangerous default,
> *especially* for perl.

We can add --autounmask-use= option that's enabled by default when
--autounmask is unspecified.

When --autounmask=y is specified, we'll have full autounmask behavior.

When --autounmask=n is specified, we'll have --autounmask-use=n behavior.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [RFC] Disable --autounmask auto-unmasking keywords / masks by default

[Zac Medico]

On 10/9/19 4:53 PM, Kent Fredric wrote:
> One of the recurring problems we face in #gentoo is end users coming
> to us with confusing problems, and their problems are exacerbated
> because their default workflow ended up with them unmasking some **
> version of perl.
> 
> There is already a bug for this behaviour [1], and comments say that
> portage doing this is "a bug", but the situations which it occurs in
> are hard to diagnose what the "real problem" is.
> 
> Much of the time, what has occurred is there was some other problem,
> and portage bodged its way around the real problem by choosing a
> solution that should be considered unacceptable, instead of presenting
> the real problem.
> 
> Some of the time, the cause is as simple as a single package being
> installed that isn't in the @world dependency graph any more, which is
> tripping up portage slot-rebuild behaviour.
> 
> In practice, what this currently means is that stable users end up
> installing *developmental/experimental* packages that exist only for
> experts and gentoo maintainers, and this is an unacceptable resolution.
> 
> If this behaviour was being triggered by anything other than portage's
> dependency resolver failing, it would be considered a serious QA
> violation.
> 
> Its understood that portage maintainers want to "fix" this behaviour so
> the problem doesn't occur, but until that can be done, the present
> default behaviour is actively harmful, and I suggest it be disabled by
> default until it can be guaranteed to give the right results.
> 
> 1: https://bugs.gentoo.org/658648
> 

I'd prefer to disable --autounmask by default and include warnings about
harmful behavior in the documentation.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 2/2] go-module-vendor.eclass: new eclass for go modules that do not vendor

[Zac Medico]

On 9/23/19 5:30 PM, William Hubbs wrote:
>> +# @FUNCTION: go-module-vendor_src_unpack
>> +# @DESCRIPTION:
>> +# Extract all archives in ${a} which are not nentioned in ${EGO_VENDOR}
>> +# to their usual locations then extract all archives mentioned in
>> +# ${EGO_VENDOR} to ${S}/vendor.
>> +go-module-vendor_src_unpack() {
>> +local f hash import line repo tarball vendor_uri
>> +if [[ -z "${EGO_VENDOR}" ]]; then
>> +die "EGO_VENDOR is not defined"
>> +fi
>> +
>> +vendor_uri="$(go-module-vendor_get_vendor_uri)"
>> +for f in $A; do
>> +[[ $vendor_uri == *"$f"* ]] && continue
>> +unpack $f
>> +done
>> +
>> +if [[ -d "${S}/vendor" ]]; then
>> +eerror "Upstream for ${P}.ebuild vendors dependencies."
>> +die "This ebuild should inherit go-module.eclass"
>> +fi
> 
> All,
> 
> I want to talk about the if block just above where I am writing.
> 
> If the vendor directory exists after the sources are unpacked, the idea
> is that upstream is vendoring their dependencies and we probably don't
> want to mess with the contents of the vendor directory in that case.
> 
> Mgorny, you suggested that there might be a valid use case for  being
> able to insert our own dependencies even when upstream bundles them for
> security. Something like that is an easy enough change (deleting the if
> block), but I want to know more about whether this is a strong case for
> it. My thought is that if the issue is reported to upstream, they should
> do a new release after updating their vendored dependencies, so this is
> more an upstream thing.
> 
> Thoughts? Is there a strong enough use case for messing with the bundled
> dependencies ourself?

If you feel like it would add unnecessary complexity, then it's probably
fine to leave that case unsupported. The worst case is that ebuild
maintainers will have to copy and modify the eclass function.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Underscores in USE flags

[Zac Medico]

On 9/20/19 8:46 AM, Mike Gilbert wrote:
> Recently, a large number of bugs were filed against packages that have
> USE flag names which contain underscores. Apparently PMS prohibits
> this except when the USE flag is part of a USE_EXPAND variable.
> 
> https://projects.gentoo.org/pms/7/pms.html#x1-23.1.4
> 
> I'm not certain when this text was added to PMS, or how many of the
> affected USE flags pre-date this policy.
> 
> Portage seems to have no issue dealing with underscores, so this
> doesn't seem to be solving any technical problem.
> 
> I am pretty sure that renaming a bunch of USE flags will cause some
> amount of end-user confusion, for very little benefit. Is enforcing
> this part of PMS really worth it?

If we take this underscore rule to its logical extreme, then we should
rename python_targets_python3_7 to python_targets_python3-7, yes?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

On 9/18/19 11:04 AM, Alec Warner wrote:
> 
> 
> On Wed, Sep 18, 2019 at 10:50 AM Michael Orlitzky  > wrote:
> 
> On 9/16/19 10:17 AM, William Hubbs wrote:
> > +
> > +# @FUNCTION: go-module_pkg_postinst
> > +# @DESCRIPTION:
> > +# Display a warning about security updates for Go programs.
> > +go-module_pkg_postinst() {
> > +     ewarn "${PN} is written in the Go programming language."
> > +     ewarn "Since this language is statically linked, security"
> > +     ewarn "updates will be handled in individual packages and
> will be"
> > +     ewarn "difficult for us to track as a distribution."
> > +     ewarn "For this reason, please update any go packages asap
> when new"
> > +     ewarn "versions enter the tree or go stable if you are
> running the"
> > +     ewarn "stable tree."
> > +}
> > +
> > +fi
> >
> 
> This word salad is 100% misinformation that gets tangled in itself
> trying to apologize for what we're about to do:
> 
>   * Go is not a "statically linked language." There's gccgo, and as Alec
>     pointed out, the official compiler has supported dynamic linking for
>     years now.
> 
> 
> I'm actually pretty fine with this wording, upstream has said not to
> dynamically link in these use cases.
>  
> 
> 
>   * Updating DOES NOT HELP AT ALL. That's the whole problem. You're
>     trying to make it sound like we haven't thrown people under a bus,
>     but saying "for this reason, please update..." is just misleading.
> 
> Here's what it should say:
> 
>   WARNING: due to a lack of manpower/interest, Go packages on Gentoo
>   are statically linked. Contrary to our existing policies and what
>   the website says, Go packages will never receive any security updates
>   on Gentoo. Use at your own risk!
> 
> 
> So if the package *maintainer* bumps each package every time it, or a
> dep has a security issue; then updating will work fine.
> I'm skeptical go maintainers are volunteering for this though.

There's a script here which helps to automate refresh of commit hashes
in EGO_VENDOR:

https://github.com/hsoft/gentoo-ego-vendor-update

Just now I've used it to refresh vendored dependencies in
net-misc/drive:


https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3993b893d4788beaad945bc82df0f4efd91ce697
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

On 9/17/19 7:10 AM, William Hubbs wrote:
> On Tue, Sep 17, 2019 at 07:36:07AM +0200, Michał Górny wrote:
>> On Mon, 2019-09-16 at 17:00 -0500, William Hubbs wrote:
>>> On Mon, Sep 16, 2019 at 11:50:12AM -0700, Zac Medico wrote:
>>>> On 9/16/19 11:35 AM, William Hubbs wrote:
>>>>> On Mon, Sep 16, 2019 at 11:01:38AM -0700, Zac Medico wrote:
>>>>>> For packages that I maintain, I'd prefer to continue using EGO_VENDOR to
>>>>>> even with packages using go.mod. I hope that this go-module.class will
>>>>>> not preclude this sort of usage. For example, the latest go-tools ebuild
>>>>>> uses EGO_VENDOR together with GOFLAGS="-mod=vendor":
>>>>>>
>>>>>> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cc6d401139526e2f9a6dbadbd31f0ff2387705f
>>>>>
>>>>> Can you elaborate on why you want to keep EGO_VENDOR?
>>>>>
>>>>> The "go mod vendor" command above downloads all the correct versions
>>>>> of the dependencies and puts them in the vendor directory, so I'm not
>>>>> sure why you would need the EGO_VENDOR variable.
>>>>
>>>> EGO_VENDOR eliminates to need to generate and host monolithic tarballs
>>>> containing vendored dependencies. It's more space-efficient in the sense
>>>> that each vendored dependency is stored in a separate tarball, so
>>>> multiple ebuilds can share the same tarball if the version of a
>>>> particular vendored dependency has not changed.
>>>
>>> I see what you are saying, but I haven't yet found a way to generate
>>> these separate tarballs that I'm comfortable with. Also, thinking about
>>> this, there will be many more tarballs on our mirrors if we store one
>>> dependency in each tarball than if we generate vendor tarballs that
>>> contain all dependencies for a package.
>>>
>>> I would consider this an enhancement to the eclass if you  still feel
>>> that we need it, but let me get the eclass into the tree first then we
>>> can work on that.
>>>
>>
>> That sounds like a bad idea.  If there are any potential enhancements
>> that can happen, I'd rather see them happen before there's a bunch of
>> ebuilds using the eclass in the wild, and potentially limiting possible
>> changes.
> 
> Like I just said on IRC, it would have been better if you responded in
> terms of discussing the enhancement itself.
> 
> The main blocker for me is that EGO_VENDOR is basically the same
> information as go.mod, but it isn't quite the same format.
> You can get close with "go list -m all", but EGO_VENDOR doesn't
> automatically handle imports that start with things like golang.org/x or
> gopkg.in; you have to manually fix those, and you would have to do that
> every time. That seems to be a lot of work for little gain.

It looks like it should not be too difficult to create a script that
will convert from go.mod to EGO_VENDOR format. For example, take this
go.mod file:

https://github.com/golang/tools/blob/master/go.mod

It contains a line like this:

golang.org/x/net v0.0.0-20190620200207-3b0461eec859

The part after the last hyphen corresponds to the commit hash which can
be used directly or expanded like this:

$ curl -sS https://api.github.com/repos/golang/net/commits/3b0461eec859 | 
jq -r .sha
3b0461eec859c4b73bb64fdc8285971fd33e3938

The github owner and repo names can resolved like this:

$ curl -sS https://golang.org/x/net | grep go-source
https://github.com/golang/net/ https://github.com/golang/net/tree/master{/dir} 
https://github.com/golang/net/blob/master{/dir}/{file}#L{line}";>

I've found that `go get` parses a similar meta element named "go-import"
here:


https://github.com/golang/go/blob/master/src/cmd/go/internal/get/discovery.go
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

On 9/16/19 11:26 AM, William Hubbs wrote:
> On Mon, Sep 16, 2019 at 10:48:14AM -0700, Zac Medico wrote:
>> On 9/16/19 7:17 AM, William Hubbs wrote:
>>> +BDEPEND=">=dev-lang/go-1.12"
>>> +
>>> +# The following go flags should be used for all go builds.
>>> +# -mod=vendor stopps downloading of dependencies from the internet.
>>> +# -v prints the names of packages as they are compiled
>>> +# -x prints commands as they are executed
>>> +export GOFLAGS="-mod=vendor -v -x"
>>
>> My experience with g-1.12.x was that you have to export GO111MODULE=on
>> or else GOFLAGS="-mod=vendor" triggers an error like this:
>>
>> build flag -mod=vendor only valid when using modules
> 
> I thought that was only if ${S} was in GOPATH, but I'll take a look real
> quick. If it is, would it be best to bump the BDEPEND or turn on the
> environment setting?

I've tested again, and in fact it appears that GOPATH="${S}" triggers
this error, so I guess it would be sufficient to ensure that GOPATH is
unset. Maybe it's cleaner to explicitly export GO111MODULE=on though.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

On 9/16/19 11:35 AM, William Hubbs wrote:
> On Mon, Sep 16, 2019 at 11:01:38AM -0700, Zac Medico wrote:
>> For packages that I maintain, I'd prefer to continue using EGO_VENDOR to
>> even with packages using go.mod. I hope that this go-module.class will
>> not preclude this sort of usage. For example, the latest go-tools ebuild
>> uses EGO_VENDOR together with GOFLAGS="-mod=vendor":
>>
>> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cc6d401139526e2f9a6dbadbd31f0ff2387705f
> 
> Can you elaborate on why you want to keep EGO_VENDOR?
> 
> The "go mod vendor" command above downloads all the correct versions
> of the dependencies and puts them in the vendor directory, so I'm not
> sure why you would need the EGO_VENDOR variable.

EGO_VENDOR eliminates to need to generate and host monolithic tarballs
containing vendored dependencies. It's more space-efficient in the sense
that each vendored dependency is stored in a separate tarball, so
multiple ebuilds can share the same tarball if the version of a
particular vendored dependency has not changed.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

On 9/16/19 7:17 AM, William Hubbs wrote:
> +# You will know the software you are packaging uses modules because
> +# it will have files named go.sum and go.mod in its top-level source
> +# directory. If it does not have these files, use the golang-* eclasses.
> +#
> +# If the software you are packaging uses modules, the next question is
> +# whether it has a directory named "vendor" at the top-level of the source 
> tree.
> +#
> +# If it doesn't, you need to create a tarball of what would be in the
> +# vendor directory and mirror it locally.
> +# If foo-1.0 is the name of your project and you have the tarball for it
> +# in your current directory,  this is done with the following commands:
> +#
> +# @CODE:
> +#
> +# tar -xf foo-1.0.tar.gz
> +# cd foo-1.0
> +# go mod vendor
> +# cd ..
> +# tar -acf foo-1.0-vendor.tar.gz foo-1.0/vendor

For packages that I maintain, I'd prefer to continue using EGO_VENDOR to
even with packages using go.mod. I hope that this go-module.class will
not preclude this sort of usage. For example, the latest go-tools ebuild
uses EGO_VENDOR together with GOFLAGS="-mod=vendor":

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cc6d401139526e2f9a6dbadbd31f0ff2387705f
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

On 9/16/19 7:17 AM, William Hubbs wrote:
> +BDEPEND=">=dev-lang/go-1.12"
> +
> +# The following go flags should be used for all go builds.
> +# -mod=vendor stopps downloading of dependencies from the internet.
> +# -v prints the names of packages as they are compiled
> +# -x prints commands as they are executed
> +export GOFLAGS="-mod=vendor -v -x"

My experience with g-1.12.x was that you have to export GO111MODULE=on
or else GOFLAGS="-mod=vendor" triggers an error like this:

build flag -mod=vendor only valid when using modules
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: go 1.13 and go modules

[Zac Medico]

On 9/9/19 10:34 AM, William Hubbs wrote:

> There is another option I want to try which is adding "go mod vendor" to
> src_unpack for go packages.

If you do that then it will violate FEATURES=network-sandbox (default)
unless you also do PROPERTIES+=" live".

We could add a separate PROPERTIES value for this, I've suggested it
before but both Michał Górny and Ulrich Mueller were against it:

https://archives.gentoo.org/gentoo-portage-dev/message/6d696661b29b6e2b8c82061e89e4718f
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] metadata/layout.conf: Allow specific PROPERTIES and RESTRICT values

[Zac Medico]

On 7/29/19 12:04 AM, Michał Górny wrote:
> On Sun, 2019-07-28 at 23:59 -0700, Zac Medico wrote:
>> Bug: https://bugs.gentoo.org/690786
>> ---
>>  metadata/layout.conf | 4 
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/metadata/layout.conf b/metadata/layout.conf
>> index 7ffc0ce6b72..c719c42513c 100644
>> --- a/metadata/layout.conf
>> +++ b/metadata/layout.conf
>> @@ -4,6 +4,10 @@
>>  # For details on this file, see the layout.conf section of the
>>  # portage(5) man page.
>>  
>> +# Allow specific PROPERTIES and RESTRICT values in ebuilds.
>> +allow-properties = interactive live
>> +allow-restrict = binchecks bindist fetch installsources mirror 
>> preserve-libs primaryuri splitdebug strip test userpriv
> 
> If I were to bikeshed, I'd go for following suit with eapis-*, i.e.:
> 
> properties-allowed = ...
> restrict-allowed = ...

Ok, done:

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5c92ab3ce076c9fb710c6cdcba71bae224215f20
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH] metadata/layout.conf: Allow specific PROPERTIES and RESTRICT values

[Zac Medico]

Bug: https://bugs.gentoo.org/690786
---
 metadata/layout.conf | 4 
 1 file changed, 4 insertions(+)

diff --git a/metadata/layout.conf b/metadata/layout.conf
index 7ffc0ce6b72..c719c42513c 100644
--- a/metadata/layout.conf
+++ b/metadata/layout.conf
@@ -4,6 +4,10 @@
 # For details on this file, see the layout.conf section of the
 # portage(5) man page.
 
+# Allow specific PROPERTIES and RESTRICT values in ebuilds.
+allow-properties = interactive live
+allow-restrict = binchecks bindist fetch installsources mirror preserve-libs 
primaryuri splitdebug strip test userpriv
+
 # manifest-hashes specify hashes used for new/updated entries
 # the current set went live on 2017-11-21, per 2017-11-12 Council meeting
 # 
https://archives.gentoo.org/gentoo-dev/message/ba2e5d9666ebd7e1bff1143485a37856
-- 
2.21.0

Re: [gentoo-dev] [PATCH] eclass: Assign PROPERTIES="live" in live eclasses.

[Zac Medico]

On 7/26/19 2:18 AM, Ulrich Müller wrote:
> Bug: https://bugs.gentoo.org/233589
> Signed-off-by: Ulrich Müller 
> ---
> 
> As previously discussed [1], add a "live" token to PROPERTIES in all
> live eclasses. This can also be used to tag live ebuilds that don't
> inherit any live eclass.
> 
> Corresponding updates for PMS [2] and Portage [3] are under way.
> 
> [1] 
> https://archives.gentoo.org/gentoo-dev/message/64b83155637bcad67478e2d2af276780
> [2] 
> https://archives.gentoo.org/gentoo-pms/message/1e1b5523a885ef4d90417a6176baa65b
> [3] 
> https://archives.gentoo.org/gentoo-portage-dev/message/8eab171cf105482062dda189ed2b0b59
> 
>  eclass/bzr.eclass| 4 +++-
>  eclass/cvs.eclass| 2 ++
>  eclass/darcs.eclass  | 4 +++-
>  eclass/git-2.eclass  | 4 +++-
>  eclass/git-r3.eclass | 4 +++-
>  eclass/golang-vcs.eclass | 4 +++-
>  eclass/mercurial.eclass  | 4 +++-
>  eclass/subversion.eclass | 2 ++
>  8 files changed, 22 insertions(+), 6 deletions(-)

Technically, they should do PROPERTIES+=" live" because PROPERTIES and
RESTRICT actually do not append automatically.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] What's going on with the tmpfiles eclasses?

[Zac Medico]

On 4/23/19 2:03 PM, Michael Orlitzky wrote:
> We have two eclasses with almost-identical functions for handling
> tmpfiles.d entries:
> 
>   1. systemd.eclass
> 
>  a. systemd_dotmpfilesd
>  b. systemd_newtmpfilesd
>  c. systemd_tmpfiles_create
> 
>   2. tmpfiles.eclass
> 
>  a. dotmpfiles
>  b. newtmpfiles
>  c. tmpfiles_process
> 
> The do/new functions are basically identical, while the create/process
> functions differ only in the fact that the one from tmpfiles.eclass
> supports opentmpfiles as well. Why do we have both? Couldn't the
> systemd.eclass ones be implemented in terms of the tmpfiles.eclass ones,
> and then deprecated (in favor of tmpfiles.eclass itself) in newer EAPIs?
> 
> Or am I missing something?

Note that systemd.eclass is lighter on dependencies, which is why I
chose it for the solution to bug 490676 [1] and bug 643386 [2] in the
sys-apps/portage ebuilds.

[1] https://bugs.gentoo.org/490676
[2] https://bugs.gentoo.org/643386
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: How to amend Gentoo ebuilds?

[Zac Medico]

On 1/7/19 2:11 PM, Oleh Kravchenko wrote:
> Hello Gentoo developers!
> 
> I would like to do minimal changes in already existed ebuilds,
> but I don't want to create and modify copy of ebuild.
> 
> Can we have in Gentoo Linux similar functionality like in Yocto?
> https://www.yoctoproject.org/docs/2.1/dev-manual/dev-manual.html#using-bbappend-files
> 

You can use ebuild phase hooks [1] in your bashrcs [2].

[1]
https://dev.gentoo.org/~zmedico/portage/doc/portage.html#config-bashrc-ebuild-phase-hooks
[2]
https://dev.gentoo.org/~zmedico/portage/doc/portage.html#config-bashrc-locations

-- 
Thanks,
Zac

Re: [gentoo-dev] [pre-GLEP r1] Gentoo binary package container format

[Zac Medico]

On 11/19/18 11:33 AM, Rich Freeman wrote:
> On Mon, Nov 19, 2018 at 2:21 PM Roy Bamford  wrote:
>>
>> "The archive members support optional OpenPGP signatures.
>> The implementations must allow the user to specify whether OpenPGP
>> signatures are to be expected in remotely fetched packages."
>>
>> Or can the user specify that only some elements need to be signed?
>>
>> Is it a problem if not all elements are signed with the same key?
>> That could happen if one person makes a  binpackage and someone
>> else updates the metadata.
>>
> 
> IMO this is going a bit into PM details for a GLEP that is about
> container formats.
> 
> Presumably any package manager is going to need to figure out what
> keys are/aren't valid and allow the user to configure this behavior.
> Users who want to go editing package innards will presumably adjust
> their package manager settings to accept their modifications, whether
> it means accepting their own sigs or disabling them.

With the GLEP as it is, the user *must* use a local signing key to sign
installed packages during the installation process if they want to be
able to verify signatures for installed packages at some point in the
future, since the binary package format does not provide a way to use
binary package signatures for this purpose.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format [gen...@jonesmz.com]

[Zac Medico]

On 11/18/18 6:51 PM, Rich Freeman wrote:
> On Sun, Nov 18, 2018 at 5:40 PM Zac Medico  wrote:
>>
>> On 11/18/18 1:55 PM, Rich Freeman wrote:
>>>
>>> My idea is to basically have portage generate a tag with all the info
>>> needed to identify the "right" package, take a hash of it, and then
>>> stick that in the filename.  Then when portage is looking for a binary
>>> package to use at install time it generates the same tag using the
>>> same algorithm and looks for a matching hash.
>>
>> We've already had this handled for a couple years now, via
>> FEATURES=binpkg-multi-instance.
> 
> According to the make.conf manpage this simply numbers builds.  So, if
> you build something twice with the same config you end up with two
> duplicate files (wasteful).  Presumably if you had a large collection
> of these packages portage would have to read the metadata within each
> one to figure out which one is appropriate to install.  That would be
> expensive if IO is slow, such as when fetching packages online
> on-demand.
> 
> But, it obviously is somewhat of an improvement for Roy's use case.
> 
> IMO using a content-hash of certain metadata would eliminate
> duplication, and based on filename alone it would be clear whether the
> sought-after binary package exists or not.  As with the build numbers
> you couldn't tell from filename inspection what packages you have, but
> if you know what you want you could immediately find it.  IMO trying
> to cram all that metadata into a filename to make them more
> transparent isn't a good idea, and using hashes lets the user set
> their own policy regarding flexibility.  Heck, you could auto-gen
> symlinks for subsets of metadata (ie, the same file could be linked
> from a file that specifies its USE flags but not its CFLAGS, so it
> would be found if either an exact hit on CFLAGS was sought or if
> CFLAGS were considered unimportant).
> 
> But, I'm certainly not suggesting that you're not allowed to go to bed
> until you've built it.  :)

The existing ${PKGDIR}/Packages file optimizes metadata access for both
local an remote access, and performs well for reasonable numbers of
packages.

If you insist on mixing binary packages in the same ${PKGDIR} for a
large number of alternative configurations, then it will not scale
unless you create a way to send your local configuration to the server
so that it can select the relevant package list for you.

However, bear in mind that mixing alternative configurations in the same
${PKGDIR} might lead to undesirable results if there is anything
relevant that is unaccounted for in the package metadata. Possible
unaccounted things may include:

1) glibc version the package was built against
2) symbols and/or sonames not accounted for by slot operator dependencies
3) soname dependencies (--usepkgonly + --ignore-soname-deps=n handles this)
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format [gen...@jonesmz.com]

[Zac Medico]

On 11/18/18 1:55 PM, Rich Freeman wrote:
> On Sun, Nov 18, 2018 at 4:10 PM Roy Bamford  wrote:
>>
>> Replying off list because I am not on the whitelist.
> 
> That seems odd.
> 
>> 1) append a uuid to each filename. Generated when the bin package file is 
>> generated.
>> 2) encode the hostname of the machine that generated the file
>> 3) encode the use flags in the filename.
> 
> So, I brought up this same issue in the earlier discussion and it was
> considered out of scope, and I think this is fair.  The GLEP does not
> specify filename, and IMO the standard for what goes INSIDE the file
> will work just fine with any future enhancements that address exactly
> this use case.
> 
> Besides your case of building for a cluster, another use case is
> having a central binary repo that portage could check and utilize when
> a user's preferences happen to match what is pre-built.
> 
> I suggest we start a different thread for any additional discussion of
> this use case.  I was thinking and it probably wouldn't be super-hard
> to actually start building something like this.  But, I don't want to
> derail this GLEP as I don't see any reason designing something like
> this needs to hold up the binary package format.  Both the existing
> and proposed binary package formats will encode any metadata needed by
> the package manager inside the file, and the only extension we need is
> to encode identifying info in the filename.
> 
> My idea is to basically have portage generate a tag with all the info
> needed to identify the "right" package, take a hash of it, and then
> stick that in the filename.  Then when portage is looking for a binary
> package to use at install time it generates the same tag using the
> same algorithm and looks for a matching hash.  If a hit is found then
> it reads the complete metadata in the file and applies all the sanity
> checks it already does.  Generating of binary packages with the hash
> cold be made optional, and portage could also be configured to first
> look for the matching hash, then fall back to the existing naming
> convention, so that it would be compatible with existing generic
> names.  So, users would get a choice as to whether they want to build
> up a library of these packages, or just have each build overwrite the
> last.
> 
> Then the next step would be to allow these files to be fetched from a
> binary repo optionally, and then finally we'd need tools to create the
> repo.  But, this step isn't needed for your use case.  With the proper
> optional switches you could utilize as much of this scheme as you
> like.
> 
> Also, you could optionally choose how much you want portage to encode
> in the tag and look for.  Are you very fussy and only want a binary
> package with matching CFLAGS/USE/whatever?  Or is just matching
> USE/arch/etc enough?  Some of the existing portage options could
> potentially be re-used here.

We've already had this handled for a couple years now, via
FEATURES=binpkg-multi-instance.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] install-qa-check.d: Support QA{,_STRICT}_INSTALL_PATHS variables (bug 670902)

[Zac Medico]

On 11/12/18 9:44 PM, Michael Orlitzky wrote:
> On 11/12/2018 06:47 PM, Zac Medico wrote:
>>>
>>> The idea being, to put it in the right place by default, and let people
>>> override it with EXTRA_ECONF if they really want to download random
>>> binaries from strangers and run them.
>>
>> I recommend to add /nix to the whitelist because this is the default
>> location for all operating systems, as shown consistently throughout the
>> installation instructions found at
>> https://nixos.org/nix/manual/#chap-installation.
> 
> I mean... I know... my argument is not that they don't tell you to do
> something dumb. If you really want the official experience, you can
> close your eyes, cross your fingers, say a prayer to RMS, and then
> follow their installation instructions:
> 
>  $ curl https://nixos.org/nix/install | sh
> 
> The fact that some people choose to use portage to install it probably
> means that they were looking for something a little less yee-haw.

The benefit of using the ebuild is the same as always, it allows for
automated upgrade, uninstall, etc.

> We
> trust the package manager to not let ebuilds do dumb things to our
> systems: no surprise network access, no random filesystem reads/writes,
> reliable uninstalls, things installed in sensible paths, etc. We
> shouldn't make exceptions to those policies without a good reason.

What does any of this have to do with the nix having a file store under
/nix?

>> The nix manual also has this explicit warning in the "Building Nix from
>> Source" section found at https://nixos.org/nix/manual/#sec-building-source:
>>
>>> Warning: It is best not to change the Nix store from its default,
>> since doing
>>> so makes it impossible to use pre-built binaries from the standard Nixpkgs
>>> channels — that is, all packages will need to be built from source.
> 
> Do I have to be that guy who suggests that if people don't want to build
> from source, then maybe they took a wrong turn back at distrowatch?

They're able to choose between building from source or using prebuilt
packages, and changing the location of the /nix file store is unnecessary.

> You can override the nix store location with EXTRA_ECONF if you really
> want to dump stuff in /nix. At which point the warning is just telling
> you what's up: you're doing something wrong.

What's inherently wrong about nix having a file store under /nix? Is
this purely about FHS?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] install-qa-check.d: Support QA{,_STRICT}_INSTALL_PATHS variables (bug 670902)

[Zac Medico]

On 11/12/18 2:34 PM, Michael Orlitzky wrote:
> On 11/12/2018 04:06 PM, Zac Medico wrote:
>> On 11/12/18 12:57 PM, Michael Orlitzky wrote:
>>> On 11/12/2018 03:33 PM, Zac Medico wrote:
>>>>
>>>> QA_INSTALL_PATHS=( /nix )
>>>
>>> That really, really, really doesn't belong there.
>>
>> I'm open to suggestions for alternatives. Ideas?
>>
> 
> /var/lib/nix?
> 
> The idea being, to put it in the right place by default, and let people
> override it with EXTRA_ECONF if they really want to download random
> binaries from strangers and run them.

I recommend to add /nix to the whitelist because this is the default
location for all operating systems, as shown consistently throughout the
installation instructions found at
https://nixos.org/nix/manual/#chap-installation.

The nix manual also has this explicit warning in the "Building Nix from
Source" section found at https://nixos.org/nix/manual/#sec-building-source:

> Warning: It is best not to change the Nix store from its default,
since doing
> so makes it impossible to use pre-built binaries from the standard Nixpkgs
> channels — that is, all packages will need to be built from source.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] install-qa-check.d: Support QA{,_STRICT}_INSTALL_PATHS variables (bug 670902)

[Zac Medico]

On 11/12/18 12:57 PM, Michael Orlitzky wrote:
> On 11/12/2018 03:33 PM, Zac Medico wrote:
>>
>> QA_INSTALL_PATHS=( /nix )
>>
> 
> That really, really, really doesn't belong there.

I'm open to suggestions for alternatives. Ideas?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH] install-qa-check.d: Support QA{,_STRICT}_INSTALL_PATHS variables (bug 670902)

[Zac Medico]

The QA_INSTALL_PATHS variable exempts paths from "unexpected paths"
warnings generated by metadata/install-qa-check.d/08gentoo-paths.
If the QA_STRICT_INSTALL_PATHS variable is set then any exemptions
in QA_INSTALL_PATHS are ignored.

Since we probably don't want to whitelist /nix for all ebuilds,
this patch will allow us to whitelist it with a setting like this
in global scope of the nix ebuilds:

QA_INSTALL_PATHS=( /nix )

Bug: https://bugs.gentoo.org/670902
Signed-off-by: Zac Medico 
---
 metadata/install-qa-check.d/08gentoo-paths | 21 +
 1 file changed, 21 insertions(+)

diff --git a/metadata/install-qa-check.d/08gentoo-paths 
b/metadata/install-qa-check.d/08gentoo-paths
index 3ee887df08f..a3289b112dd 100644
--- a/metadata/install-qa-check.d/08gentoo-paths
+++ b/metadata/install-qa-check.d/08gentoo-paths
@@ -60,6 +60,27 @@ gentoo_path_check() {
 
${shopt_save}
 
+   if [[ ${#bad_paths[@]} -gt 0 && ${QA_INSTALL_PATHS} &&
+   ${QA_STRICT_INSTALL_PATHS-unset} == unset ]]; then
+   local filtered_paths=()
+   local grep_args=()
+   local qa_install_paths
+   if [[ $(declare -p QA_INSTALL_PATHS) == "declare -a "* ]]; then
+   qa_install_paths=( "${QA_INSTALL_PATHS[@]}" )
+   else
+   set -f
+   qa_install_paths=( ${QA_INSTALL_PATHS} )
+   set +f
+   fi
+   for x in "${qa_install_paths[@]}"; do
+   grep_args+=( -e "^/${x#/}\$" )
+   done
+   while read -r -d ''; do
+   [[ ${REPLY} ]] && filtered_paths+=( "${REPLY}" )
+   done < <(printf -- '%s\0' "${bad_paths[@]}" | grep -zv 
"${grep_args[@]}")
+   bad_paths=( "${filtered_paths[@]}" )
+   fi
+
# report
# --
if [[ -n ${bad_paths[@]} ]]; then
-- 
2.18.1

Re: [gentoo-dev] Add GOBIN to ENV_UNSET in make.defaults

[Zac Medico]

On 10/20/2018 04:21 AM, Pacho Ramos wrote:
> It seems that random values in GOBIN can affect the building of some packages:
> https://bugs.gentoo.org/631776
> https://bugs.gentoo.org/636506
> https://bugs.gentoo.org/638572
> 
> I would then append it to ENV_UNSET in make.defaults to get that variable 
> unset
> without needing to do the same for every ebuild that could be affected by this
> 
> Any issues against this?

Seems reasonable, since the only purpose of GOBIN is to override the
directory where 'go install' will install a command. If we unset it
unconditionally, it means that the location will predictably default to
GOPATH/bin, which is exactly what we want.

We could handle it in the golang-build_src_install function, but that
wouldn't cover things that call 'go install' via a script or Makefile.

> Thanks
>
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Zac Medico]

On 10/01/2018 11:23 AM, Zac Medico wrote:
> On 10/01/2018 11:16 AM, Michał Górny wrote:
>> On Mon, 2018-10-01 at 19:23 +0200, Andreas Sturmlechner wrote:
>>> On Montag, 1. Oktober 2018 17:48:16 CEST Michał Górny wrote:
>>>> On Mon, 2018-10-01 at 08:19 -0700, Zac Medico wrote:
>>>>> /usr/share/doc level directories
>>>>> 
>>>>> /usr/share/doc/${PF}
>>>>>
>>>>> The first bug report [2] is for qt-core, which installs documentation
>>>>> into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
>>>>> includes ebuild revision such as -r1, -r2, and so on).
>>>>
>>>> No, it doesn't.  There's no /usr/share/doc/qtcore-5.11.1 on my system.
>>>
>>> This is coming from dev-qt/qt-docs.
>>
>> Nope, still not /usr/share/doc/qt*core*-...
>>
>>>  It is a problem because any other package 
>>> building QCH API docs with cross-references to Qt API needs to install 
>>> below 
>>> this path, and will generate the same QA warning (currently 
>>> kde-frameworks/* 
>>> does this).
>>
>> Yes.  That is why I believe that hardcoding the exception in every
>> package is simply wrong.  Wouldn't it be cleaner to account for the path
>> in the QA check?
> 
> There may be cases where we want to fix the ebuild to use
> /usr/share/doc/${PF} though, right?

Until this QA check has adjustable whitelist support, we can consider it
an unstable work in progress. Therefore, I'd like for the QA team to
move it gentoo/metadata/install-qa-check.d/08gentoo-paths until it has
matured. It's safe to commit it to the gentoo repository now, and it
will become active when portage's internal copy is removed. I've filed
this bug report to track progress:

https://bugs.gentoo.org/667604
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Zac Medico]

On 10/01/2018 01:04 PM, Sergei Trofimovich wrote:
> On Mon, 1 Oct 2018 08:19:29 -0700
> Zac Medico  wrote:
> 
>> Hi all,
>>
>> The ~arch version of portage hs a new QA check that reports installation
>> of files outside of directories that have been whitelisted [1]. The
>> current whitelist includes:
>>
>> directories common to / and /usr
>> 
>> bin lib lib32 lib64 libx32 sbin
>>
>> top level directories
>> 
>> boot dev etc opt srv usr var
>>
>> /usr level directories
>> 
>> include libexec share src
>>
>> /usr/share/doc level directories
>> 
>> /usr/share/doc/${PF}
> 
> As this will break existing ebuilds I'd suggest guarding it against
> next EAPI. Out of top of by head the change will break at least
> crossdev outright. At least for two reasons:
> 1. building a gcc cross-compiler usually refers to $SYSROOT/sys-include.
>'git grep sys-include' will show a bunch of ebuilds that create that
>symlinks outside the space.
> 2. cross-building target libc is currently done on the host's emerge
>   and installs into /usr/$CTARGET/.
> 
> I think whitelist would be able to cover these use cases as they are
> a function of ${CTARGET} (or $CATEGORY) value.

The QA check currently does not die, it only calls eqawarn. So the worst
case is a message like this:

> QA: other
> The ebuild is installing to one or more unexpected paths:
>
>   /usr/share/doc/qt-5.11.1
>
> Please fix the ebuild to use correct FHS/Gentoo policy paths.

It will be annoying if this message triggers for every package installed
by crossdev, so we should do something about that.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Zac Medico]

On 10/01/2018 11:16 AM, Michał Górny wrote:
> On Mon, 2018-10-01 at 19:23 +0200, Andreas Sturmlechner wrote:
>> On Montag, 1. Oktober 2018 17:48:16 CEST Michał Górny wrote:
>>> On Mon, 2018-10-01 at 08:19 -0700, Zac Medico wrote:
>>>> /usr/share/doc level directories
>>>> 
>>>> /usr/share/doc/${PF}
>>>>
>>>> The first bug report [2] is for qt-core, which installs documentation
>>>> into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
>>>> includes ebuild revision such as -r1, -r2, and so on).
>>>
>>> No, it doesn't.  There's no /usr/share/doc/qtcore-5.11.1 on my system.
>>
>> This is coming from dev-qt/qt-docs.
> 
> Nope, still not /usr/share/doc/qt*core*-...
> 
>>  It is a problem because any other package 
>> building QCH API docs with cross-references to Qt API needs to install below 
>> this path, and will generate the same QA warning (currently kde-frameworks/* 
>> does this).
> 
> Yes.  That is why I believe that hardcoding the exception in every
> package is simply wrong.  Wouldn't it be cleaner to account for the path
> in the QA check?

There may be cases where we want to fix the ebuild to use
/usr/share/doc/${PF} though, right?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Zac Medico]

On 10/01/2018 08:48 AM, Michał Górny wrote:
> On Mon, 2018-10-01 at 08:19 -0700, Zac Medico wrote:
>> I have created a patch that will allow ebuilds to whitelist directories
>> by setting a QA_INSTALL_PATHS variable [3], however @mgorny said:
> 
> @mgorny was seriously concerned about trigger-happy patch authors who
> find it urgent to silence QA warnings without even bothering to properly
> discuss the problem.

Discussion takes time, and I'm a very busy person, sorry about that.
Anyway, eventually we are forced to discuss those issues that really matter.

> And is continuously concerned about people who
> want something but don't bother starting the discussion, and instead
> rely on somebody else to start the discussion, even without having
> the appropriate knowledge on what the problem is in the first place.

> In other words, this is something that should be discussed on a case-by-
> case basis.  Not the usual Gentoo thing of 'I don't like this QA
> warning, let's silence it quickly and go on ignoring everyone, whether
> my package is broken or not'.

Maybe we should implement a policy requiring gentoo-dev mailing list
proposals for QA_INSTALL_PATHS usage in gentoo ebuilds (similar to the
policy for news items and eclass changes)?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] RFC: Portage QA check for FHS/Gentoo policy paths, for top-level dirs and /usr/share/doc

[Zac Medico]

Hi all,

The ~arch version of portage hs a new QA check that reports installation
of files outside of directories that have been whitelisted [1]. The
current whitelist includes:

directories common to / and /usr

bin lib lib32 lib64 libx32 sbin

top level directories

boot dev etc opt srv usr var

/usr level directories

include libexec share src

/usr/share/doc level directories

/usr/share/doc/${PF}

The first bug report [2] is for qt-core, which installs documentation
into  /usr/share/doc/${PN}-${PV} instead of /usr/share/doc/${PF} (${PF}
includes ebuild revision such as -r1, -r2, and so on).

I have created a patch that will allow ebuilds to whitelist directories
by setting a QA_INSTALL_PATHS variable [3], however @mgorny said:

On Mon, Oct 1, 2018 at 12:34 AM Michał Górny  wrote:
> QA NAK on this.  The purpose of QA checks is not for ebuilds to
> officially override them because histerical raisins.

[1]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=d5f97eaa464736a454c8ad410f4acd4fccdf2324
[2] https://bugs.gentoo.org/667280
[3] https://bugs.gentoo.org/667378
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: removal of keyword arm-linux

[Zac Medico]

On 08/26/2018 08:52 PM, Benda Xu wrote:
> The gentoo-x86 CVS history shows the commit author was mainly
> Zac (zmedico).

I have an arm prefix that I have used on chromebooks, and I'd be happy
to use the arm keyword instead of arm-linux.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] Add section about defining "Test Dependencies"

[Zac Medico]

On 08/26/2018 01:08 AM, Michał Górny wrote:
> On Sun, 2018-08-26 at 00:58 -0700, Zac Medico wrote:
>> On 08/26/2018 12:39 AM, Michał Górny wrote:
>>> On Sun, 2018-08-26 at 00:28 -0700, Zac Medico wrote:
>>>> On 08/25/2018 07:11 PM, Andrew Savchenko wrote:
>>>>> On Sat, 25 Aug 2018 14:24:02 -0400 Mike Gilbert wrote:
>>>>>> On Sat, Aug 25, 2018 at 1:41 AM Andrew Savchenko  
>>>>>> wrote:
>>>>>>>
>>>>>>> On Fri, 24 Aug 2018 14:24:06 -0400 Mike Gilbert wrote:
>>>>>>>> ---
>>>>>>>>  general-concepts/dependencies/text.xml | 38 ++
>>>>>>>>  1 file changed, 38 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/general-concepts/dependencies/text.xml 
>>>>>>>> b/general-concepts/dependencies/text.xml
>>>>>>>> index 2f10380..64be9dc 100644
>>>>>>>> --- a/general-concepts/dependencies/text.xml
>>>>>>>> +++ b/general-concepts/dependencies/text.xml
>>>>>>>> @@ -578,6 +578,44 @@ valid.
>>>>>>>>  
>>>>>>>>  
>>>>>>>>
>>>>>>>> +
>>>>>>>> +Test Dependencies
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +Packages often have optional dependencies that are needed only when 
>>>>>>>> running
>>>>>>>> +tests. These should be specified in DEPEND behind a USE flag. Often, 
>>>>>>>> the
>>>>>>>> +'test' USE flag is used for this purpose.
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +Since testing will likely fail when test dependencies are not 
>>>>>>>> installed, the
>>>>>>>> +test phase should be disabled in this case. This may be accomplished 
>>>>>>>> via USE
>>>>>>>> +conditionals in the RESTRICT variable.
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +If other optional features must be enabled/disabled when testing, 
>>>>>>>> REQUIRED_USE
>>>>>>>> +may be set to express this.
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +# Define some USE flags
>>>>>>>> +IUSE="debug test"
>>>>>>>> +
>>>>>>>> +# Disable test phase when test USE flag is disabled
>>>>>>>> +RESTRICT="!test? ( test )"
>>>>>>>
>>>>>>> I do not understand why we need this useless code. If test USE flag
>>>>>>> is disabled, tests must be disabled as well. It is PM's job and
>>>>>>> there is no need to put this obvious stuff into each ebuild with
>>>>>>> tests and extra deps. I see no reason to support running src_test()
>>>>>>> with USE="-test".
>>>>>>
>>>>>> PMS does not specify that behavior (skipping src_test with USE=-test).
>>>>>> It is better to define the requrement explicitly rather than relying
>>>>>> on a Portage-specific behavior.
>>>>>
>>>>> Then PMS should be fixed. Putting useless code in thousands
>>>>> of ebuilds due to bureaucratic reasons is ridiculous. Having strict
>>>>> conformance to the PMS is good, but common sense should still be
>>>>> considered.
>>>>
>>>> Since PMS doesn't specify the behavior of FEATURES, I suppose we could
>>>> make FEATURES=test imply RESTRICT="!test? ( test )". Would there be any
>>>> drawbacks to that?
>>>
>>> Except for Portage once again silently letting developers pass with non-
>>> PMS behavior, and making other PMs fail with the ebuilds?
>>
>> Is it really sane for other PMs to behave that way though?
>>
>> In PMS it says src_test "may be disabled by user too, using a
>> PM-specific mechanism", which doesn't explicitly rule out behavior based
>> on USE settings.
> 
> It doesn't rule out disabling based on the phase of the moon either, or
> my mood which is degrading badly with this discussion.

Well, using package.use for this doesn't really make sense within the
context of ebuilds that don't have IUSE=test, which is an argument for
people to use package.env as a consistent means to selectively disable
unit tests.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] Add section about defining "Test Dependencies"

[Zac Medico]

On 08/26/2018 12:39 AM, Michał Górny wrote:
> On Sun, 2018-08-26 at 00:28 -0700, Zac Medico wrote:
>> On 08/25/2018 07:11 PM, Andrew Savchenko wrote:
>>> On Sat, 25 Aug 2018 14:24:02 -0400 Mike Gilbert wrote:
>>>> On Sat, Aug 25, 2018 at 1:41 AM Andrew Savchenko  
>>>> wrote:
>>>>>
>>>>> On Fri, 24 Aug 2018 14:24:06 -0400 Mike Gilbert wrote:
>>>>>> ---
>>>>>>  general-concepts/dependencies/text.xml | 38 ++
>>>>>>  1 file changed, 38 insertions(+)
>>>>>>
>>>>>> diff --git a/general-concepts/dependencies/text.xml 
>>>>>> b/general-concepts/dependencies/text.xml
>>>>>> index 2f10380..64be9dc 100644
>>>>>> --- a/general-concepts/dependencies/text.xml
>>>>>> +++ b/general-concepts/dependencies/text.xml
>>>>>> @@ -578,6 +578,44 @@ valid.
>>>>>>  
>>>>>>  
>>>>>>
>>>>>> +
>>>>>> +Test Dependencies
>>>>>> +
>>>>>> +
>>>>>> +
>>>>>> +Packages often have optional dependencies that are needed only when 
>>>>>> running
>>>>>> +tests. These should be specified in DEPEND behind a USE flag. Often, the
>>>>>> +'test' USE flag is used for this purpose.
>>>>>> +
>>>>>> +
>>>>>> +
>>>>>> +Since testing will likely fail when test dependencies are not 
>>>>>> installed, the
>>>>>> +test phase should be disabled in this case. This may be accomplished 
>>>>>> via USE
>>>>>> +conditionals in the RESTRICT variable.
>>>>>> +
>>>>>> +
>>>>>> +
>>>>>> +If other optional features must be enabled/disabled when testing, 
>>>>>> REQUIRED_USE
>>>>>> +may be set to express this.
>>>>>> +
>>>>>> +
>>>>>> +
>>>>>> +# Define some USE flags
>>>>>> +IUSE="debug test"
>>>>>> +
>>>>>> +# Disable test phase when test USE flag is disabled
>>>>>> +RESTRICT="!test? ( test )"
>>>>>
>>>>> I do not understand why we need this useless code. If test USE flag
>>>>> is disabled, tests must be disabled as well. It is PM's job and
>>>>> there is no need to put this obvious stuff into each ebuild with
>>>>> tests and extra deps. I see no reason to support running src_test()
>>>>> with USE="-test".
>>>>
>>>> PMS does not specify that behavior (skipping src_test with USE=-test).
>>>> It is better to define the requrement explicitly rather than relying
>>>> on a Portage-specific behavior.
>>>
>>> Then PMS should be fixed. Putting useless code in thousands
>>> of ebuilds due to bureaucratic reasons is ridiculous. Having strict
>>> conformance to the PMS is good, but common sense should still be
>>> considered.
>>
>> Since PMS doesn't specify the behavior of FEATURES, I suppose we could
>> make FEATURES=test imply RESTRICT="!test? ( test )". Would there be any
>> drawbacks to that?
> 
> Except for Portage once again silently letting developers pass with non-
> PMS behavior, and making other PMs fail with the ebuilds?

Is it really sane for other PMs to behave that way though?

In PMS it says src_test "may be disabled by user too, using a
PM-specific mechanism", which doesn't explicitly rule out behavior based
on USE settings.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] Add section about defining "Test Dependencies"

[Zac Medico]

On 08/25/2018 07:11 PM, Andrew Savchenko wrote:
> On Sat, 25 Aug 2018 14:24:02 -0400 Mike Gilbert wrote:
>> On Sat, Aug 25, 2018 at 1:41 AM Andrew Savchenko  wrote:
>>>
>>> On Fri, 24 Aug 2018 14:24:06 -0400 Mike Gilbert wrote:
 ---
  general-concepts/dependencies/text.xml | 38 ++
  1 file changed, 38 insertions(+)

 diff --git a/general-concepts/dependencies/text.xml 
 b/general-concepts/dependencies/text.xml
 index 2f10380..64be9dc 100644
 --- a/general-concepts/dependencies/text.xml
 +++ b/general-concepts/dependencies/text.xml
 @@ -578,6 +578,44 @@ valid.
  
  

 +
 +Test Dependencies
 +
 +
 +
 +Packages often have optional dependencies that are needed only when 
 running
 +tests. These should be specified in DEPEND behind a USE flag. Often, the
 +'test' USE flag is used for this purpose.
 +
 +
 +
 +Since testing will likely fail when test dependencies are not installed, 
 the
 +test phase should be disabled in this case. This may be accomplished via 
 USE
 +conditionals in the RESTRICT variable.
 +
 +
 +
 +If other optional features must be enabled/disabled when testing, 
 REQUIRED_USE
 +may be set to express this.
 +
 +
 +
 +# Define some USE flags
 +IUSE="debug test"
 +
 +# Disable test phase when test USE flag is disabled
 +RESTRICT="!test? ( test )"
>>>
>>> I do not understand why we need this useless code. If test USE flag
>>> is disabled, tests must be disabled as well. It is PM's job and
>>> there is no need to put this obvious stuff into each ebuild with
>>> tests and extra deps. I see no reason to support running src_test()
>>> with USE="-test".
>>
>> PMS does not specify that behavior (skipping src_test with USE=-test).
>> It is better to define the requrement explicitly rather than relying
>> on a Portage-specific behavior.
> 
> Then PMS should be fixed. Putting useless code in thousands
> of ebuilds due to bureaucratic reasons is ridiculous. Having strict
> conformance to the PMS is good, but common sense should still be
> considered.

Since PMS doesn't specify the behavior of FEATURES, I suppose we could
make FEATURES=test imply RESTRICT="!test? ( test )". Would there be any
drawbacks to that?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] use.desc: Improve description of USE=test

[Zac Medico]

On 08/26/2018 12:14 AM, Zac Medico wrote:
> On 08/24/2018 04:56 PM, Francesco Riosa wrote:
>>
>>
>> Il giorno sab 25 ago 2018 alle ore 01:45 Zac Medico > <mailto:zmed...@gentoo.org>> ha scritto:
>>
>> On 08/24/2018 04:14 PM, Francesco Riosa wrote:
>> >
>> > Il 24/08/18 19:08, Mike Gilbert ha scritto:
>> >> On Fri, Aug 24, 2018 at 10:45 AM Kent Fredric > <mailto:ken...@gentoo.org>> wrote:
>> >>> On Fri, 24 Aug 2018 10:27:01 -0400
>> >>> Mike Gilbert mailto:flop...@gentoo.org>> wrote:
>> >>>
>> >>>> If you want to define behavior that can be relied upon in
>> ebuilds, it
>> >>>> should be specified in PMS. PMS does not define any meaning for the
>> >>>> "test" USE flag.
>> >>> We should eschew idealism about how the world *should* behave,
>> and avoid
>> >>> making portage a steaming garbage heap in order to comply with a
>> >>> terrible PMS specification of a heavily used feature.
>> >> Portage still works just fine for most people who would enable
>> >> FEATURES=test. Stop exaggerating.
>> >>
>> > People enabling FEATURES=test on most but not all packages had some
>> > troubles and they need to disable them in both package.{env,use}
>> > I'd like to have a flag to reinstate the previous portage
>> behaviour, but
>> > that's probably too late
>>
>> Would a FEATURES setting that implies RESTRICT="!test? ( test )" for all
>> ebuilds do what you want?
>>
>>
>> "want" is too much,
>> but it would have been pleasant to have something to keep portage
>> dependancies working as before.
>> "testrespectuse" or "usetestrespectfeatures" come to mind or another
>> command line option.
> 
> Maybe you are not using the latest version of portage? The current
> behavior with sys-apps/portage-2.3.48 seems pretty sane to me, for
> example see this behavior with dev-python/setuptools-40.0.0 which has
> empty RESTRICT:
> 
> $ emerge -pvq --nodeps setuptools
> [ebuild   R   ] dev-python/setuptools-40.0.0  USE="-test" 
> PYTHON_TARGETS="pypy pypy3 python2_7 python3_4 python3_5 python3_6 python3_7" 
> $ FEATURES=test emerge -pvq --nodeps setuptools
> [ebuild   R   ] dev-python/setuptools-40.0.0  USE="test*" 
> PYTHON_TARGETS="pypy pypy3 python2_7 python3_4 python3_5 python3_6 python3_7" 
> $ FEATURES=test USE=-test emerge -pvq --nodeps setuptools
> [ebuild   R   ] dev-python/setuptools-40.0.0  USE="-test" 
> PYTHON_TARGETS="pypy pypy3 python2_7 python3_4 python3_5 python3_6 python3_7" 

Oh, I see that src_test runs even with USE=-test there, so I see why
people are complaining. Anyway, an implicit RESTRICT="!test? ( test )"
setting would solve that.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] use.desc: Improve description of USE=test

[Zac Medico]

On 08/24/2018 04:56 PM, Francesco Riosa wrote:
> 
> 
> Il giorno sab 25 ago 2018 alle ore 01:45 Zac Medico  <mailto:zmed...@gentoo.org>> ha scritto:
> 
> On 08/24/2018 04:14 PM, Francesco Riosa wrote:
> >
> > Il 24/08/18 19:08, Mike Gilbert ha scritto:
> >> On Fri, Aug 24, 2018 at 10:45 AM Kent Fredric  <mailto:ken...@gentoo.org>> wrote:
> >>> On Fri, 24 Aug 2018 10:27:01 -0400
> >>> Mike Gilbert mailto:flop...@gentoo.org>> wrote:
> >>>
> >>>> If you want to define behavior that can be relied upon in
> ebuilds, it
> >>>> should be specified in PMS. PMS does not define any meaning for the
> >>>> "test" USE flag.
> >>> We should eschew idealism about how the world *should* behave,
> and avoid
> >>> making portage a steaming garbage heap in order to comply with a
> >>> terrible PMS specification of a heavily used feature.
> >> Portage still works just fine for most people who would enable
> >> FEATURES=test. Stop exaggerating.
> >>
> > People enabling FEATURES=test on most but not all packages had some
> > troubles and they need to disable them in both package.{env,use}
> > I'd like to have a flag to reinstate the previous portage
> behaviour, but
> > that's probably too late
> 
> Would a FEATURES setting that implies RESTRICT="!test? ( test )" for all
> ebuilds do what you want?
> 
> 
> "want" is too much,
> but it would have been pleasant to have something to keep portage
> dependancies working as before.
> "testrespectuse" or "usetestrespectfeatures" come to mind or another
> command line option.

Maybe you are not using the latest version of portage? The current
behavior with sys-apps/portage-2.3.48 seems pretty sane to me, for
example see this behavior with dev-python/setuptools-40.0.0 which has
empty RESTRICT:

$ emerge -pvq --nodeps setuptools
[ebuild   R   ] dev-python/setuptools-40.0.0  USE="-test" PYTHON_TARGETS="pypy 
pypy3 python2_7 python3_4 python3_5 python3_6 python3_7" 
$ FEATURES=test emerge -pvq --nodeps setuptools
[ebuild   R   ] dev-python/setuptools-40.0.0  USE="test*" PYTHON_TARGETS="pypy 
pypy3 python2_7 python3_4 python3_5 python3_6 python3_7" 
$ FEATURES=test USE=-test emerge -pvq --nodeps setuptools
[ebuild   R   ] dev-python/setuptools-40.0.0  USE="-test" PYTHON_TARGETS="pypy 
pypy3 python2_7 python3_4 python3_5 python3_6 python3_7" 
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] use.desc: Improve description of USE=test

[Zac Medico]

On 08/24/2018 04:14 PM, Francesco Riosa wrote:
> 
> Il 24/08/18 19:08, Mike Gilbert ha scritto:
>> On Fri, Aug 24, 2018 at 10:45 AM Kent Fredric  wrote:
>>> On Fri, 24 Aug 2018 10:27:01 -0400
>>> Mike Gilbert  wrote:
>>>
 If you want to define behavior that can be relied upon in ebuilds, it
 should be specified in PMS. PMS does not define any meaning for the
 "test" USE flag.
>>> We should eschew idealism about how the world *should* behave, and avoid
>>> making portage a steaming garbage heap in order to comply with a
>>> terrible PMS specification of a heavily used feature.
>> Portage still works just fine for most people who would enable
>> FEATURES=test. Stop exaggerating.
>>
> People enabling FEATURES=test on most but not all packages had some
> troubles and they need to disable them in both package.{env,use}
> I'd like to have a flag to reinstate the previous portage behaviour, but
> that's probably too late

Would a FEATURES setting that implies RESTRICT="!test? ( test )" for all
ebuilds do what you want?
-- 
Thanks,
Zac

Re: [gentoo-dev] [PATCH] meson.eclass: require at least meson-0.41.1

[Zac Medico]

On 07/23/2018 07:56 AM, Mike Gilbert wrote:
> On Sun, Jul 22, 2018 at 11:27 PM Zac Medico  wrote:
>>
>> Require newer meson in order to avoid build failures triggered
>> if >=meson-0.41.1 is not installed soon enough. For example,
>> I experienced bug 649264 because I upgraded xorg-proto and
>> libxshmfence packages before meson.
>>
>> Fixes: https://bugs.gentoo.org/649264
> 
> I suggest jumping to 0.45.1 (latest stable on all archs). Feel free to
> push that.

Pushed:

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bbe649a35dc8e06e55abfdf8e98a743d1e50d5a0

-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] meson.eclass: require at least meson-0.41.1

[Zac Medico]

On 07/23/2018 01:08 AM, Mart Raudsepp wrote:
> Ühel kenal päeval, P, 22.07.2018 kell 20:27, kirjutas Zac Medico:
>> Require newer meson in order to avoid build failures triggered
>> if >=meson-0.41.1 is not installed soon enough. For example,
>> I experienced bug 649264 because I upgraded xorg-proto and
>> libxshmfence packages before meson.
>>
>> Fixes: https://bugs.gentoo.org/649264
> 
> Closes, not Fixes?

Closes is more accurate, yes.

>> ---
>>  eclass/meson.eclass | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/eclass/meson.eclass b/eclass/meson.eclass
>> index f2202a04593..2523bec57ac 100644
>> --- a/eclass/meson.eclass
>> +++ b/eclass/meson.eclass
>> @@ -59,7 +59,7 @@ EXPORT_FUNCTIONS src_configure src_compile src_test
>> src_install
>>  if [[ -z ${_MESON_ECLASS} ]]; then
>>  _MESON_ECLASS=1
>>  
>> -MESON_DEPEND=">=dev-util/meson-0.40.0
>> +MESON_DEPEND=">=dev-util/meson-0.41.1
> 
> By my understanding this should be 0.44.1, not 0.41.1.
> At least 0.43.0 is one of the broken versions.

0.44.1 is what I meant to type, yes.

> 0.44.1 is also the lowest available version in tree; raising
> MESON_DEPEND to that would as a side-effect not require ebuilds to have
> their own meson depend, when they need newer versions (>=0.41 was
> common in gnome 3.26, might be >=0.44 in newer).

Great!
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] [PATCH] meson.eclass: require at least meson-0.41.1

[Zac Medico]

Require newer meson in order to avoid build failures triggered
if >=meson-0.41.1 is not installed soon enough. For example,
I experienced bug 649264 because I upgraded xorg-proto and
libxshmfence packages before meson.

Fixes: https://bugs.gentoo.org/649264
---
 eclass/meson.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/meson.eclass b/eclass/meson.eclass
index f2202a04593..2523bec57ac 100644
--- a/eclass/meson.eclass
+++ b/eclass/meson.eclass
@@ -59,7 +59,7 @@ EXPORT_FUNCTIONS src_configure src_compile src_test 
src_install
 if [[ -z ${_MESON_ECLASS} ]]; then
 _MESON_ECLASS=1
 
-MESON_DEPEND=">=dev-util/meson-0.40.0
+MESON_DEPEND=">=dev-util/meson-0.41.1
>=dev-util/ninja-1.7.2"
 
 # @ECLASS-VARIABLE: MESON_AUTO_DEPEND
-- 
2.13.6

Re: [gentoo-dev] Adding USE=udev to linux profiles

[Zac Medico]

On 07/21/2018 12:01 AM, Dennis Schridde wrote:
> On Friday, 20 July 2018 08:25:05 CEST Michael Orlitzky wrote:
>> On 07/20/2018 02:12 AM, Mart Raudsepp wrote:
>>> Ok, I can see that point of view for make.conf.
>>> I can't agree with changes in other profiles though, as other profile
>>> will fall under the same category in USE_ORDER (in fact, it's the same
>>> thing, as the end USE from "defaults" comes from your selected profile
>>> and it's "parent" cascade, not taken from linux profile). But maybe you
>>> have it tested and know it's a problem. Have you?
>>
>> Yes, when you set USE=foo in profile A and USE="-foo" in profile A/B,
>> the end result is USE="foo -foo" which is the same thing as USE="-foo".
>> The default portage USE_ORDER makes the profile flags more important
>> than IUSE defaults, so the USE="-foo" from the profile clobbers any
>> IUSE="+foo" defaults.
>>
>> If no default was set in profile A, then no override would be necessary
>> in profile A/B, and the resulting USE="" would not override the IUSE
>> default.
> 
> It appears to me that the difficulty stems from use flags being tri-state 
> (enabled, disabled, unset), but us having only operators to enable (+) and 
> disable (-) them in make.conf and profiles.
> 
> What about adding a third operator, e.g. `^`, that resets a use flag to the 
> unset state?

Sure, why not? So ^flag would mean that the flag state propagates from
the settings in IUSE. It's also conceivable that we could add a way for
profiles to modify the effective IUSE defaults, via new operators in
package.use or by introducing a new file such as package.use.default.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: moving default location of portage tree

[Zac Medico]

On 07/09/2018 03:27 PM, M. J. Everitt wrote:
> On 09/07/18 23:12, Zac Medico wrote:
>> On 07/09/2018 02:34 PM, Kristian Fiskerstrand wrote:
>>> I'd mostly argue any such change should only affect new systems
>>>
>> Yes, changing defaults for existing systems would be annoying.
>>
>> My recommendation is to have catalyst set the new defaults in the stage
>> tarballs.
>>
>> When sys-apps/portage changes its internal defaults, I'd like for the
>> upgrade process to call a tool that generates configuration files when
>> necessary to ensure that the existing paths remain constant.
> I think it should be possible for RelEng to make a start on catalyst
> updates - is there anything that would inhibit going ahead with this,
> potentially?

No, nothing. Whatever catalyst puts it the default config will become
our new default.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: moving default location of portage tree

[Zac Medico]

On 07/09/2018 02:34 PM, Kristian Fiskerstrand wrote:
> 
> I'd mostly argue any such change should only affect new systems
>

Yes, changing defaults for existing systems would be annoying.

My recommendation is to have catalyst set the new defaults in the stage
tarballs.

When sys-apps/portage changes its internal defaults, I'd like for the
upgrade process to call a tool that generates configuration files when
necessary to ensure that the existing paths remain constant.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: moving default location of portage tree

[Zac Medico]

On 07/09/2018 01:07 PM, Zac Medico wrote:
> On 07/09/2018 01:00 PM, William Hubbs wrote:
>> On Mon, Jul 09, 2018 at 08:36:33PM +0200, Ulrich Mueller wrote:
>>>>>>>> On Mon, 9 Jul 2018, Rich Freeman wrote:
>>>
>>>> I'd also consider /var/cache here as well.  FHS specifically suggests
>>>> using it for web caches and the like (let's set aside the issue with
>>>> making that global), though for the most part it is more metadata
>>>> caching.  A key principle is that it can be wiped without loss of
>>>> data, and I think that is generally true for the repository since it
>>>> can be synced.
>>>
>>> I don't think that criterium is fulfilled, because you cannot easily
>>> restore the previous state after it's been wiped. At least not when
>>> syncing from a rsync mirror (which may have been updated in the mean
>>> time).
>>  
>> The criteria for /var/cache do not require being able to restore the
>> exact previous state; they just require that the application be able to
>> regenerate or restore the data , so they are definitely fulfilled.[1].
> 
> If it's an rsync tree then we cannot restore the precise state, for
> example you might not be able to rebuild one of your installed packages
> if the corresponding ebuild has been removed upstream.

Whoops I didn't mean to simply repeat what Ulrich said. My point is that
the spirit of the FHS might be that "nothing is lost", but you certainly
can lose some valuable state if it's an rsync tree.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: moving default location of portage tree

[Zac Medico]

On 07/09/2018 01:00 PM, William Hubbs wrote:
> On Mon, Jul 09, 2018 at 08:36:33PM +0200, Ulrich Mueller wrote:
>>> On Mon, 9 Jul 2018, Rich Freeman wrote:
>>
>>> I'd also consider /var/cache here as well.  FHS specifically suggests
>>> using it for web caches and the like (let's set aside the issue with
>>> making that global), though for the most part it is more metadata
>>> caching.  A key principle is that it can be wiped without loss of
>>> data, and I think that is generally true for the repository since it
>>> can be synced.
>>
>> I don't think that criterium is fulfilled, because you cannot easily
>> restore the previous state after it's been wiped. At least not when
>> syncing from a rsync mirror (which may have been updated in the mean
>> time).
>  
> The criteria for /var/cache do not require being able to restore the
> exact previous state; they just require that the application be able to
> regenerate or restore the data , so they are definitely fulfilled.[1].

If it's an rsync tree then we cannot restore the precise state, for
example you might not be able to rebuild one of your installed packages
if the corresponding ebuild has been removed upstream.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

On 07/08/2018 02:59 PM, Zac Medico wrote:
> On 07/08/2018 02:50 PM, Aaron W. Swenson wrote:
>> On July 8, 2018 5:38:48 PM EDT, Zac Medico  wrote:
>>
>> On 07/08/2018 02:18 PM, Michał Górny wrote:
>>
>> W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico
>> napisał:
>>
>> On 07/08/2018 01:18 PM, Zac Medico wrote:
>>
>> On 07/08/2018 01:08 PM, Michał Górny wrote:
>>
>> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700,
>> użytkownik Zac Medico
>> napisał:
>>
>> On 07/08/2018 11:42 AM, Michał Górny wrote:
>>
>>         W dniu nie, 08.07.2018 o godzinie 11∶04 
>> -0700, użytkownik Zac Medico
>> napisał:
>>
>> On 07/08/2018 06:56 AM, Michał Górny wrote:
>>
>> W dniu nie, 08.07.2018 o godzinie
>> 15∶02 +0200, użytkownik Kristian
>> Fiskerstrand napisał:
>>
>> On 07/08/2018 08:53 AM, Michał
>> Górny wrote:
>>
>> Is safe git syncing
>> implemented already? If not,
>> maybe finish it first and
>> cover both with a single
>> news item. Git is going to
>> be more efficient here, so
>> people may want to learn
>> they have an alternative.
>>
>>
>> Why complicate things, and
>> increase wait for something that
>> benefits
>> most users, just to give
>> alternatives to a few using
>> non-default sync
>> mechanism. Securing git
>> distribution is a whole
>> different ballpark.
>>
>>
>>
>> Let me rephrase. Let's say I'm using
>> rsync. This new feature is
>> something positive but it breaks my
>> use case (for one of the listed
>> reasons -- overlayfs, inode use,
>> small fs cache). After reading this
>> news item, I learn that my only
>> option is to disable the new feature.
>>
>> Now, I would appreciate being told
>> that there's an alternate sync method
>> that handles secure updates without
>> having all those drawbacks.
>>
>>
>> The thing is, the normal git tree
>> doesn't even provide pre-generated
>> metadata, and I see then gentoo-mirror
>> repo that provides metadata does
>> not have commits signed with an release key:
>>
>> 
>> https://github.com/gentoo-mirror/gentoo/commits/stable
>>
>> So I'm really not comfortable
>> recommending git to anyone at this point.
>>
>>
>> Wrong twice.
>>
>> Firstly, the canonical URL is:
>>
>> 
>> https://anongit.gentoo.org/git/repo/sync/gentoo.git
>> (https://gitweb.gentoo.org/repo/sync/gentoo.git)
>>
>> Secondly, the merge commits (i.e. top
>> commits that are verified
>> by Portage) are signed by dedicated key that

[gentoo-dev] News Item: Portage rsync hardlink support [v2]

[Zac Medico]

In v2 there's a suggestion to use git for better efficiency than rsync,
as requested by Michał Górny:

Title: Portage rsync hardlink support
Author: Zac Medico 
Posted: 2018-07-11
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: sys-apps/portage

For users of the rsync tree, beginning with sys-apps/portage-2.3.42,
the default behavior for sync operations will use hardlinks in order
to ensure that a repository remains in a valid state if something
goes wrong [1]. For example, if signature verification fails during a
sync operation, the new hardlink behavior will preserve the previous
state of the repository.

The new behavior may conflict with configurations that restrict the
use of hardlinks, such as overlay filesystems. Therefore, users will
have to set "sync-allow-hardlinks = no" in repos.conf if they have
a configuration that restricts the use of hardlinks, but this should
not be very common:

[DEFAULT]
sync-allow-hardlinks = no

Note that it possible to sync more efficiently using git [2] instead
of rsync, though git consumes an increasing amount of disk space over
time because shallow pull is currently not supported [3].

[1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync
--link-dest to implement atomic repository updates (and abort if
signature verification fails)
[2] https://wiki.gentoo.org/wiki/Portage_Security#git-mirror_repo
[3] https://bugs.gentoo.org/552814 sync using git pull --depth=1 by
default
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

On 07/08/2018 02:50 PM, Aaron W. Swenson wrote:
> On July 8, 2018 5:38:48 PM EDT, Zac Medico  wrote:
> 
> On 07/08/2018 02:18 PM, Michał Górny wrote:
> 
> W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico
> napisał:
> 
> On 07/08/2018 01:18 PM, Zac Medico wrote:
> 
> On 07/08/2018 01:08 PM, Michał Górny wrote:
> 
> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700,
>     użytkownik Zac Medico
> napisał:
> 
> On 07/08/2018 11:42 AM, Michał Górny wrote:
> 
> W dniu nie, 08.07.2018 o godzinie 11∶04 
> -0700, użytkownik Zac Medico
> napisał:
> 
> On 07/08/2018 06:56 AM, Michał Górny wrote:
> 
> W dniu nie, 08.07.2018 o godzinie
> 15∶02 +0200, użytkownik Kristian
> Fiskerstrand napisał:
> 
> On 07/08/2018 08:53 AM, Michał
> Górny wrote:
> 
> Is safe git syncing
> implemented already? If not,
> maybe finish it first and
> cover both with a single
> news item. Git is going to
> be more efficient here, so
> people may want to learn
> they have an alternative.
> 
> 
> Why complicate things, and
> increase wait for something that
> benefits
> most users, just to give
> alternatives to a few using
> non-default sync
> mechanism. Securing git
> distribution is a whole
> different ballpark.
> 
> 
> 
> Let me rephrase. Let's say I'm using
> rsync. This new feature is
> something positive but it breaks my
> use case (for one of the listed
> reasons -- overlayfs, inode use,
> small fs cache). After reading this
> news item, I learn that my only
> option is to disable the new feature.
> 
> Now, I would appreciate being told
> that there's an alternate sync method
> that handles secure updates without
> having all those drawbacks.
> 
> 
> The thing is, the normal git tree
> doesn't even provide pre-generated
> metadata, and I see then gentoo-mirror
> repo that provides metadata does
> not have commits signed with an release key:
> 
> 
> https://github.com/gentoo-mirror/gentoo/commits/stable
> 
> So I'm really not comfortable
> recommending git to anyone at this point.
> 
> 
> Wrong twice.
> 
> Firstly, the canonical URL is:
> 
> 
> https://anongit.gentoo.org/git/repo/sync/gentoo.git
> (https://gitweb.gentoo.org/repo/sync/gentoo.git)
> 
> Secondly, the merge commits (i.e. top
> commits that are verified
> by Portage) are signed by dedicated key that
> is part of the infra key
> set. In other words, it works out of the box.
> 
> 
> Is there any documentation that shows users how
> to migrate to git, and
> what the pros and cons might be? Maybe its
> worthy 

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

On 07/08/2018 02:18 PM, Michał Górny wrote:
> W dniu nie, 08.07.2018 o godzinie 14∶11 -0700, użytkownik Zac Medico
> napisał:
>> On 07/08/2018 01:18 PM, Zac Medico wrote:
>>> On 07/08/2018 01:08 PM, Michał Górny wrote:
>>>> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico
>>>> napisał:
>>>>> On 07/08/2018 11:42 AM, Michał Górny wrote:
>>>>>> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
>>>>>> napisał:
>>>>>>> On 07/08/2018 06:56 AM, Michał Górny wrote:
>>>>>>>> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
>>>>>>>> Fiskerstrand napisał:
>>>>>>>>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>>>>>>>>> Is safe git syncing implemented already? If not, maybe finish it 
>>>>>>>>>> first and cover both with a single news item. Git is going to be 
>>>>>>>>>> more efficient here, so people may want to learn they have an 
>>>>>>>>>> alternative.
>>>>>>>>>
>>>>>>>>> Why complicate things, and increase wait for something that benefits
>>>>>>>>> most users, just to give alternatives to a few using non-default sync
>>>>>>>>> mechanism. Securing git distribution is a whole different ballpark.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Let me rephrase.  Let's say I'm using rsync.  This new feature is
>>>>>>>> something positive but it breaks my use case (for one of the listed
>>>>>>>> reasons -- overlayfs, inode use, small fs cache).  After reading this
>>>>>>>> news item, I learn that my only option is to disable the new feature.
>>>>>>>>
>>>>>>>> Now, I would appreciate being told that there's an alternate sync 
>>>>>>>> method
>>>>>>>> that handles secure updates without having all those drawbacks.
>>>>>>>
>>>>>>> The thing is, the normal git tree doesn't even provide pre-generated
>>>>>>> metadata, and I see then gentoo-mirror repo that provides metadata does
>>>>>>> not have commits signed with an release key:
>>>>>>>
>>>>>>> https://github.com/gentoo-mirror/gentoo/commits/stable
>>>>>>>
>>>>>>> So I'm really not comfortable recommending git to anyone at this point.
>>>>>>
>>>>>> Wrong twice.
>>>>>>
>>>>>> Firstly, the canonical URL is:
>>>>>>
>>>>>>   https://anongit.gentoo.org/git/repo/sync/gentoo.git
>>>>>>   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
>>>>>>
>>>>>> Secondly, the merge commits (i.e. top commits that are verified
>>>>>> by Portage) are signed by dedicated key that is part of the infra key
>>>>>> set.  In other words, it works out of the box.
>>>>>
>>>>> Is there any documentation that shows users how to migrate to git, and
>>>>> what the pros and cons might be? Maybe its worthy of its own news item.
>>>>
>>>> Maybe.  I don't really know, and don't think it's a good idea to show 30
>>>> news item of things users might like on every new Gentoo install.
>>>
>>> Well if instructions for setting up git sync and associated pros/cons
>>> are not documented anywhere then I won't advise anyone to use it.
>>
>> I've attempted to configure it for myself, and this is what it does:
>>
>>  * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
>>  * Refreshing keys from keyserver ...
>>[ ok ]
>>  * No valid signature found: unable to verify signature (missing key?)
>>
> 
> Please report a bug and attach your configuration along with keyring
> version.

It works after upgrading to openpgp-keys-gentoo-release-20180706 from
openpgp-keys-gentoo-release-20180323.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

On 07/08/2018 01:18 PM, Zac Medico wrote:
> On 07/08/2018 01:08 PM, Michał Górny wrote:
>> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico
>> napisał:
>>> On 07/08/2018 11:42 AM, Michał Górny wrote:
>>>> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
>>>> napisał:
>>>>> On 07/08/2018 06:56 AM, Michał Górny wrote:
>>>>>> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
>>>>>> Fiskerstrand napisał:
>>>>>>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>>>>>>> Is safe git syncing implemented already? If not, maybe finish it first 
>>>>>>>> and cover both with a single news item. Git is going to be more 
>>>>>>>> efficient here, so people may want to learn they have an alternative.
>>>>>>>
>>>>>>> Why complicate things, and increase wait for something that benefits
>>>>>>> most users, just to give alternatives to a few using non-default sync
>>>>>>> mechanism. Securing git distribution is a whole different ballpark.
>>>>>>>
>>>>>>
>>>>>> Let me rephrase.  Let's say I'm using rsync.  This new feature is
>>>>>> something positive but it breaks my use case (for one of the listed
>>>>>> reasons -- overlayfs, inode use, small fs cache).  After reading this
>>>>>> news item, I learn that my only option is to disable the new feature.
>>>>>>
>>>>>> Now, I would appreciate being told that there's an alternate sync method
>>>>>> that handles secure updates without having all those drawbacks.
>>>>>
>>>>> The thing is, the normal git tree doesn't even provide pre-generated
>>>>> metadata, and I see then gentoo-mirror repo that provides metadata does
>>>>> not have commits signed with an release key:
>>>>>
>>>>> https://github.com/gentoo-mirror/gentoo/commits/stable
>>>>>
>>>>> So I'm really not comfortable recommending git to anyone at this point.
>>>>
>>>> Wrong twice.
>>>>
>>>> Firstly, the canonical URL is:
>>>>
>>>>   https://anongit.gentoo.org/git/repo/sync/gentoo.git
>>>>   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
>>>>
>>>> Secondly, the merge commits (i.e. top commits that are verified
>>>> by Portage) are signed by dedicated key that is part of the infra key
>>>> set.  In other words, it works out of the box.
>>>
>>> Is there any documentation that shows users how to migrate to git, and
>>> what the pros and cons might be? Maybe its worthy of its own news item.
>>
>> Maybe.  I don't really know, and don't think it's a good idea to show 30
>> news item of things users might like on every new Gentoo install.
> 
> Well if instructions for setting up git sync and associated pros/cons
> are not documented anywhere then I won't advise anyone to use it.

I've attempted to configure it for myself, and this is what it does:

 * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
 * Refreshing keys from keyserver ...
   [ ok ]
 * No valid signature found: unable to verify signature (missing key?)


-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

On 07/08/2018 01:08 PM, Michał Górny wrote:
> W dniu nie, 08.07.2018 o godzinie 11∶57 -0700, użytkownik Zac Medico
> napisał:
>> On 07/08/2018 11:42 AM, Michał Górny wrote:
>>> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
>>> napisał:
>>>> On 07/08/2018 06:56 AM, Michał Górny wrote:
>>>>> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
>>>>> Fiskerstrand napisał:
>>>>>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>>>>>> Is safe git syncing implemented already? If not, maybe finish it first 
>>>>>>> and cover both with a single news item. Git is going to be more 
>>>>>>> efficient here, so people may want to learn they have an alternative.
>>>>>>
>>>>>> Why complicate things, and increase wait for something that benefits
>>>>>> most users, just to give alternatives to a few using non-default sync
>>>>>> mechanism. Securing git distribution is a whole different ballpark.
>>>>>>
>>>>>
>>>>> Let me rephrase.  Let's say I'm using rsync.  This new feature is
>>>>> something positive but it breaks my use case (for one of the listed
>>>>> reasons -- overlayfs, inode use, small fs cache).  After reading this
>>>>> news item, I learn that my only option is to disable the new feature.
>>>>>
>>>>> Now, I would appreciate being told that there's an alternate sync method
>>>>> that handles secure updates without having all those drawbacks.
>>>>
>>>> The thing is, the normal git tree doesn't even provide pre-generated
>>>> metadata, and I see then gentoo-mirror repo that provides metadata does
>>>> not have commits signed with an release key:
>>>>
>>>> https://github.com/gentoo-mirror/gentoo/commits/stable
>>>>
>>>> So I'm really not comfortable recommending git to anyone at this point.
>>>
>>> Wrong twice.
>>>
>>> Firstly, the canonical URL is:
>>>
>>>   https://anongit.gentoo.org/git/repo/sync/gentoo.git
>>>   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
>>>
>>> Secondly, the merge commits (i.e. top commits that are verified
>>> by Portage) are signed by dedicated key that is part of the infra key
>>> set.  In other words, it works out of the box.
>>
>> Is there any documentation that shows users how to migrate to git, and
>> what the pros and cons might be? Maybe its worthy of its own news item.
> 
> Maybe.  I don't really know, and don't think it's a good idea to show 30
> news item of things users might like on every new Gentoo install.

Well if instructions for setting up git sync and associated pros/cons
are not documented anywhere then I won't advise anyone to use it.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

On 07/08/2018 11:42 AM, Michał Górny wrote:
> W dniu nie, 08.07.2018 o godzinie 11∶04 -0700, użytkownik Zac Medico
> napisał:
>> On 07/08/2018 06:56 AM, Michał Górny wrote:
>>> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
>>> Fiskerstrand napisał:
>>>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>>>> Is safe git syncing implemented already? If not, maybe finish it first 
>>>>> and cover both with a single news item. Git is going to be more efficient 
>>>>> here, so people may want to learn they have an alternative.
>>>>
>>>> Why complicate things, and increase wait for something that benefits
>>>> most users, just to give alternatives to a few using non-default sync
>>>> mechanism. Securing git distribution is a whole different ballpark.
>>>>
>>>
>>> Let me rephrase.  Let's say I'm using rsync.  This new feature is
>>> something positive but it breaks my use case (for one of the listed
>>> reasons -- overlayfs, inode use, small fs cache).  After reading this
>>> news item, I learn that my only option is to disable the new feature.
>>>
>>> Now, I would appreciate being told that there's an alternate sync method
>>> that handles secure updates without having all those drawbacks.
>>
>> The thing is, the normal git tree doesn't even provide pre-generated
>> metadata, and I see then gentoo-mirror repo that provides metadata does
>> not have commits signed with an release key:
>>
>> https://github.com/gentoo-mirror/gentoo/commits/stable
>>
>> So I'm really not comfortable recommending git to anyone at this point.
> 
> Wrong twice.
> 
> Firstly, the canonical URL is:
> 
>   https://anongit.gentoo.org/git/repo/sync/gentoo.git
>   (https://gitweb.gentoo.org/repo/sync/gentoo.git)
> 
> Secondly, the merge commits (i.e. top commits that are verified
> by Portage) are signed by dedicated key that is part of the infra key
> set.  In other words, it works out of the box.

Is there any documentation that shows users how to migrate to git, and
what the pros and cons might be? Maybe its worthy of its own news item.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

On 07/08/2018 06:56 AM, Michał Górny wrote:
> W dniu nie, 08.07.2018 o godzinie 15∶02 +0200, użytkownik Kristian
> Fiskerstrand napisał:
>> On 07/08/2018 08:53 AM, Michał Górny wrote:
>>> Is safe git syncing implemented already? If not, maybe finish it first and 
>>> cover both with a single news item. Git is going to be more efficient here, 
>>> so people may want to learn they have an alternative.
>>
>> Why complicate things, and increase wait for something that benefits
>> most users, just to give alternatives to a few using non-default sync
>> mechanism. Securing git distribution is a whole different ballpark.
>>
> 
> Let me rephrase.  Let's say I'm using rsync.  This new feature is
> something positive but it breaks my use case (for one of the listed
> reasons -- overlayfs, inode use, small fs cache).  After reading this
> news item, I learn that my only option is to disable the new feature.
> 
> Now, I would appreciate being told that there's an alternate sync method
> that handles secure updates without having all those drawbacks.

The thing is, the normal git tree doesn't even provide pre-generated
metadata, and I see then gentoo-mirror repo that provides metadata does
not have commits signed with an release key:

https://github.com/gentoo-mirror/gentoo/commits/stable

So I'm really not comfortable recommending git to anyone at this point.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

On 07/08/2018 02:28 AM, Toralf Förster wrote:
> On 07/08/2018 08:08 AM, Zac Medico wrote:
>> Please review.
>>
>> Title: Portage rsync hardlink support
>> Author: Zac Medico 
>> Posted: 2018-07-11
>> Revision: 1
>> News-Item-Format: 2.0
>> Display-If-Installed: sys-apps/portage
>> IMO there's another heads-up for users having an unsual configuration:
> 
> So we do speak about files under /usr/portage itself and not about that
> dir (==changing its inode number), right?
> 
> B/c  otherwise there's another heads-up for people bind-mounting
> /usr/portage onto chrooted images.

This case will continue to work because the download happens in a
subdirectory, and after it's verified we call rsync again to apply the
changes to the top-level directory.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

On 07/08/2018 02:15 AM, Michał Górny wrote:
> Dnia 8 lipca 2018 09:14:06 CEST, Zac Medico  napisał(a):
>> On 07/07/2018 11:53 PM, Michał Górny wrote:
>>> Dnia 8 lipca 2018 08:08:31 CEST, Zac Medico 
>> napisał(a):
>>>> Please review.
>>>>
>>>> Title: Portage rsync hardlink support
>>>> Author: Zac Medico 
>>>> Posted: 2018-07-11
>>>> Revision: 1
>>>> News-Item-Format: 2.0
>>>> Display-If-Installed: sys-apps/portage
>>>>
>>>> For users of the rsync tree, beginning with sys-apps/portage-2.3.42,
>>>> the default behavior for sync operations will use hardlinks in order
>>>> to ensure that a repository remains in a valid state if something
>>>> goes wrong [1]. For example, if signature verification fails during
>> a
>>>> sync operation, the new hardlink behavior will preserve the previous
>>>> state of the repository.
>>>>
>>>> The new behavior may conflict with configurations that restrict the
>>>> use of hardlinks, such as overlay filesystems. Therefore, users will
>>>> have to set "sync-allow-hardlinks = no" in repos.conf if they have
>>>> a configuration that restricts the use of hardlinks, but this should
>>>> not be very common:
>>>>
>>>> [DEFAULT]
>>>> sync-allow-hardlinks = no
>>>>
>>>> [1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync
>>>>--link-dest to implement atomic repository updates (and abort if
>>>>signature verification fails)
>>>
>>> Is safe git syncing implemented already? If not, maybe finish it
>> first and cover both with a single news item. Git is going to be more
>> efficient here, so people may want to learn they have an alternative.
>>
>> Yeah there's already a patch for git sync [1] but I'd rather not make
>> this news item more complicated than it needs to be. I wouldn't have
>> bothered with a news item except that I want to give people some
>> warning
>> in case they are using overlayfs [2]. I think the efficiency difference
>> between rsync and git here are pretty negligible for most people.
> 
> Are you sure about that? That might have been the case so far but this 
> hardlink tree may actually tip the balance.

Even if it takes twice a long (which it doesn't), the difference is
negligible for most people because they usually don't sync more than
once per day.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

On 07/07/2018 11:53 PM, Michał Górny wrote:
> Dnia 8 lipca 2018 08:08:31 CEST, Zac Medico  napisał(a):
>> Please review.
>>
>> Title: Portage rsync hardlink support
>> Author: Zac Medico 
>> Posted: 2018-07-11
>> Revision: 1
>> News-Item-Format: 2.0
>> Display-If-Installed: sys-apps/portage
>>
>> For users of the rsync tree, beginning with sys-apps/portage-2.3.42,
>> the default behavior for sync operations will use hardlinks in order
>> to ensure that a repository remains in a valid state if something
>> goes wrong [1]. For example, if signature verification fails during a
>> sync operation, the new hardlink behavior will preserve the previous
>> state of the repository.
>>
>> The new behavior may conflict with configurations that restrict the
>> use of hardlinks, such as overlay filesystems. Therefore, users will
>> have to set "sync-allow-hardlinks = no" in repos.conf if they have
>> a configuration that restricts the use of hardlinks, but this should
>> not be very common:
>>
>> [DEFAULT]
>> sync-allow-hardlinks = no
>>
>> [1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync
>>--link-dest to implement atomic repository updates (and abort if
>>signature verification fails)
> 
> Is safe git syncing implemented already? If not, maybe finish it first and 
> cover both with a single news item. Git is going to be more efficient here, 
> so people may want to learn they have an alternative.

Yeah there's already a patch for git sync [1] but I'd rather not make
this news item more complicated than it needs to be. I wouldn't have
bothered with a news item except that I want to give people some warning
in case they are using overlayfs [2]. I think the efficiency difference
between rsync and git here are pretty negligible for most people.

[1] https://bugs.gentoo.org/660372
[2]
https://www.brunsware.de/blog/gentoo/portage-tree-squashfs-overlayfs.html
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] News Item: Portage rsync hardlink support

[Zac Medico]

Please review.

Title: Portage rsync hardlink support
Author: Zac Medico 
Posted: 2018-07-11
Revision: 1
News-Item-Format: 2.0
Display-If-Installed: sys-apps/portage

For users of the rsync tree, beginning with sys-apps/portage-2.3.42,
the default behavior for sync operations will use hardlinks in order
to ensure that a repository remains in a valid state if something
goes wrong [1]. For example, if signature verification fails during a
sync operation, the new hardlink behavior will preserve the previous
state of the repository.

The new behavior may conflict with configurations that restrict the
use of hardlinks, such as overlay filesystems. Therefore, users will
have to set "sync-allow-hardlinks = no" in repos.conf if they have
a configuration that restricts the use of hardlinks, but this should
not be very common:

[DEFAULT]
sync-allow-hardlinks = no

[1] https://bugs.gentoo.org/660410 sys-apps/portage: use rsync
--link-dest to implement atomic repository updates (and abort if
signature verification fails)
-- 
Thanks,
Zac

[gentoo-dev] RFC: making ebuilds that provide mkfs.* programs include kernel config checks for fcaps (or other xattrs)

[Zac Medico]

Hi,

As suggested by Wojciech Myrda in bug 659486 [1], it would be helpful if
sys-fs/e2fsprogs would use the linux-info eclass to warn if the kernel
configuration doesn't include CONFIG_EXT4_FS_SECURITY=y when the user
has expressed a desire to use file capabilities. This idea can be
extended to any ebuild that provides an mkfs.* program. Thoughts?

[1] https://bugs.gentoo.org/659486#c11
-- 
Thanks,
Zac

Re: [gentoo-dev] Re: EAPI 7, BDEPEND and pkg_*inst

[Zac Medico]

On 06/05/2018 01:26 AM, Michael Haubenwallner wrote:
> On 06/04/2018 01:40 PM, Michał Górny wrote:
>> I think the 'closest' thing to right would be to use BDEPEND+RDEPEND. 
>> It won't cover cross+binpkg but I guess it's as good as you can get.
>>
> 
> Having BDEPEND in RDEPEND will fix binpkg, but break cross again - which
> BDEPEND aims to help firsthand (as far as I understood).
> 
> To get both binpkg and cross working, although not combined together,
> what about something like this workaround:
> 
> EAPI=7
> IUSE="+bindist"
> BDEPEND="some-cbuild/tool"
> RDEPEND="bindist? ( ${BDEPEND} )"
> 
> It should be easy for cross profiles to use.mask "bindist".
> 
> For cross (without binpkg), BDEPEND should still be around during pkg_*inst,
> even if not specified per PMS.

If BDEPEND isn't around, you can use the emerge --with-bdeps=y option to
handle this case.
-- 
Thanks,
Zac

Re: [gentoo-dev] EAPI 7, BDEPEND and pkg_*inst

[Zac Medico]

On 05/30/2018 04:49 AM, Michael Haubenwallner wrote:
> Hi,
> 
> HOORAY, seems like EAPI 7 might be able to obsolete the "prefix-chaining"
> portage patch I've carried in prefix-overlay all the time, thank you for that!
> 
> However, a first thing being unclear already came up when bumping EAPI 6 to 7:
> 
> For example, the current app-misc/ca-certificates ebuild (EAPI 6) contains:
> 
>  # c_rehash: we run `c_rehash`
>  # debianutils: we run `run-parts`
>  RDEPEND="${DEPEND}
>  app-misc/c_rehash
>  sys-apps/debianutils"
> 
>  pkg_postinst() {
> if [ -d "${EROOT}/usr/local/share/ca-certificates" ] ; then
> # if the user has local certs, we need to rebuild again
> # to include their stuff in the db.
> # However it's too overzealous when the user has custom certs in 
> place.
> # --fresh is to clean up dangling symlinks
> "${EROOT}"/usr/sbin/update-ca-certificates --root "${ROOT}"
> fi
>  }
> 
> Thing is, these RDEPENDs are not really required to "run" ca-certificates, 
> but to
> administrate it - which eventually is done on the CBUILD machine (from within 
> the
> ebuild, like in pkg_postinst currently), not necessarily on the CHOST machine.
> 
> So I do not necessarily want these RDEPENDs to be installed on the CHOST 
> machine,
> given that they may not be executed from within the CBUILD machine at all.
> 
> So the first idea is to move both RDEPENDs into BDEPEND.  But then, they are
> not guaranteed to be available during pkg_postinst - like for a binary 
> package.

Right, so it really belongs in RDEPEND *and* BDEPEND.

> Question now is: Is this wrong behaviour in the ebuild,
> or is this something where EAPI 7 is still insufficient for?

If we want to tune the dependencies more finely, we'll need new EAPI.

> When this is wrong (probably independent of EAPI 7 already) in the ebuild:
> How can the ebuild get such a use case right, especially with EAPI 7?

What's wrong with putting it in both?

> Thanks!
> /haubi/
> 
-- 
Thanks,
Zac

Re: [gentoo-dev] EAPI 7 can be used in the Gentoo repository

[Zac Medico]

On 05/06/2018 11:43 AM, Geaaru wrote:
> Hi,
> 
> I read documentation about new features for configure profiles, thanks.
> 
> In the past I open a discussion about permit use of annotation
> :: under profiles package.mask to mask from overlay for
> example gentoo/upstream packages.
> Is now supported by eapi 7?

There's nothing like that in EAPI 7, but there's an open request for a
portage profile-formats extension here:

  https://bugs.gentoo.org/651208

> Thanks in advance.
> G.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: virtual/init for init process

[Zac Medico]

On 04/27/2018 11:43 AM, Rich Freeman wrote:
> On Fri, Apr 27, 2018 at 2:18 PM Zac Medico  wrote:
> 
>> Actually, if things like sys-apps/s6 or sys-process/runit remain as
>> choices for virtual/init, this isn't going to solve the problem of
>> sys-apps/sysvinit being removed by emerge --depclean. In fact, if
>> virtual/init is not in the system set, then emerge --depclean will not
>> even warn about removing a system package when it removes
> sys-apps/sysvinit.
> 
> Well, at least we don't have to worry about users accidentally uninstalling
> openssh, nano, or e2fsprogs (too bad if they use any other filesystem
> though).  :)
> 
> In general we don't have a great way of telling portage what we're actually
> using when various packages do more than one thing.  If you have busybox
> installed you could conceivably remove a lot of other packages, but most
> users don't actually intend for this to happen.
> 
> A USE flag that just pulls in a dep seems like the least evil solution
> unless somebody wants to come up with a more general one.

A general solution might take the form of a standard interface for
interaction between eselect modules and package managers, as discussed
in this bug about telling the package manager which kernel you're using:

https://bugs.gentoo.org/283587
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: virtual/init for init process

[Zac Medico]

On 04/27/2018 08:45 AM, William Hubbs wrote:
> On Fri, Apr 27, 2018 at 12:48:09AM -0700, Zac Medico wrote:
>> On 04/26/2018 11:34 PM, Kent Fredric wrote:
>>> On Thu, 26 Apr 2018 13:35:15 -0700
>>> Zac Medico  wrote:
>>>
>>>> emerge --depclean, resulting in an unbootable system. Just say-in.
>>>
>>> And depclean being very verbose doesn't do many favours here either.
>>>
>>> ( I regularly do >500 package depcleans and spotting things that aren't 
>>> meant to be
>>>  culled amongst that list is a bit of a challenge )> 
>>
>> At least for system packages, it will show a warning like the one shown
>> here:
>>
>>https://bugs.gentoo.org/642484#c0
>>
>> Hopefully that message helps those that are paying enough attention.
>> What can we do for those that overlook the warning message, other than
>> give them some rescue instructions for making their system boot again?
> 
> Another option suggested to me was to add a use flag to OpenRC that
> pulls in a dependency on virtual/init instead of adding virtual/init to
> @system. The suggestion originally was to add it to rdepend, but that
> doesn't feel right to me since openrc really doesn't have a runtime
> dependency on init, so I'm thinking something like this:
> 
> IUSE="... +separate-init"
> 
> PDEPEND="
> ...
> separate-init? ( virtual/init )"
> 
> Thoughts?

Actually, if things like sys-apps/s6 or sys-process/runit remain as
choices for virtual/init, this isn't going to solve the problem of
sys-apps/sysvinit being removed by emerge --depclean. In fact, if
virtual/init is not in the system set, then emerge --depclean will not
even warn about removing a system package when it removes sys-apps/sysvinit.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: virtual/init for init process

[Zac Medico]

On 04/27/2018 08:43 AM, Pacho Ramos wrote:
> El vie, 27-04-2018 a las 00:48 -0700, Zac Medico escribió:
>> On 04/26/2018 11:34 PM, Kent Fredric wrote:
>>> On Thu, 26 Apr 2018 13:35:15 -0700
>>> Zac Medico  wrote:
>>>
>>>> emerge --depclean, resulting in an unbootable system. Just say-in.
>>>
>>> And depclean being very verbose doesn't do many favours here either.
>>>
>>> ( I regularly do >500 package depcleans and spotting things that aren't
>>> meant to be
>>>  culled amongst that list is a bit of a challenge )> 
>>
>> At least for system packages, it will show a warning like the one shown
>> here:
>>
>>https://bugs.gentoo.org/642484#c0
>>
>> Hopefully that message helps those that are paying enough attention.
>> What can we do for those that overlook the warning message, other than
>> give them some rescue instructions for making their system boot again?
> 
> Have you think in changing a bit the behavior of depclean to *not* depclean
> system packages and ask administrator to do something like "emerge -a 
> --depclean 
> --force" to force the depclean of that packages?
> 
> That could help to prevent that mistakes I think

I don't want this system packages warning to do anything other that
change the display, since it's really based on heuristics. The sysvinit
package wouldn't be removed if it was the only provider of virtual/init,
and it only triggers the warning because it's the preferred provider.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: virtual/init for init process

[Zac Medico]

On 04/27/2018 08:45 AM, William Hubbs wrote:
> On Fri, Apr 27, 2018 at 12:48:09AM -0700, Zac Medico wrote:
>> On 04/26/2018 11:34 PM, Kent Fredric wrote:
>>> On Thu, 26 Apr 2018 13:35:15 -0700
>>> Zac Medico  wrote:
>>>
>>>> emerge --depclean, resulting in an unbootable system. Just say-in.
>>>
>>> And depclean being very verbose doesn't do many favours here either.
>>>
>>> ( I regularly do >500 package depcleans and spotting things that aren't 
>>> meant to be
>>>  culled amongst that list is a bit of a challenge )> 
>>
>> At least for system packages, it will show a warning like the one shown
>> here:
>>
>>https://bugs.gentoo.org/642484#c0
>>
>> Hopefully that message helps those that are paying enough attention.
>> What can we do for those that overlook the warning message, other than
>> give them some rescue instructions for making their system boot again?
> 
> Another option suggested to me was to add a use flag to OpenRC that
> pulls in a dependency on virtual/init instead of adding virtual/init to
> @system. The suggestion originally was to add it to rdepend, but that
> doesn't feel right to me since openrc really doesn't have a runtime
> dependency on init, so I'm thinking something like this:
> 
> IUSE="... +separate-init"
> 
> PDEPEND="
> ...
> separate-init? ( virtual/init )"
> 
> Thoughts?

Yes, I like this idea.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: virtual/init for init process

[Zac Medico]

On 04/26/2018 11:34 PM, Kent Fredric wrote:
> On Thu, 26 Apr 2018 13:35:15 -0700
> Zac Medico  wrote:
> 
>> emerge --depclean, resulting in an unbootable system. Just say-in.
> 
> And depclean being very verbose doesn't do many favours here either.
> 
> ( I regularly do >500 package depcleans and spotting things that aren't meant 
> to be
>  culled amongst that list is a bit of a challenge )> 

At least for system packages, it will show a warning like the one shown
here:

   https://bugs.gentoo.org/642484#c0

Hopefully that message helps those that are paying enough attention.
What can we do for those that overlook the warning message, other than
give them some rescue instructions for making their system boot again?
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: virtual/init for init process

[Zac Medico]

On 04/26/2018 01:08 PM, William Hubbs wrote:
> After some discussion on IRC, I need to drop busybox from this virtual
> because it is in packages already. OpenRC needs to be dropped because
> for now we do not have a way to make sure both openrc and sysvinit get
> installed in stage 3 if openrc is listed both in virtual/init and
> virtual/service-manager.

I predict that a non-zero number of users that have sys-apps/s6 or
sys-process/runit will accidentally/blindly remove sys-apps/sysvinit via
emerge --depclean, resulting in an unbootable system. Just say-in.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: app-portage/repoman/

[Zac Medico]

On 03/30/2018 08:16 AM, William Hubbs wrote:
> On Fri, Mar 30, 2018 at 12:42:09PM +0200, Michał Górny wrote:
>> Dnia 30 marca 2018 11:01:33 CEST, Zac Medico  napisał(a):
>>> On 03/30/2018 01:42 AM, Michał Górny wrote:
>>>> Dnia 30 marca 2018 09:07:03 CEST, Zac Medico 
>>> napisał(a):
>>>>> On 03/29/2018 10:20 PM, Michał Górny wrote:
>>>>>> Dnia 30 marca 2018 06:30:14 CEST, Zac Medico 
>>>>> napisał(a):
>>>>>>> commit: 3f04d4d93d00afa5242a0c9459487c9eea7e9a6f
>>>>>>> Author: Zac Medico  gentoo  org>
>>>>>>> AuthorDate: Fri Mar 30 04:18:05 2018 +
>>>>>>> Commit: Zac Medico  gentoo  org>
>>>>>>> CommitDate: Fri Mar 30 04:30:05 2018 +
>>>>>>> URL:   
>>>>>>> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3f04d4d9
>>>>>>>
>>>>>>> app-portage/repoman: version bump to 2.3.8
>>>>>>>
>>>>>>> This release adds support for plugin module systems.
>>>>>>>
>>>>>>> See:
>>>>> https://wiki.gentoo.org/wiki/Project:Portage/Repoman-Module-specs
>>>>>>> Package-Manager: Portage-2.3.27_p1, Repoman-2.3.8
>>>>>>
>>>>>> Please revert this immediately. This was never finished, had open
>>>>> concerns that were never addressed and the added Gentoo repository
>>>>> files were never subject to proper RFC/review that was requested.
>>>>>
>>>>> I've disabled this feature by default, and hidden it behind an
>>>>> --experimental-repository-modules= option.
>>>>
>>>> That isn't going to work for me. Given that the yml files were added
>>> to repository without prior discussion, and contrary to the promises
>>> given by Portage team such discussion hasn't been started before
>>> merging the changes to master, I'd like to request removing all those
>>> files and not reintroducing them until Portage team specifies their
>>> format and brings it to open discussion.
>>>
>>> Yeah I agree that the format really needs a proper specification. I've
>>> gone ahead and removed them:
>>>
>>> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f62e76a1e0202dcf55faa2b0fd9b21a1cd184dc8
>>
>> Thanks. I hope this will let us start with a clean slate towards making a 
>> good format.
> 
> If there is some distro policy that this repoman enhancement violated,
> that's fine and I would like to know about it, but I'm not aware of it.

The format is definitely not fully baked, and will likely change, so I
think it's better to wait for it to mature some more.

> If there isn't, as far as I know, there is no reason for it to be
> blocked.

This is not really blocking anything, people can go ahead and use the
repoman --experimental-repository-modules=y option, even add it to
REPOMAN_DEFAULT_OPTS, and it will just work.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: app-portage/repoman/

[Zac Medico]

On 03/30/2018 01:42 AM, Michał Górny wrote:
> Dnia 30 marca 2018 09:07:03 CEST, Zac Medico  napisał(a):
>> On 03/29/2018 10:20 PM, Michał Górny wrote:
>>> Dnia 30 marca 2018 06:30:14 CEST, Zac Medico 
>> napisał(a):
>>>> commit: 3f04d4d93d00afa5242a0c9459487c9eea7e9a6f
>>>> Author: Zac Medico  gentoo  org>
>>>> AuthorDate: Fri Mar 30 04:18:05 2018 +
>>>> Commit: Zac Medico  gentoo  org>
>>>> CommitDate: Fri Mar 30 04:30:05 2018 +
>>>> URL:   
>>>> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3f04d4d9
>>>>
>>>> app-portage/repoman: version bump to 2.3.8
>>>>
>>>> This release adds support for plugin module systems.
>>>>
>>>> See:
>> https://wiki.gentoo.org/wiki/Project:Portage/Repoman-Module-specs
>>>> Package-Manager: Portage-2.3.27_p1, Repoman-2.3.8
>>>
>>> Please revert this immediately. This was never finished, had open
>> concerns that were never addressed and the added Gentoo repository
>> files were never subject to proper RFC/review that was requested.
>>
>> I've disabled this feature by default, and hidden it behind an
>> --experimental-repository-modules= option.
> 
> That isn't going to work for me. Given that the yml files were added to 
> repository without prior discussion, and contrary to the promises given by 
> Portage team such discussion hasn't been started before merging the changes 
> to master, I'd like to request removing all those files and not reintroducing 
> them until Portage team specifies their format and brings it to open 
> discussion.

Yeah I agree that the format really needs a proper specification. I've
gone ahead and removed them:

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f62e76a1e0202dcf55faa2b0fd9b21a1cd184dc8
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: [gentoo-commits] repo/gentoo:master commit in: app-portage/repoman/

[Zac Medico]

On 03/29/2018 10:20 PM, Michał Górny wrote:
> Dnia 30 marca 2018 06:30:14 CEST, Zac Medico  napisał(a):
>> commit: 3f04d4d93d00afa5242a0c9459487c9eea7e9a6f
>> Author: Zac Medico  gentoo  org>
>> AuthorDate: Fri Mar 30 04:18:05 2018 +
>> Commit: Zac Medico  gentoo  org>
>> CommitDate: Fri Mar 30 04:30:05 2018 +
>> URL:   
>> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3f04d4d9
>>
>> app-portage/repoman: version bump to 2.3.8
>>
>> This release adds support for plugin module systems.
>>
>> See: https://wiki.gentoo.org/wiki/Project:Portage/Repoman-Module-specs
>> Package-Manager: Portage-2.3.27_p1, Repoman-2.3.8
> 
> Please revert this immediately. This was never finished, had open concerns 
> that were never addressed and the added Gentoo repository files were never 
> subject to proper RFC/review that was requested.

I've disabled this feature by default, and hidden it behind an
--experimental-repository-modules= option.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New Portage fork: sys-apps/portage-mgorny

[Zac Medico]

On 03/26/2018 09:48 AM, Thomas Deutschmann wrote:
> On 2018-03-23 18:44, Patrick McLean wrote:
>> At my (and zmedico's) employer we use Gentoo heavily (all of our servers
>> run it), and have a few large internal overlays and hundreds of internal
>> profiles. There are packages in upstream Gentoo that we maintain an
>> internal fork of, and it would be extremely useful if we could mask the
>> ::gentoo version of something so a version bump does not cause it to be
>> installed instead of our forked version.
> 
> I have the same need, but it works for me: I have several packages in
> /etc/portage/package.mask directory with content like
> 
>   /::gentoo
> 
> to make sure the package from Gentoo repository isn't used.
> 
> But I guess you are talking about a different thing?

The issue is that people are using profiles hosted in repositories other
than gentoo, complete with profiles.desc entries (eselect profile
supports this), and they would like to have the ability to use ::repo
atoms in these profiles.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New Portage fork: sys-apps/portage-mgorny

[Zac Medico]

On 03/25/2018 02:02 AM, Kent Fredric wrote:
> On Sat, 24 Mar 2018 21:43:41 -0700
> Zac Medico  wrote:
> 
>> But if the majority demographic is as you describe, then they shouldn't
>> be using anything having dependencies that require package.unmask or **
>> keywords changes.
> 
> Again, they *dont*, the problem is portage makes the mistake of
> thinking they do.
> 
> This happens especially around virtuals where there is an existing
> problem of portage not doing the right thing when perl-core/* exists in
> some definition.
> 
> I don't have details on hand to give you as to how this happens,
> but I've seen this happen often enough around packages *I maintain* and
> *I* can't explain why portage is trying to install it, only that
> --auto-unmask-keep-masks=y makes the problem mysteriously go away.

An issue like that involving REQUIRED_USE has been reported, and today
I've created a patch that corrects the problem:

https://bugs.gentoo.org/622462

> The question for me is not "auto unmask is good" vs "autounmask is
> bad", autounmask is fine on its own and is very useful.
> 
> Its the default of --autounmask-keep-masks=n that I find short on value.
> 
> If anything, I suggest there needs to be an
> --autounmask-keep-masks=conditional, or something, that narrows the
> range of solutions portage will try and only attempt to unmask ** or
> package.mask in the following conditions:
> 
> - An explicitly masked package/version is explicitly requested on the command 
> line.
> - A package is a direct dependency of of the above
> - As above, but for the world file
> 
> That is, assume the only reason for masked packages to be considered is:
> - The user in some way directly requested them
> - A logical consequence of the user directly requesting a masked package

It's possible that bug 622462 is not the only way to trigger this
problem. If anyone experiences a problem like this, then a bug report
would be appreciated.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New Portage fork: sys-apps/portage-mgorny

[Zac Medico]

On 03/24/2018 07:26 PM, Kent Fredric wrote:
> On Sat, 24 Mar 2018 13:44:49 -0700
> Zac Medico  wrote:
> 
>> That only happens when dependency satisfaction fails by normal means.
> 
> And when that happens, it is better to bail and go "Uh oh, something bad",
> not "oh, right, lets install something that will likely make things
> worse and additional work to fix"

I don't think it's possible to have defaults that satisfy everyone. My
hope that the --autounmask default will be helpful to some people, and I
advise people to use --autounmask=n if it's not helpful.

> Its a regular occurrence that we have to tell people about this on #gentoo.

Normally, it emerge shows a message like the following when it creates
package.mask or ** keywords changes:

NOTE: The --autounmask-keep-masks option will prevent emerge
  from creating package.unmask or ** keyword changes.

>>> That default gets people using broken openssl and experimental
>>> packages blindly without them ever having intended on getting into
>>> experimental waters.  
>>
>> If people can't be bothered to understand the meaning of package.mask
>> and keywords changes, should they really be using Gentoo?
> 
> And its not *entirely* true that this is the case. Toralf used to
> complain portage couldn't find a resoultion and would try unmasking
> insane stuff in the process of tinderboxing.
> 
> But lo and behold, by removing the ability to unmask ** and
> package.mask, he reported a significant improvement in the ability to
> test.

That's great. I really don't expect the default to work well in every
situation.

> "RTFM?" is a terrible response to "you have bad defaults that make
> things break" because that default is *only* useful to people who would
> consider using things that have *zero* expectation that they would work.

The --autounmask behavior only triggers when a dependency is encountered
that cannot be satisfied by normal means. So, it means that the user is
already using masked packages, or they have expressed a desire to
install a masked package.

> And that is not any majority demographic of the Gentoo user base.
> 
> Its not a useless feature, but its a feature that should only be
> enabled after reading the documentation.
But if the majority demographic is as you describe, then they shouldn't
be using anything having dependencies that require package.unmask or **
keywords changes.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature