[gentoo-portage-dev] [PATCH] eapply_user: Make idempotent per changes to EAPI 6

[Michał Górny]

---
 bin/phase-helpers.sh | 4 
 1 file changed, 4 insertions(+)

diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index 2fea0b2..511a41a 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -1070,6 +1070,10 @@ fi
 
 if ___eapi_has_eapply_user; then
eapply_user() {
+   local tagfile=${T}/.portage_user_patches_applied
+   [[ -f ${tagfile} ]] && return
+   >> "${tagfile}"
+
local basedir=${PORTAGE_CONFIGROOT%/}/etc/portage/patches
 
local d applied
-- 
2.6.3

Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels

[Mike Frysinger]

On 11 Nov 2015 10:32, Michał Górny wrote:
> I'm not convinced we ought to do this for EAPI < 6. It is a breaking
> change after all, and as such changes the behavior of EAPI < 6 ebuilds.

that is a false statement.  anything not working with bash-3.2 is already
broken according to the PMS.

> There are some ebuilds/eclasses that have bash version checks,
> and execute bash-4 code when bash-4 is available. As far as I
> understand, this will effectively prohibit bash-4 code even though
> BASH_VERSINFO will still indicate bash-4 is being used.

no, that's not what it does.  it changes behavior for code that itself
has changed behavior between versions.  it does not disable any newer
functionality.  it would have been nice to have a knob that would turn
off newer functionality as well, but upstream didn't seem keen on it.
-mike


signature.asc
Description: Digital signature

Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels

[Mike Frysinger]

On 11 Nov 2015 07:33, Ulrich Mueller wrote:
> > On Tue, 10 Nov 2015, Mike Frysinger wrote:
> > +   # Set the compat level in case things change with newer ones.  We must 
> > not
> > +   # export this into the env otherwise we might break  other shell 
> > scripts we
> > +   # execute (e.g. ones in /usr/bin).
> > +   BASH_COMPAT="${maj}.${min}"
> > +
> > +   # The variable above is new to bash-4.3.  For older versions, we have 
> > to use
> > +   # a compat knob.  Further, the compat knob only exists with older 
> > versions
> > +   # (e.g. bash-4.3 has compat42 but not compat43).  This means we only 
> > need to
> > +   # turn the knob with older EAPIs, and only when running newer bash 
> > versions:
> > +   # there is no bash-3.3 (it went 3.2 to 4.0), and when requiring 
> > bash-4.2, the
> > +   # var works with bash-4.3+, and you don't need to set compat to 4.2 
> > when you
> > +   # are already running 4.2.
> > +   if __eapi_bash_3_2 && [[ ${BASH_VERSINFO[0]} -gt 3 ]] ; then
> > +   shopt -s compat32
> > +   fi
> 
> Wouldn't this profit from an additional test for  understood the upstream discussion correctly, they were thinking about
> dropping the compat* options in some future version?

my take away was that they weren't going to be adding new compat levels.
i don't think they were planning on dropping existing ones.
-mike


signature.asc
Description: Digital signature

Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels

[Michał Górny]

On Tue, 10 Nov 2015 23:39:29 -0500
Mike Frysinger  wrote:

> To try and provide better stability across bash versions,
> set the language compat level based on the current EAPI.
> ---
>  bin/eapi.sh   |  8 
>  bin/ebuild.sh | 39 +++
>  2 files changed, 47 insertions(+)
> 
> diff --git a/bin/eapi.sh b/bin/eapi.sh
> index 528e6f2..b236344 100644
> --- a/bin/eapi.sh
> +++ b/bin/eapi.sh
> @@ -191,3 +191,11 @@ ___eapi_enables_failglob_in_global_scope() {
>  ___eapi_enables_globstar() {
>   [[ ${1-${EAPI-0}} =~ ^(4-python|5-progress)$ ]]
>  }
> +
> +__eapi_bash_3_2() {
> + [[ ${1-${EAPI-0}} =~ 
> ^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
> +}
> +
> +__eapi_bash_4_2() {
> + [[ ${1-${EAPI-0}} =~ ^(6)$ ]]
> +}
> diff --git a/bin/ebuild.sh b/bin/ebuild.sh
> index 75a9d24..2d09fb8 100755
> --- a/bin/ebuild.sh
> +++ b/bin/ebuild.sh
> @@ -6,8 +6,47 @@
>  # Make sure it's before everything so we don't mess aliases that follow.
>  unalias -a
>  
> +# Make sure this isn't exported to scripts we execute.
> +unset BASH_COMPAT
> +
>  source "${PORTAGE_BIN_PATH}/isolated-functions.sh" || exit 1
>  
> +# Set up the bash version compatibility level.
> +__check_bash_version() {
> + # Figure out which min version of bash we require.
> + local maj min
> + if __eapi_bash_3_2 ; then
> + maj=3 min=2
> + elif __eapi_bash_4_2 ; then
> + maj=4 min=2
> + else
> + return
> + fi
> +
> + # Make sure the active bash is sane.
> + if [[ ${BASH_VERSINFO[0]} -lt ${maj} ]] ||
> +[[ ${BASH_VERSINFO[0]} -eq ${maj} && ${BASH_VERSINFO[1]} -lt ${min} 
> ]] ; then
> + die ">=bash-${maj}.${min} is required"
> + fi
> +
> + # Set the compat level in case things change with newer ones.  We must 
> not
> + # export this into the env otherwise we might break  other shell 
> scripts we
> + # execute (e.g. ones in /usr/bin).
> + BASH_COMPAT="${maj}.${min}"
> +
> + # The variable above is new to bash-4.3.  For older versions, we have 
> to use
> + # a compat knob.  Further, the compat knob only exists with older 
> versions
> + # (e.g. bash-4.3 has compat42 but not compat43).  This means we only 
> need to
> + # turn the knob with older EAPIs, and only when running newer bash 
> versions:
> + # there is no bash-3.3 (it went 3.2 to 4.0), and when requiring 
> bash-4.2, the
> + # var works with bash-4.3+, and you don't need to set compat to 4.2 
> when you
> + # are already running 4.2.
> + if __eapi_bash_3_2 && [[ ${BASH_VERSINFO[0]} -gt 3 ]] ; then
> + shopt -s compat32
> + fi
> +}
> +__check_bash_version
> +
>  if [[ $EBUILD_PHASE != depend ]] ; then
>   source "${PORTAGE_BIN_PATH}/phase-functions.sh" || die
>   source "${PORTAGE_BIN_PATH}/save-ebuild-env.sh" || die

I'm not convinced we ought to do this for EAPI < 6. It is a breaking
change after all, and as such changes the behavior of EAPI < 6 ebuilds.

There are some ebuilds/eclasses that have bash version checks,
and execute bash-4 code when bash-4 is available. As far as I
understand, this will effectively prohibit bash-4 code even though
BASH_VERSINFO will still indicate bash-4 is being used.

-- 
Best regards,
Michał Górny



pgpsuVssU_YOb.pgp
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels

[Ulrich Mueller]

> On Wed, 11 Nov 2015, Michał Górny wrote:

> I'm not convinced we ought to do this for EAPI < 6. It is a breaking
> change after all, and as such changes the behavior of EAPI < 6
> ebuilds.

Which according to PMS should assume bash version 3.2. Also, AFAICS
the only feature where the compat settings could have any impact is
this setting from compat42:

   If set, bash does not process the replacement string in the pattern
   substitution word expansion using quote removal.

All other compat changes affect either only POSIX mode, or interactive
mode, or string comparison (where the compat32 setting disables locale
specific behaviour and uses strcmp instead, so effectively the compat
setting should be saner).

> There are some ebuilds/eclasses that have bash version checks,
> and execute bash-4 code when bash-4 is available. As far as I
> understand, this will effectively prohibit bash-4 code even though
> BASH_VERSINFO will still indicate bash-4 is being used. 

The compat settings don't disable any new features. They only restore
previous behaviour where there was an incompatible change.

Ulrich


pgpUvQguzzIMa.pgp
Description: PGP signature

Re: [gentoo-portage-dev] [PATCH] egencache: parallelize --update-changelogs (bug 565540)

[Robin H. Johnson]

Confirmed to work; please merge (also my prior patch that fixes relative
GIT_DIR implications, or ACK and I will merge it myself).

Speedup is less than expected however.
Running with:
--jobs 10 --load-average 6
yields a ~3.3x speedup on the previous non-parallel version.

With the system load average stabilizing around 2.9.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail : robb...@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-portage-dev] [PATCH] egencache: fix results when GIT_DIR is used in the environment.

[Zac Medico]

On 11/11/2015 02:30 PM, robb...@gentoo.org wrote:
> From: "Robin H. Johnson" 
> 
> If GIT_DIR is used, and .git is outside the root of the checkout, then
> --work-tree=... needs to be specified, otherwise any Git command that
> relies on relative directories to the root will be wrong.
> 
> Signed-off-by: Robin H. Johnson 
> ---
>  bin/egencache | 31 +++
>  1 file changed, 23 insertions(+), 8 deletions(-)

Looks good.

> + '--relative=%s' % (cp, ),

This addition could be mentioned in the commit message.

>   def run(self):
> - repo_path = self._portdb.porttrees[0]
> - os.chdir(repo_path)
> + os.chdir(self._repo_path)

The chdir calls make me nervous, since we should be careful to ensure
that they are timed correctly when we introduce parallelization. I'd
love to add a cwd argument for the grab method to pass into Popen, in
order to eliminate all of the chdir calls.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH v2] ebuild: set up bash compat levels

[Zac Medico]

On 11/11/2015 01:00 PM, Mike Frysinger wrote:
> To try and provide better stability across bash versions,
> set the language compat level based on the current EAPI.
> This does not ban newer features, it tells bash to use
> the older bash behavior when the behavior changes across
> versions.
> ---
>  bin/eapi.sh|  8 
>  bin/ebuild.sh  | 42 ++
>  bin/save-ebuild-env.sh |  2 +-
>  3 files changed, 51 insertions(+), 1 deletion(-)

Looks good now.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels

[Zac Medico]

On 11/11/2015 01:11 PM, Mike Frysinger wrote:
> On 11 Nov 2015 13:04, Zac Medico wrote:
>> On 11/11/2015 12:55 PM, Mike Frysinger wrote:
>>> On 11 Nov 2015 11:42, Zac Medico wrote:
 On 11/10/2015 08:39 PM, Mike Frysinger wrote:
> +# Set up the bash version compatibility level.
> +__check_bash_version() {

 Please unset all new internal function inside bin/save-ebuild-env.sh.
 Note that it already uses this line to unset functions beginning with
 ___eapi:

unset -f $(compgen -A function ___eapi_)
>>>
>>> why don't we create a new namespace for portage funcs ?  something like 
>>> __e* ?
>>
>> That works for me. According to PMS, we're free to do anything we want
>> as long as it begins with at least 2 underscores.
> 
> interesting.  why don't we just unmap all things that begin with 2 underscores
> then ?  or maybe 3 ?
> -mike
> 

Well, that's sort of a "greedy" approach when you consider that you
might wipe out a function defined in an eclass or ebuild. Check this to
see what might be filtered:

  bzgrep ^__ /var/db/pkg/*/*/environment.bz2

A nice compromise is to choose a namespace like __portage or something
like that.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels

[Zac Medico]

On 11/11/2015 10:33 PM, Zac Medico wrote:
> On 11/11/2015 01:11 PM, Mike Frysinger wrote:
>> On 11 Nov 2015 13:04, Zac Medico wrote:
>>> On 11/11/2015 12:55 PM, Mike Frysinger wrote:
 On 11 Nov 2015 11:42, Zac Medico wrote:
> On 11/10/2015 08:39 PM, Mike Frysinger wrote:
>> +# Set up the bash version compatibility level.
>> +__check_bash_version() {
>
> Please unset all new internal function inside bin/save-ebuild-env.sh.
> Note that it already uses this line to unset functions beginning with
> ___eapi:
>
>unset -f $(compgen -A function ___eapi_)

 why don't we create a new namespace for portage funcs ?  something like 
 __e* ?
>>>
>>> That works for me. According to PMS, we're free to do anything we want
>>> as long as it begins with at least 2 underscores.
>>
>> interesting.  why don't we just unmap all things that begin with 2 
>> underscores
>> then ?  or maybe 3 ?
>> -mike
>>
> 
> Well, that's sort of a "greedy" approach when you consider that you
> might wipe out a function defined in an eclass or ebuild. Check this to
> see what might be filtered:
> 
>   bzgrep ^__ /var/db/pkg/*/*/environment.bz2
> 
> A nice compromise is to choose a namespace like __portage or something
> like that.
> 

Also note that some internals have been intentionally preserved in
environment.bz2. For example, __eapi6_src_install exposes the default
src_install implementation, which someone might examine for debugging
purposes.
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH] egencache: parallelize --update-changelogs (bug 565540)

[Zac Medico]

Use the TaskScheduler class to parallelize GenChangeLogs. Fix
AsyncFunction so it does not re-define 'args' in __slots__.

X-Gentoo-Bug: 565540
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=565540
---
 bin/egencache| 24 +++-
 pym/portage/util/_async/AsyncFunction.py |  5 -
 2 files changed, 23 insertions(+), 6 deletions(-)

diff --git a/bin/egencache b/bin/egencache
index 51d115a..76eb00b 100755
--- a/bin/egencache
+++ b/bin/egencache
@@ -55,7 +55,9 @@ from portage.const import TIMESTAMP_FORMAT
 from portage.manifest import guessManifestFileType
 from portage.package.ebuild._parallel_manifest.ManifestScheduler import 
ManifestScheduler
 from portage.util import cmp_sort_key, writemsg_level
+from portage.util._async.AsyncFunction import AsyncFunction
 from portage.util._async.run_main_scheduler import run_main_scheduler
+from portage.util._async.TaskScheduler import TaskScheduler
 from portage.util._eventloop.global_event_loop import global_event_loop
 from portage import cpv_getkey
 from portage.dep import Atom, isjustname
@@ -748,7 +750,8 @@ class _special_filename(_filename_base):
return self.file_name < other.file_name
 
 class GenChangeLogs(object):
-   def __init__(self, portdb, changelog_output, changelog_reversed):
+   def __init__(self, portdb, changelog_output, changelog_reversed,
+   max_jobs=None, max_load=None):
self.returncode = os.EX_OK
self._portdb = portdb
self._wrapper = textwrap.TextWrapper(
@@ -758,6 +761,8 @@ class GenChangeLogs(object):
)
self._changelog_output = changelog_output
self._changelog_reversed = changelog_reversed
+   self._max_jobs = max_jobs
+   self._max_load = max_load
 
@staticmethod
def grab(cmd):
@@ -882,7 +887,7 @@ class GenChangeLogs(object):
 
output.close()
 
-   def run(self):
+   def _task_iter(self):
repo_path = self._portdb.porttrees[0]
os.chdir(repo_path)
 
@@ -908,7 +913,12 @@ class GenChangeLogs(object):
cmod = 0
 
if float(cmod) < float(lmod):
-   self.generate_changelog(cp)
+   yield 
AsyncFunction(target=self.generate_changelog, args=[cp])
+
+   def run(self):
+   return run_main_scheduler(
+   TaskScheduler(self._task_iter(), 
event_loop=global_event_loop(),
+   max_jobs=self._max_jobs, 
max_load=self._max_load))
 
 def egencache_main(args):
 
@@ -1149,8 +1159,12 @@ def egencache_main(args):
if options.update_changelogs:
gen_clogs = GenChangeLogs(portdb,
changelog_output=options.changelog_output,
-   changelog_reversed=options.changelog_reversed)
-   gen_clogs.run()
+   changelog_reversed=options.changelog_reversed,
+   max_jobs=options.jobs,
+   max_load=options.load_average)
+   signum = gen_clogs.run()
+   if signum is not None:
+   sys.exit(128 + signum)
ret.append(gen_clogs.returncode)
 
if options.write_timestamp:
diff --git a/pym/portage/util/_async/AsyncFunction.py 
b/pym/portage/util/_async/AsyncFunction.py
index b6142a2..40f6c5e 100644
--- a/pym/portage/util/_async/AsyncFunction.py
+++ b/pym/portage/util/_async/AsyncFunction.py
@@ -15,7 +15,10 @@ class AsyncFunction(ForkProcess):
"result" attribute after the forked process has exited.
"""
 
-   __slots__ = ('args', 'kwargs', 'result', 'target',
+   # NOTE: This class overrides the meaning of the SpawnProcess 'args'
+   # attribute, and uses it to hold the positional arguments for the
+   # 'target' function.
+   __slots__ = ('kwargs', 'result', 'target',
'_async_func_reader', '_async_func_reader_pw')
 
def _start(self):
-- 
2.4.9

Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels

[Zac Medico]

On 11/10/2015 08:39 PM, Mike Frysinger wrote:
> To try and provide better stability across bash versions,
> set the language compat level based on the current EAPI.
> ---
>  bin/eapi.sh   |  8 
>  bin/ebuild.sh | 39 +++
>  2 files changed, 47 insertions(+)
> 
> diff --git a/bin/eapi.sh b/bin/eapi.sh
> index 528e6f2..b236344 100644
> --- a/bin/eapi.sh
> +++ b/bin/eapi.sh
> @@ -191,3 +191,11 @@ ___eapi_enables_failglob_in_global_scope() {
>  ___eapi_enables_globstar() {
>   [[ ${1-${EAPI-0}} =~ ^(4-python|5-progress)$ ]]
>  }
> +
> +__eapi_bash_3_2() {
> + [[ ${1-${EAPI-0}} =~ 
> ^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
> +}
> +
> +__eapi_bash_4_2() {
> + [[ ${1-${EAPI-0}} =~ ^(6)$ ]]
> +}
> diff --git a/bin/ebuild.sh b/bin/ebuild.sh
> index 75a9d24..2d09fb8 100755
> --- a/bin/ebuild.sh
> +++ b/bin/ebuild.sh
> @@ -6,8 +6,47 @@
>  # Make sure it's before everything so we don't mess aliases that follow.
>  unalias -a
>  
> +# Make sure this isn't exported to scripts we execute.
> +unset BASH_COMPAT
> +
>  source "${PORTAGE_BIN_PATH}/isolated-functions.sh" || exit 1
>  
> +# Set up the bash version compatibility level.
> +__check_bash_version() {

Please unset all new internal function inside bin/save-ebuild-env.sh.
Note that it already uses this line to unset functions beginning with
___eapi:

   unset -f $(compgen -A function ___eapi_)

However, your __eapi functions will not be matched because they only
begin with 2 underscores.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels

[Mike Frysinger]

On 11 Nov 2015 13:04, Zac Medico wrote:
> On 11/11/2015 12:55 PM, Mike Frysinger wrote:
> > On 11 Nov 2015 11:42, Zac Medico wrote:
> >> On 11/10/2015 08:39 PM, Mike Frysinger wrote:
> >>> +# Set up the bash version compatibility level.
> >>> +__check_bash_version() {
> >>
> >> Please unset all new internal function inside bin/save-ebuild-env.sh.
> >> Note that it already uses this line to unset functions beginning with
> >> ___eapi:
> >>
> >>unset -f $(compgen -A function ___eapi_)
> > 
> > why don't we create a new namespace for portage funcs ?  something like 
> > __e* ?
> 
> That works for me. According to PMS, we're free to do anything we want
> as long as it begins with at least 2 underscores.

interesting.  why don't we just unmap all things that begin with 2 underscores
then ?  or maybe 3 ?
-mike


signature.asc
Description: Digital signature

[gentoo-portage-dev] [PATCH] Warn if LC_CTYPE does not toupper()/tolower() ASCII chars correctly

[Michał Górny]

Output a warning if LC_CTYPE is set to a value that causes libc
toupper() and/or tolower() conversions not apply correctly to printable
ASCII characters.
---
 pym/_emerge/actions.py | 40 
 1 file changed, 40 insertions(+)

diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
index 7f1cb59..31825ae 100644
--- a/pym/_emerge/actions.py
+++ b/pym/_emerge/actions.py
@@ -26,6 +26,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.dbapi._similar_name_search:similar_name_search',
'portage.debug',
'portage.news:count_unread_news,display_news_notifications',
+   'portage.util._ctypes:find_library,LoadLibrary',
'portage.util._get_vm_info:get_vm_info',
'portage.emaint.modules.sync.sync:SyncRepos',
'_emerge.chk_updated_cfg_files:chk_updated_cfg_files',
@@ -2451,6 +2452,43 @@ def getgccversion(chost):
 # lead them to report invalid bugs.
 _emerge_features_warn = frozenset(['keeptemp', 'keepwork'])
 
+def check_case_conversion_sanity():
+   # test for insane LC_CTYPE
+   libc_fn = find_library("c")
+   if libc_fn is not None:
+   libc = LoadLibrary(libc_fn)
+   if libc is not None:
+   lc = list(range(ord('a'), ord('z')+1))
+   uc = list(range(ord('A'), ord('Z')+1))
+   rlc = [libc.tolower(c) for c in uc]
+   ruc = [libc.toupper(c) for c in lc]
+
+   if lc != rlc or uc != ruc:
+   msg = ("WARNING: The LC_CTYPE variable is set 
to " +
+   "a locale that does not convert between 
lowercase " +
+   "and uppercase ASCII characters 
correctly. This " +
+   "can break ebuilds and cause issues in 
programs " +
+   "that rely on latin character 
conversion. " +
+   "Please consider enabling another 
locale (such as " +
+   "en_US.UTF-8) in /etc/locale.gen and 
setting it " +
+   "as LC_TYPE in make.conf.")
+   msg = [l for l in textwrap.wrap(msg, 70)]
+   msg.append("")
+   if uc != ruc:
+   msg.extend([
+   "  %s -> %s" % (''.join([chr(x) 
for x in lc]),
+   ''.join([chr(x) for x 
in ruc])),
+   "  %28s: %s" % ('expected',
+   ''.join([chr(x) for x 
in uc]))])
+   if lc != rlc:
+   msg.extend([
+   "  %s -> %s" % (''.join([chr(x) 
for x in uc]),
+   ''.join([chr(x) for x 
in rlc])),
+   "  %28s: %s" % ('expected',
+   ''.join([chr(x) for x 
in lc]))])
+   writemsg_level("".join(["!!! %s\n" % l for l in 
msg]),
+   level=logging.ERROR, noiselevel=-1)
+
 def validate_ebuild_environment(trees):
features_warn = set()
for myroot in trees:
@@ -2467,6 +2505,8 @@ def validate_ebuild_environment(trees):
for line in textwrap.wrap(msg, 65):
out.ewarn(line)
 
+   check_case_conversion_sanity()
+
 def check_procfs():
procfs_path = '/proc'
if platform.system() not in ("Linux",) or \
-- 
2.6.3

Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels

[Mike Frysinger]

On 11 Nov 2015 11:42, Zac Medico wrote:
> On 11/10/2015 08:39 PM, Mike Frysinger wrote:
> > +# Set up the bash version compatibility level.
> > +__check_bash_version() {
> 
> Please unset all new internal function inside bin/save-ebuild-env.sh.
> Note that it already uses this line to unset functions beginning with
> ___eapi:
> 
>unset -f $(compgen -A function ___eapi_)

why don't we create a new namespace for portage funcs ?  something like __e* ?

> However, your __eapi functions will not be matched because they only
> begin with 2 underscores.

that wasn't intentional.  i'll change it to 3.
-mike


signature.asc
Description: Digital signature

[gentoo-portage-dev] [PATCH v2] ebuild: set up bash compat levels

[Mike Frysinger]

To try and provide better stability across bash versions,
set the language compat level based on the current EAPI.
This does not ban newer features, it tells bash to use
the older bash behavior when the behavior changes across
versions.
---
 bin/eapi.sh|  8 
 bin/ebuild.sh  | 42 ++
 bin/save-ebuild-env.sh |  2 +-
 3 files changed, 51 insertions(+), 1 deletion(-)

diff --git a/bin/eapi.sh b/bin/eapi.sh
index 528e6f2..cd3e1a4 100644
--- a/bin/eapi.sh
+++ b/bin/eapi.sh
@@ -191,3 +191,11 @@ ___eapi_enables_failglob_in_global_scope() {
 ___eapi_enables_globstar() {
[[ ${1-${EAPI-0}} =~ ^(4-python|5-progress)$ ]]
 }
+
+___eapi_bash_3_2() {
+   [[ ${1-${EAPI-0}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+}
+
+___eapi_bash_4_2() {
+   [[ ${1-${EAPI-0}} =~ ^(6)$ ]]
+}
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 75a9d24..78a93f0 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -6,8 +6,50 @@
 # Make sure it's before everything so we don't mess aliases that follow.
 unalias -a
 
+# Make sure this isn't exported to scripts we execute.
+unset BASH_COMPAT
+
 source "${PORTAGE_BIN_PATH}/isolated-functions.sh" || exit 1
 
+# Set up the bash version compatibility level.  This does not disable
+# features when running with a newer version, but makes it so that when
+# bash changes behavior in an incompatible way, the older behavior is
+# used instead.
+__check_bash_version() {
+   # Figure out which min version of bash we require.
+   local maj min
+   if ___eapi_bash_3_2 ; then
+   maj=3 min=2
+   elif ___eapi_bash_4_2 ; then
+   maj=4 min=2
+   else
+   return
+   fi
+
+   # Make sure the active bash is sane.
+   if [[ ${BASH_VERSINFO[0]} -lt ${maj} ]] ||
+  [[ ${BASH_VERSINFO[0]} -eq ${maj} && ${BASH_VERSINFO[1]} -lt ${min} 
]] ; then
+   die ">=bash-${maj}.${min} is required"
+   fi
+
+   # Set the compat level in case things change with newer ones.  We must 
not
+   # export this into the env otherwise we might break  other shell 
scripts we
+   # execute (e.g. ones in /usr/bin).
+   BASH_COMPAT="${maj}.${min}"
+
+   # The variable above is new to bash-4.3.  For older versions, we have 
to use
+   # a compat knob.  Further, the compat knob only exists with older 
versions
+   # (e.g. bash-4.3 has compat42 but not compat43).  This means we only 
need to
+   # turn the knob with older EAPIs, and only when running newer bash 
versions:
+   # there is no bash-3.3 (it went 3.2 to 4.0), and when requiring 
bash-4.2, the
+   # var works with bash-4.3+, and you don't need to set compat to 4.2 
when you
+   # are already running 4.2.
+   if ___eapi_bash_3_2 && [[ ${BASH_VERSINFO[0]} -gt 3 ]] ; then
+   shopt -s compat32
+   fi
+}
+__check_bash_version
+
 if [[ $EBUILD_PHASE != depend ]] ; then
source "${PORTAGE_BIN_PATH}/phase-functions.sh" || die
source "${PORTAGE_BIN_PATH}/save-ebuild-env.sh" || die
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index 477ed28..1120297 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -75,7 +75,7 @@ __save_ebuild_env() {
__ebuild_main __ebuild_phase __ebuild_phase_with_hooks \
__ebuild_arg_to_phase __ebuild_phase_funcs default \
__unpack_tar __unset_colors \
-   __source_env_files __try_source \
+   __source_env_files __try_source __check_bash_version \
__eqaquote __eqatag \
${QA_INTERCEPTORS}
 
-- 
2.6.2

Re: [gentoo-portage-dev] [PATCH] ebuild: set up bash compat levels

[Zac Medico]

On 11/11/2015 12:55 PM, Mike Frysinger wrote:
> On 11 Nov 2015 11:42, Zac Medico wrote:
>> On 11/10/2015 08:39 PM, Mike Frysinger wrote:
>>> +# Set up the bash version compatibility level.
>>> +__check_bash_version() {
>>
>> Please unset all new internal function inside bin/save-ebuild-env.sh.
>> Note that it already uses this line to unset functions beginning with
>> ___eapi:
>>
>>unset -f $(compgen -A function ___eapi_)
> 
> why don't we create a new namespace for portage funcs ?  something like __e* ?

That works for me. According to PMS, we're free to do anything we want
as long as it begins with at least 2 underscores.
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH] egencache: fix results when GIT_DIR is used in the environment.

[robbat2]

From: "Robin H. Johnson" 

If GIT_DIR is used, and .git is outside the root of the checkout, then
--work-tree=... needs to be specified, otherwise any Git command that
relies on relative directories to the root will be wrong.

Signed-off-by: Robin H. Johnson 
---
 bin/egencache | 31 +++
 1 file changed, 23 insertions(+), 8 deletions(-)

diff --git a/bin/egencache b/bin/egencache
index 51d115a..2117dd9 100755
--- a/bin/egencache
+++ b/bin/egencache
@@ -758,6 +758,16 @@ class GenChangeLogs(object):
)
self._changelog_output = changelog_output
self._changelog_reversed = changelog_reversed
+   self._repo_path = self._portdb.porttrees[0]
+   # --work-tree=... must be passed to Git if GIT_DIR is used
+   # and GIT_DIR is not a child of the root of the checkout
+   # eg:
+   # GIT_DIR=$parent/work/.git/
+   # work-tree=$parent/staging/
+   # If work-tree is not passed, Git tries to use the shared
+   # parent of the current directory and the $GIT_DIR, which can
+   # be outside the root of the checkout.
+   self._work_tree = '--work-tree=%s' % self._repo_path
 
@staticmethod
def grab(cmd):
@@ -766,6 +776,7 @@ class GenChangeLogs(object):
encoding=_encodings['stdio'], errors='strict')
 
def generate_changelog(self, cp):
+
try:
output = io.open(self._changelog_output,
mode='w', encoding=_encodings['repo.content'],
@@ -785,7 +796,7 @@ class GenChangeLogs(object):
''' % (cp, time.strftime('%Y'
 
# now grab all the commits
-   revlist_cmd = ['git', 'rev-list']
+   revlist_cmd = ['git', self._work_tree, 'rev-list']
if self._changelog_reversed:
revlist_cmd.append('--reverse')
revlist_cmd.extend(['HEAD', '--', '.'])
@@ -797,12 +808,17 @@ class GenChangeLogs(object):
# --no-renames to avoid getting more complex records on 
the list
# --format to get the timestamp, author and commit 
description
# --root to make it work fine even with the initial 
commit
-   # --relative to get paths relative to ebuilddir
+   # --relative=$cp to get paths relative to ebuilddir
# -r (recursive) to get per-file changes
# then the commit-id and path.
 
-   cinfo = self.grab(['git', 'diff-tree', '--name-status', 
'--no-renames',
-   '--format=%ct %cN <%cE>%n%B', '--root', 
'--relative', '-r',
+   cinfo = self.grab(['git', self._work_tree, 'diff-tree',
+   '--name-status',
+   '--no-renames',
+   '--format=%ct %cN <%cE>%n%B',
+   '--root',
+   '--relative=%s' % (cp, ),
+   '-r',
c, '--', '.']).rstrip('\n').split('\n')
 
# Expected output:
@@ -883,8 +899,7 @@ class GenChangeLogs(object):
output.close()
 
def run(self):
-   repo_path = self._portdb.porttrees[0]
-   os.chdir(repo_path)
+   os.chdir(self._repo_path)
 
if 'git' not in FindVCS():
writemsg_level(
@@ -894,10 +909,10 @@ class GenChangeLogs(object):
return
 
for cp in self._portdb.cp_all():
-   os.chdir(os.path.join(repo_path, cp))
+   os.chdir(os.path.join(self._repo_path, cp))
# Determine whether ChangeLog is up-to-date by comparing
# the newest commit timestamp with the ChangeLog 
timestamp.
-   lmod = self.grab(['git', 'log', '--format=%ct', '-1', 
'.'])
+   lmod = self.grab(['git', self._work_tree, 'log', 
'--format=%ct', '-1', '.'])
if not lmod:
# This cp has not been added to the repo.
continue
-- 
2.3.0