Re: [gentoo-portage-dev] [PATCH] bin/estrip: avoid copying directories in FEATURES=installsources

2021-07-18 Thread Sergei Trofimovich 
On Sat, 17 Jul 2021 15:34:12 -0700
Zac Medico  wrote:

> On 7/17/21 12:59 PM, Sergei Trofimovich wrote:
> > Initially problem is noticed on gcc-11 as a full ${WORKDIR} syncing
> > into /usr/src/debug. It happens because `debug.sources` sometimes
> > contains directory. For example on bash-5 it has:
> > 
> > $ grep -zv '/<[^/>]*>$' debug.sources | LANG=C sort -z -u  | sed -e 
> > 's/\x00/\n/g'
> > bash-5.0/
> > bash-5.0/alias.c
> > ...
> > 
> > This causes syncing object files, config.log, final binaries
> > and other unexpected data. The change avoids syncking paths
> > that end with '/'.
> > 
> > Signed-off-by: Sergei Trofimovich 
> > ---
> >  bin/estrip | 3 +++
> >  1 file changed, 3 insertions(+)
> > 
> > diff --git a/bin/estrip b/bin/estrip
> > index 7ef1ec35c..6cca0d04b 100755
> > --- a/bin/estrip
> > +++ b/bin/estrip
> > @@ -464,7 +464,10 @@ if [[ -s ${tmpdir}/debug.sources ]] && \
> >  then
> > __vecho "installsources: rsyncing source files"
> > [[ -d ${D%/}/${prepstrip_sources_dir#/} ]] || mkdir -p 
> > "${D%/}/${prepstrip_sources_dir#/}"
> > +   # skip installation of ".../" (system headers? why inner slashes 
> > are forbidden?)
> > +   # skip syncing of ".../foo/" (complete directories)
> > grep -zv '/<[^/>]*>$' "${tmpdir}"/debug.sources | \
> > +   grep -zv '/$' | \
> > (cd "${WORKDIR}"; LANG=C sort -z -u | \
> > rsync -tL0 --chmod=ugo-st,a+r,go-w,Da+x,Fa-x --files-from=- 
> > "${WORKDIR}/" "${D%/}/${prepstrip_sources_dir#/}/" )
> >  
> >   
> 
> Looks good. Merged with both grep calls combined via grep -e. Thanks!
> 
> https://gitweb.gentoo.org/proj/portage.git/commit/?id=e083c8bf20d8488d329e3dccd643c28429e6fe30

TIL 'grep -e'! Thank you!

-- 

  Sergei


pgprH8NkjP86o.pgp
Description: Цифровая подпись OpenPGP

[gentoo-portage-dev] [PATCH] bin/estrip: avoid copying directories in FEATURES=installsources

2021-07-17 Thread Sergei Trofimovich 
Initially problem is noticed on gcc-11 as a full ${WORKDIR} syncing
into /usr/src/debug. It happens because `debug.sources` sometimes
contains directory. For example on bash-5 it has:

$ grep -zv '/<[^/>]*>$' debug.sources | LANG=C sort -z -u  | sed -e 
's/\x00/\n/g'
bash-5.0/
bash-5.0/alias.c
...

This causes syncing object files, config.log, final binaries
and other unexpected data. The change avoids syncking paths
that end with '/'.

Signed-off-by: Sergei Trofimovich 
---
 bin/estrip | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/bin/estrip b/bin/estrip
index 7ef1ec35c..6cca0d04b 100755
--- a/bin/estrip
+++ b/bin/estrip
@@ -464,7 +464,10 @@ if [[ -s ${tmpdir}/debug.sources ]] && \
 then
__vecho "installsources: rsyncing source files"
[[ -d ${D%/}/${prepstrip_sources_dir#/} ]] || mkdir -p 
"${D%/}/${prepstrip_sources_dir#/}"
+   # skip installation of ".../" (system headers? why inner slashes 
are forbidden?)
+   # skip syncing of ".../foo/" (complete directories)
grep -zv '/<[^/>]*>$' "${tmpdir}"/debug.sources | \
+   grep -zv '/$' | \
(cd "${WORKDIR}"; LANG=C sort -z -u | \
rsync -tL0 --chmod=ugo-st,a+r,go-w,Da+x,Fa-x --files-from=- 
"${WORKDIR}/" "${D%/}/${prepstrip_sources_dir#/}/" )
 
-- 
2.32.0

[gentoo-portage-dev] [PATCH] emerge: drop FEATURES=distcc-pump support, bug #702146

2019-12-06 Thread Sergei Trofimovich 
'distcc' dustributes code generation for preprocessed files.

'pump' distributes preprocessing and code generation of files
and imposes very strict requirement:

"""
Note that distcc's pump-mode assumes that sources files will
not be modified during the lifetime of the include server, so
modifying source files during a build may cause inconsistent
results.
"""

`src_configure()` (where we used to start include server before
this change) almost always violates that requirement.

It is not uncommon to generate more intermediate source files
as a package builds (`bison`, `flex`, child `./configure` calls
from `make`) and thus quite unsafe to use `pump`.

This change drops `FEATURES=distcc-pump` and leaves only
FEATURES=distcc. This way all the proprocessing happens as expected
and only code generation is offloaded.

Closes: https://bugs.gentoo.org/702146
Signed-off-by: Sergei Trofimovich 
---
 bin/phase-functions.sh | 17 -
 lib/_emerge/EbuildPhase.py |  2 +-
 lib/portage/const.py   |  1 -
 3 files changed, 1 insertion(+), 19 deletions(-)

diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index 92fcd3929..73f8cee9b 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -403,19 +403,6 @@ __dyn_prepare() {
trap - SIGINT SIGQUIT
 }
 
-# @FUNCTION: __start_distcc
-# @DESCRIPTION:
-# Start distcc-pump if necessary.
-__start_distcc() {
-   if has distcc $FEATURES && has distcc-pump $FEATURES ; then
-   if [[ -z $INCLUDE_SERVER_PORT ]] || [[ ! -w 
$INCLUDE_SERVER_PORT ]] ; then
-   # adding distcc to PATH repeatedly results in fatal 
distcc recursion :)
-   eval $(pump --startup | grep -v PATH)
-   trap "pump --shutdown >/dev/null" EXIT
-   fi
-   fi
-}
-
 __dyn_configure() {
 
if [[ -e $PORTAGE_BUILDDIR/.configured ]] ; then
@@ -435,7 +422,6 @@ __dyn_configure() {
fi
 
trap __abort_configure SIGINT SIGQUIT
-   __start_distcc
 
__ebuild_phase pre_src_configure
 
@@ -469,7 +455,6 @@ __dyn_compile() {
fi
 
trap __abort_compile SIGINT SIGQUIT
-   __start_distcc
 
__ebuild_phase pre_src_compile
 
@@ -493,7 +478,6 @@ __dyn_test() {
fi
 
trap "__abort_test" SIGINT SIGQUIT
-   __start_distcc
 
if [[ -d ${S} ]]; then
cd "${S}"
@@ -541,7 +525,6 @@ __dyn_install() {
return 0
fi
trap "__abort_install" SIGINT SIGQUIT
-   __start_distcc
 
# Handle setting QA_* based on QA_PREBUILT
# Those variables shouldn't be needed before src_install()
diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
index 4104cefa7..50e3dd1f4 100644
--- a/lib/_emerge/EbuildPhase.py
+++ b/lib/_emerge/EbuildPhase.py
@@ -49,7 +49,7 @@ class EbuildPhase(CompositeTask):
 
# FEATURES displayed prior to setup phase
_features_display = (
-   "ccache", "compressdebug", "distcc", "distcc-pump", "fakeroot",
+   "ccache", "compressdebug", "distcc", "fakeroot",
"installsources", "keeptemp", "keepwork", "network-sandbox",
"network-sandbox-proxy", "nostrip", "preserve-libs", "sandbox",
"selinux", "sesandbox", "splitdebug", "suidctl", "test",
diff --git a/lib/portage/const.py b/lib/portage/const.py
index 36b33af92..e95039fd5 100644
--- a/lib/portage/const.py
+++ b/lib/portage/const.py
@@ -142,7 +142,6 @@ SUPPORTED_FEATURES   = frozenset([
"config-protect-if-modified",
"digest",
"distcc",
-   "distcc-pump",
"distlocks",
"downgrade-backup",
"ebuild-locks",
-- 
2.24.0

Re: [gentoo-portage-dev] [PATCH] repoman: add --include-profiles=PROFILES

2019-11-18 Thread Sergei Trofimovich 
On Mon, 18 Nov 2019 16:45:58 -0800
Zac Medico  wrote:

> On 11/18/19 4:21 PM, Sergei Trofimovich wrote:
> > repoman slows down ~linearly with amount of profiles being scanned.
> > In case of amd64 we have 28 stable profiles.
> > 
> > To speed up processing and fit into time budged of various CIs we can
> > split the work across different processes that handle different profiles.
> > 
> > Example benchmark on ::haskell overlay:
> > $ ./repoman full --include-arches=amd64
> > ~65 minutes
> > $ ./repoman full --include-profiles=default/linux/amd64/17.0
> > ~4 minutes
> > This allows for a crude sharding of work across processes and allows for
> > cheap tree-wide scans for early failures.
> > 
> > Bug: https://bugs.gentoo.org/700456
> > Signed-off-by: Sergei Trofimovich 
> > ---
> >  repoman/lib/repoman/actions.py  | 4 
> >  repoman/lib/repoman/argparser.py| 7 +++
> >  repoman/lib/repoman/modules/scan/depend/__init__.py | 3 ++-
> >  repoman/lib/repoman/modules/scan/depend/profile.py  | 9 +++--
> >  repoman/lib/repoman/scanner.py  | 5 +
> >  repoman/man/repoman.1   | 4 
> >  6 files changed, 29 insertions(+), 3 deletions(-)
> > 
> > diff --git a/repoman/lib/repoman/actions.py b/repoman/lib/repoman/actions.py
> > index 1c9989a72..92d4d4e94 100644
> > --- a/repoman/lib/repoman/actions.py
> > +++ b/repoman/lib/repoman/actions.py
> > @@ -412,6 +412,10 @@ the whole commit message to abort.
> > report_options.append(
> > "--include-arches=\"%s\"" %
> > " ".join(sorted(self.scanner.include_arches)))
> > +   if self.scanner.include_profiles is not None:
> > +   report_options.append(
> > +   "--include-profiles=\"%s\"" %
> > +   " ".join(sorted(self.scanner.include_profiles)))
> >  
> > if portage_version is None:
> > sys.stderr.write("Failed to insert portage version in 
> > message!\n")
> > diff --git a/repoman/lib/repoman/argparser.py 
> > b/repoman/lib/repoman/argparser.py
> > index fa0e6ff90..670a0e91d 100644
> > --- a/repoman/lib/repoman/argparser.py
> > +++ b/repoman/lib/repoman/argparser.py
> > @@ -164,6 +164,13 @@ def parse_args(argv, repoman_default_opts):
> > 'A space separated list of arches used to '
> > 'filter the selection of profiles for dependency 
> > checks'))
> >  
> > +   parser.add_argument(
> > +   '--include-profiles',
> > +   dest='include_profiles', metavar='PROFILES', action='append',
> > +   help=(
> > +   'A space separated list of profiles used to '
> > +   'define the selection of profiles for dependency 
> > checks'))
> > +
> > parser.add_argument(
> > '-d', '--include-dev', dest='include_dev', action='store_true',
> > default=False,
> > diff --git a/repoman/lib/repoman/modules/scan/depend/__init__.py 
> > b/repoman/lib/repoman/modules/scan/depend/__init__.py
> > index c3cc0ddeb..9068760bb 100644
> > --- a/repoman/lib/repoman/modules/scan/depend/__init__.py
> > +++ b/repoman/lib/repoman/modules/scan/depend/__init__.py
> > @@ -19,7 +19,8 @@ module_spec = {
> > 'func_desc': {
> > },
> > 'mod_kwargs': ['qatracker', 'portdb', 'profiles', 
> > 'options',
> > -   'repo_metadata', 'repo_settings', 
> > 'include_arches', 'caches',
> > +   'repo_metadata', 'repo_settings', 
> > 'include_arches',
> > +   'include_profiles', 'caches',
> > 'repoman_incrementals', 'env', 'have', 
> > 'dev_keywords'
> > ],
> > 'func_kwargs': {
> > diff --git a/repoman/lib/repoman/modules/scan/depend/profile.py 
> > b/repoman/lib/repoman/modules/scan/depend/profile.py
> > index d980f4eca..0b1d74483 100644
> > --- a/repoman/lib/repoman/modules/scan/depend/profile.py
> > +++ b/repoman/lib/repoman/modules/scan/depend/profile.py
> > @@ -33,6 +33,7 @@ class ProfileDependsChecks(ScanBase):
> > @param options: cli options
> > @param repo_settings: repository settings insta

[gentoo-portage-dev] [PATCH] repoman: add --include-profiles=PROFILES

2019-11-18 Thread Sergei Trofimovich 
repoman slows down ~linearly with amount of profiles being scanned.
In case of amd64 we have 28 stable profiles.

To speed up processing and fit into time budged of various CIs we can
split the work across different processes that handle different profiles.

Example benchmark on ::haskell overlay:
$ ./repoman full --include-arches=amd64
~65 minutes
$ ./repoman full --include-profiles=default/linux/amd64/17.0
~4 minutes
This allows for a crude sharding of work across processes and allows for
cheap tree-wide scans for early failures.

Bug: https://bugs.gentoo.org/700456
Signed-off-by: Sergei Trofimovich 
---
 repoman/lib/repoman/actions.py  | 4 
 repoman/lib/repoman/argparser.py| 7 +++
 repoman/lib/repoman/modules/scan/depend/__init__.py | 3 ++-
 repoman/lib/repoman/modules/scan/depend/profile.py  | 9 +++--
 repoman/lib/repoman/scanner.py  | 5 +
 repoman/man/repoman.1   | 4 
 6 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/repoman/lib/repoman/actions.py b/repoman/lib/repoman/actions.py
index 1c9989a72..92d4d4e94 100644
--- a/repoman/lib/repoman/actions.py
+++ b/repoman/lib/repoman/actions.py
@@ -412,6 +412,10 @@ the whole commit message to abort.
report_options.append(
"--include-arches=\"%s\"" %
" ".join(sorted(self.scanner.include_arches)))
+   if self.scanner.include_profiles is not None:
+   report_options.append(
+   "--include-profiles=\"%s\"" %
+   " ".join(sorted(self.scanner.include_profiles)))
 
if portage_version is None:
sys.stderr.write("Failed to insert portage version in 
message!\n")
diff --git a/repoman/lib/repoman/argparser.py b/repoman/lib/repoman/argparser.py
index fa0e6ff90..670a0e91d 100644
--- a/repoman/lib/repoman/argparser.py
+++ b/repoman/lib/repoman/argparser.py
@@ -164,6 +164,13 @@ def parse_args(argv, repoman_default_opts):
'A space separated list of arches used to '
'filter the selection of profiles for dependency 
checks'))
 
+   parser.add_argument(
+   '--include-profiles',
+   dest='include_profiles', metavar='PROFILES', action='append',
+   help=(
+   'A space separated list of profiles used to '
+   'define the selection of profiles for dependency 
checks'))
+
parser.add_argument(
'-d', '--include-dev', dest='include_dev', action='store_true',
default=False,
diff --git a/repoman/lib/repoman/modules/scan/depend/__init__.py 
b/repoman/lib/repoman/modules/scan/depend/__init__.py
index c3cc0ddeb..9068760bb 100644
--- a/repoman/lib/repoman/modules/scan/depend/__init__.py
+++ b/repoman/lib/repoman/modules/scan/depend/__init__.py
@@ -19,7 +19,8 @@ module_spec = {
'func_desc': {
},
'mod_kwargs': ['qatracker', 'portdb', 'profiles', 
'options',
-   'repo_metadata', 'repo_settings', 
'include_arches', 'caches',
+   'repo_metadata', 'repo_settings', 
'include_arches',
+   'include_profiles', 'caches',
'repoman_incrementals', 'env', 'have', 
'dev_keywords'
],
'func_kwargs': {
diff --git a/repoman/lib/repoman/modules/scan/depend/profile.py 
b/repoman/lib/repoman/modules/scan/depend/profile.py
index d980f4eca..0b1d74483 100644
--- a/repoman/lib/repoman/modules/scan/depend/profile.py
+++ b/repoman/lib/repoman/modules/scan/depend/profile.py
@@ -33,6 +33,7 @@ class ProfileDependsChecks(ScanBase):
@param options: cli options
@param repo_settings: repository settings instance
@param include_arches: set
+   @param include_profiles: set
@param caches: dictionary of our caches
@param repoman_incrementals: tuple
@param env: the environment
@@ -46,6 +47,7 @@ class ProfileDependsChecks(ScanBase):
self.options = kwargs.get('options')
self.repo_settings = kwargs.get('repo_settings')
self.include_arches = kwargs.get('include_arches')
+   self.include_profiles = kwargs.get('include_profiles')
self.caches = kwargs.get('caches')
self.repoman_incrementals = kwargs.get('repoman_incrementals')
self.env = kwargs.get('env')
@@ -81,8 +83,11 @@ class ProfileDependsChecks(ScanBase):
if arch not in self.include_arches:

Re: [gentoo-portage-dev] Avoid stripping ld.so ?

2018-08-30 Thread Sergei Trofimovich 
On Thu, 30 Aug 2018 07:37:16 +
Joakim Tjernlund  wrote:

> > > Tweaking ebuild to expand STRIP_MASK for ld.so might help.  
> > 
> > OK, can I use STRIP_MASK outside ebuilds, like in /etc/portage/make.conf or
> > in my profile? Something like STRIP_MASK="/lib64/ld-* /lib/ld-*"

I believe you can't. You would have to resort to /etc/portage/bashrc
hacks to override ebuilds' value. Requesting new feature from dev-portage@
sounds reasonable.

For example, add STRIP_MASK_USER to be controlled by user.

STRIP_MASK is normally used for cases when software breaks when
stripped (examples: not-quite-ELF files, not-yet-linked .o files). Not
something for convenience or niche purposes.

STRIP_MASK_USER might be a good way to fine-grain desired policy
of stripping package on top of existing FEATURES=nostrip knob.

-- 

  Sergei

Re: [gentoo-portage-dev] Avoid stripping ld.so ?

2018-08-29 Thread Sergei Trofimovich 
On Wed, 29 Aug 2018 20:10:54 +
Joakim Tjernlund  wrote:

> Is there a way for portage to avoid strippning debug syms for ld.so ?
Tweaking ebuild to expand STRIP_MASK for ld.so might help.

> I would like to avoid building all of glibc with debug syms.
s/building/installing, right?

You will have to build all (or large part) of glibc with debug
symbols anyways as ld.so is linked of multiple source files.

-- 

  Sergei

[gentoo-portage-dev] Re: [PATCH v2] repoman: new QA error: slot operator under '||' alternative

2016-06-19 Thread Sergei Trofimovich 
On Sat, 18 Jun 2016 14:15:54 -0700
Zac Medico <zmed...@gentoo.org> wrote:

> On 06/18/2016 12:27 PM, Sergei Trofimovich wrote:
> > +   if in_any_of and atom.slot_operator:
> > +   qatracker.add_error("dependency.badslotop", 
> > relative_path +
> > +   ": %s: '%s' uses ':=' slot operator under '||' 
> > dep clause." %
> > +   (mytype, atom))  
> 
> It should be like this:
> 
> if in_any_of and atom.slot_operator == '=':
> 
> Because the case of atom.slot_operator == '*' is not forbidden.

Nice catch! Sent V4.

-- 

  Sergei


pgpyl5mQIj5Eq.pgp
Description: Цифровая подпись OpenPGP

[gentoo-portage-dev] [PATCH v4] repoman: new QA error: slot operator under '||' alternative

2016-06-18 Thread Sergei Trofimovich 
From: Sergei Trofimovich <siarh...@google.com>

A few links discussing what can go wrong with slot operator under '||':

https://archives.gentoo.org/gentoo-dev/message/81a4e1a1f5767e20009628ecd33da8d4

https://archives.gentoo.org/gentoo-dev/message/66ff016a055e57b6027abcd36734e0e3

The main problem here is how vdb gets updated when you have a dependency of 
style:
RDEPEND="|| ( foo:= bar:= )"
depending on what you have installed in system: only 'foo'/'bar' or both 'foo' 
and 'bar'.

I'm about to add actual test for some examples to gen-b0rk.

New repoman complains on them as follows:

RDEPEND="|| ( =not-broken/pkg1-subslot-0:= =not-broken/pkg1-subslot-1:0= )"

Yields:

  dependency.badslotop [fatal]  2
   ebuild-test/RDEPEND-any-of-slotop/RDEPEND-any-of-slotop-0.ebuild: RDEPEND: 
'=not-broken/pkg1-subslot-0:=' uses ':=' slot operator under '||' dep clause.
   ebuild-test/RDEPEND-any-of-slotop/RDEPEND-any-of-slotop-0.ebuild: RDEPEND: 
'=not-broken/pkg1-subslot-1:0=' uses ':=' slot operator under '||' dep clause.

CC: dol...@gentoo.org
CC: q...@gentoo.org
CC: mgo...@gentoo.org
Signed-off-by: Sergei Trofimovich <siarh...@google.com>
---
V4: forbid only ':=' style slot operator. Zac pointed out ':*' is fine here.
V3: unbaked tree traversal back to _traverse_tree, moved dependency parser out
to check_slotop() helper
V2: made check_slotop function local, baked tree traversal in it
 .../repoman/modules/scan/depend/_depend_checks.py  | 45 ++
 repoman/pym/repoman/qa_data.py |  2 +
 2 files changed, 47 insertions(+)

diff --git a/repoman/pym/repoman/modules/scan/depend/_depend_checks.py 
b/repoman/pym/repoman/modules/scan/depend/_depend_checks.py
index 4e1d216..3f6c93e 100644
--- a/repoman/pym/repoman/modules/scan/depend/_depend_checks.py
+++ b/repoman/pym/repoman/modules/scan/depend/_depend_checks.py
@@ -3,11 +3,52 @@
 
 from _emerge.Package import Package
 
+from portage.dep import Atom
+
 from repoman.check_missingslot import check_missingslot
 # import our initialized portage instance
 from repoman._portage import portage
 from repoman.qa_data import suspect_virtual, suspect_rdepend
 
+def check_slotop(depstr, is_valid_flag, badsyntax, mytype,
+   qatracker, relative_path):
+   '''Checks if RDEPEND uses ':=' slot operator
+   in '||' style dependencies.'''
+
+   try:
+   # to find use of ':=' in '||' we preserve
+   # tree structure of dependencies
+   my_dep_tree = portage.dep.use_reduce(
+   depstr,
+   flat=False,
+   matchall=1,
+   is_valid_flag=is_valid_flag,
+   opconvert=True,
+   token_class=portage.dep.Atom)
+   except portage.exception.InvalidDependString as e:
+   my_dep_tree = None
+   badsyntax.append(str(e))
+
+   def _traverse_tree(dep_tree, in_any_of):
+   # leaf
+   if isinstance(dep_tree, Atom):
+   atom = dep_tree
+   if in_any_of and atom.slot_operator == '=':
+   qatracker.add_error("dependency.badslotop",
+   "%s: %s: '%s' uses ':=' slot operator 
under '||' dep clause." %
+   (relative_path, mytype, atom))
+
+   # branches
+   if isinstance(dep_tree, list):
+   if len(dep_tree) == 0:
+   return
+   # entering any-of
+   if dep_tree[0] == '||':
+   _traverse_tree(dep_tree[1:], in_any_of=True)
+   else:
+   for branch in dep_tree:
+   _traverse_tree(branch, 
in_any_of=in_any_of)
+   _traverse_tree(my_dep_tree, False)
 
 def _depend_checks(ebuild, pkg, portdb, qatracker, repo_metadata):
'''Checks the ebuild dependencies for errors
@@ -117,6 +158,10 @@ def _depend_checks(ebuild, pkg, portdb, qatracker, 
repo_metadata):
 
type_list.extend([mytype] * (len(badsyntax) - len(type_list)))
 
+   if runtime:
+   check_slotop(mydepstr, pkg.iuse.is_valid_flag,
+   badsyntax, mytype, qatracker, 
ebuild.relative_path)
+
for m, b in zip(type_list, badsyntax):
if m.endswith("DEPEND"):
qacat = "dependency.syntax"
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index b9475e8..48ab389 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -58,6 +58,8 @@ qahelp = {
"Ebuild has a dependency that refers to an unknown package"
" (which may be valid if it is a blocker

[gentoo-portage-dev] [PATCH v2] repoman: new QA error: slot operator under '||' alternative

2016-06-18 Thread Sergei Trofimovich 
From: Sergei Trofimovich <siarh...@google.com>

A few links discussing what can go wrong with slot operator under '||':

https://archives.gentoo.org/gentoo-dev/message/81a4e1a1f5767e20009628ecd33da8d4

https://archives.gentoo.org/gentoo-dev/message/66ff016a055e57b6027abcd36734e0e3

The main problem here is how vdb gets updated when you have a dependency of 
style:
RDEPEND="|| ( foo:= bar:= )"
depending on what you have installed in system: only 'foo'/'bar' or both 'foo' 
and 'bar'.

I'm about to add actual test for some examples to gen-b0rk.

New repoman complains on them as follows:

RDEPEND="|| ( =not-broken/pkg1-subslot-0:= =not-broken/pkg1-subslot-1:0= )"

Yields:

  dependency.badslotop [fatal]  2
   ebuild-test/RDEPEND-any-of-slotop/RDEPEND-any-of-slotop-0.ebuild: RDEPEND: 
'=not-broken/pkg1-subslot-0:=' uses ':=' slot operator under '||' dep clause.
   ebuild-test/RDEPEND-any-of-slotop/RDEPEND-any-of-slotop-0.ebuild: RDEPEND: 
'=not-broken/pkg1-subslot-1:0=' uses ':=' slot operator under '||' dep clause.

CC: dol...@gentoo.org
CC: q...@gentoo.org
CC: mgo...@gentoo.org
Signed-off-by: Sergei Trofimovich <siarh...@google.com>
---
V2: made check_slotop function local, baked tree traversal in it
 .../repoman/modules/scan/depend/_depend_checks.py  | 42 ++
 repoman/pym/repoman/qa_data.py |  2 ++
 2 files changed, 44 insertions(+)

diff --git a/repoman/pym/repoman/modules/scan/depend/_depend_checks.py 
b/repoman/pym/repoman/modules/scan/depend/_depend_checks.py
index 4e1d216..2ee7e9c 100644
--- a/repoman/pym/repoman/modules/scan/depend/_depend_checks.py
+++ b/repoman/pym/repoman/modules/scan/depend/_depend_checks.py
@@ -3,11 +3,37 @@
 
 from _emerge.Package import Package
 
+from portage.dep import Atom
+
 from repoman.check_missingslot import check_missingslot
 # import our initialized portage instance
 from repoman._portage import portage
 from repoman.qa_data import suspect_virtual, suspect_rdepend
 
+def check_slotop(dep_tree, mytype, qatracker, relative_path, in_any_of=False):
+   '''Checks if RDEPEND uses ':=' slot operator
+   in '||' style dependencies.'''
+
+   # leaf
+   if isinstance(dep_tree, Atom):
+   atom = dep_tree
+   if in_any_of and atom.slot_operator:
+   qatracker.add_error("dependency.badslotop", 
relative_path +
+   ": %s: '%s' uses ':=' slot operator under '||' 
dep clause." %
+   (mytype, atom))
+
+   # branches
+   if isinstance(dep_tree, list):
+   if len(dep_tree) == 0:
+   return
+   # entering any-of
+   if dep_tree[0] == '||':
+   check_slotop(dep_tree[1:], mytype, qatracker, 
relative_path,
+   in_any_of=True)
+   else:
+   for branch in dep_tree:
+   check_slotop(branch, mytype, qatracker, 
relative_path,
+   in_any_of=in_any_of)
 
 def _depend_checks(ebuild, pkg, portdb, qatracker, repo_metadata):
'''Checks the ebuild dependencies for errors
@@ -117,6 +143,22 @@ def _depend_checks(ebuild, pkg, portdb, qatracker, 
repo_metadata):
 
type_list.extend([mytype] * (len(badsyntax) - len(type_list)))
 
+   if runtime:
+   try:
+   # to find use of ':=' in '||' we preserve
+   # tree structure of dependencies
+   hier_atoms = portage.dep.use_reduce(
+   mydepstr,
+   flat=False,
+   matchall=1,
+   is_valid_flag=pkg.iuse.is_valid_flag,
+   opconvert=True,
+   token_class=token_class)
+   except portage.exception.InvalidDependString as e:
+   hier_atoms = None
+   badsyntax.append(str(e))
+   check_slotop(hier_atoms, mytype, qatracker, 
ebuild.relative_path)
+
for m, b in zip(type_list, badsyntax):
if m.endswith("DEPEND"):
qacat = "dependency.syntax"
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index b9475e8..48ab389 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -58,6 +58,8 @@ qahelp = {
"Ebuild has a dependency that refers to an unknown package"
" (which may be valid if it is a blocker for a renamed/removed 
package,"
" or is an alternative choice provided by an overlay)"),
+

[gentoo-portage-dev] [PATCH v2] repoman/modules/.../profile.py: typo in docstring s/dependancy/dependency/

2016-06-18 Thread Sergei Trofimovich 
From: Sergei Trofimovich <siarh...@google.com>

Signed-off-by: Sergei Trofimovich <siarh...@google.com>
---
V2: updated subject
 repoman/pym/repoman/modules/scan/depend/profile.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/repoman/pym/repoman/modules/scan/depend/profile.py 
b/repoman/pym/repoman/modules/scan/depend/profile.py
index a714a93..cf3d9a8 100644
--- a/repoman/pym/repoman/modules/scan/depend/profile.py
+++ b/repoman/pym/repoman/modules/scan/depend/profile.py
@@ -51,7 +51,7 @@ class ProfileDependsChecks(ScanBase):
self.repo_metadata = kwargs.get('repo_metadata')
 
def check(self, **kwargs):
-   '''Perform profile dependant dependancy checks
+   '''Perform profile dependant dependency checks
 
@param arches:
@param pkg: Package in which we check (object).
-- 
2.9.0

[gentoo-portage-dev] [PATCH v2] portage/dep/depcheck.py: Whitespace, add indentation level after newline

2016-06-18 Thread Sergei Trofimovich 
From: Sergei Trofimovich <siarh...@google.com>

CC: dol...@gentoo.org
Signed-off-by: Sergei Trofimovich <siarh...@google.com>
---
V2: updated subject
 pym/portage/dep/dep_check.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pym/portage/dep/dep_check.py b/pym/portage/dep/dep_check.py
index 9af4e65..76e5d82 100644
--- a/pym/portage/dep/dep_check.py
+++ b/pym/portage/dep/dep_check.py
@@ -688,8 +688,8 @@ def dep_check(depstring, mydbapi, mysettings, use="yes", 
mode=None, myuse=None,
else:
try:
mysplit = use_reduce(depstring, uselist=myusesplit,
-   masklist=mymasks, matchall=(use=="all"), 
excludeall=useforce,
-   opconvert=True, token_class=Atom, eapi=eapi)
+   masklist=mymasks, matchall=(use=="all"), 
excludeall=useforce,
+   opconvert=True, token_class=Atom, eapi=eapi)
except InvalidDependString as e:
return [0, "%s" % (e,)]
 
-- 
2.9.0

[gentoo-portage-dev] [PATCH] repoman: whitespace: add indentation level after newline

2016-06-18 Thread Sergei Trofimovich 
From: Sergei Trofimovich <siarh...@google.com>

Signed-off-by: Sergei Trofimovich <siarh...@google.com>
---
 pym/portage/dep/dep_check.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pym/portage/dep/dep_check.py b/pym/portage/dep/dep_check.py
index 9af4e65..76e5d82 100644
--- a/pym/portage/dep/dep_check.py
+++ b/pym/portage/dep/dep_check.py
@@ -688,8 +688,8 @@ def dep_check(depstring, mydbapi, mysettings, use="yes", 
mode=None, myuse=None,
else:
try:
mysplit = use_reduce(depstring, uselist=myusesplit,
-   masklist=mymasks, matchall=(use=="all"), 
excludeall=useforce,
-   opconvert=True, token_class=Atom, eapi=eapi)
+   masklist=mymasks, matchall=(use=="all"), 
excludeall=useforce,
+   opconvert=True, token_class=Atom, eapi=eapi)
except InvalidDependString as e:
return [0, "%s" % (e,)]
 
-- 
2.9.0

Re: [gentoo-portage-dev] [PATCH 2/2] runtests: fix output for skipped pythons

2016-03-24 Thread Sergei Trofimovich 
On Sun, 20 Mar 2016 14:53:01 +0100
Alexander Berntsen  wrote:

> Nice! This LGMT too, and I'll commit and push it Monday with the other
> one unless there are any objections.

Thanks :)

-- 

  Sergei


pgpKDeifVM2Uk.pgp
Description: Цифровая подпись OpenPGP

[gentoo-portage-dev] [PATCH 1/2] cleanup: fix spaces to tabs in depgraph.py

2016-03-19 Thread Sergei Trofimovich 
From: Sergei Trofimovich <siarh...@google.com>

Signed-off-by: Sergei Trofimovich <siarh...@google.com>
---
 pym/_emerge/depgraph.py | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index b58fbda..cfaafa3 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -5126,9 +5126,9 @@ class depgraph(object):
break
 
writemsg("\nemerge: there are no %s to satisfy " %
-("binary packages" if
- self._frozen_config.myopts.get("--usepkgonly", "y") == True
- else "ebuilds") + green(xinfo) + ".\n", noiselevel=-1)
+   ("binary packages" if
+   self._frozen_config.myopts.get("--usepkgonly", 
"y") == True
+   else "ebuilds") + green(xinfo) + ".\n", 
noiselevel=-1)
if isinstance(myparent, AtomArg) and \
not cp_exists and \
self._frozen_config.myopts.get(
@@ -6937,7 +6937,7 @@ class depgraph(object):
for root in implicit_libc_roots:
vardb = self._frozen_config.trees[root]["vartree"].dbapi
for atom in self._expand_virt_from_graph(root,
-   portage.const.LIBC_PACKAGE_ATOM):
+   portage.const.LIBC_PACKAGE_ATOM):
if atom.blocker:
continue
for pkg in 
self._dynamic_config._package_tracker.match(root, atom):
-- 
2.7.4

Re: [gentoo-portage-dev] [PATCH] repoman: Finally deprecate base.eclass

2015-10-22 Thread Sergei Trofimovich 
On Sun, 11 Oct 2015 09:23:20 +0200
Michał Górny  wrote:

> Contributors are repeatedly adding base.eclass uses, so we should
> finally make the deprecation formal, even at the cost of adding warnings
> for some frequently used eclasses.
> 
> Signed-off-by: Michał Górny 
> ---
>  pym/repoman/checks/ebuilds/checks.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/pym/repoman/checks/ebuilds/checks.py 
> b/pym/repoman/checks/ebuilds/checks.py
> index ae6d96e..a00d518 100644
> --- a/pym/repoman/checks/ebuilds/checks.py
> +++ b/pym/repoman/checks/ebuilds/checks.py
> @@ -409,6 +409,7 @@ class InheritDeprecated(LineCheck):
>  
>   # deprecated eclass : new eclass (False if no new eclass)
>   deprecated_eclasses = {
> + "base": False,
>   "bash-completion": "bash-completion-r1",
>   "boost-utils": False,
>   "distutils": "distutils-r1",
> -- 
> 2.6.1

I would also suggest:

- adding a @DEPRECATED documentation stanza
   right into base.eclass with a migration strategy link
- and a ewarn/eqawarn call to flag all offenders and prevent
   new ebuilds to creep dependency back

-- 

  Sergei


signature.asc
Description: PGP signature

[gentoo-portage-dev] [PATCH] bin/eapi.sh: default to EAPI=0 when EAPI is unset

2015-01-17 Thread Sergei Trofimovich 
Observed as a breakage on binutils ebuilds:

 ERROR: sys-devel/binutils-2.24-r3::gentoo failed (depend phase):
  use() calls are not allowed in global scope
Call stack:
ebuild.sh, line 584:  Called source 'binutils-2.24-r3.ebuild,
ebuild.sh, line   7:  Called inherit 'toolchain-binutils'
ebuild.sh, line 280:  Called __qa_source 
 '/gentoo-32k/gentoo-x86/eclass/toolchain-binutils.e
ebuild.sh, line  80:  Called source 
 '/gentoo-32k/gentoo-x86/eclass/toolchain-binutils.eclass
  toolchain-binutils.eclass, line 106:  Called use 'multislot'
  ebuild.sh, line  47:  Called die
 The specific snippet of code:
   # These functions die because calls to them during the depend phase

Zac confirms it's a safe thing to use ${EAPI-0} here:
 ebuild.sh unsets the EAPI just before it sources the ebuild,
 and then it does this after it sources the ebuild:

   [ ${EAPI+set} = set ] || EAPI=0

 So, for any code that is called while the ebuild is being sourced, using
 ${EAPI-0} would be correct.

For consistency and defense against future copy/paste errors
converted all uses of ${EAPI} for ${EAPI-0} in 'bin/eapi.sh'.

Signed-off-by: Sergei Trofimovich sly...@gentoo.org
---
 bin/eapi.sh | 90 ++---
 1 file changed, 45 insertions(+), 45 deletions(-)

diff --git a/bin/eapi.sh b/bin/eapi.sh
index 7e7b54b..f1c677e 100644
--- a/bin/eapi.sh
+++ b/bin/eapi.sh
@@ -5,189 +5,189 @@
 # PHASES
 
 ___eapi_has_pkg_pretend() {
-   [[ ! ${1-${EAPI}} =~ ^(0|1|2|3)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3)$ ]]
 }
 
 ___eapi_has_src_prepare() {
-   [[ ! ${1-${EAPI}} =~ ^(0|1)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ ^(0|1)$ ]]
 }
 
 ___eapi_has_src_configure() {
-   [[ ! ${1-${EAPI}} =~ ^(0|1)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ ^(0|1)$ ]]
 }
 
 ___eapi_default_src_test_disables_parallel_jobs() {
-   [[ ${1-${EAPI}} =~ ^(0|1|2|3|4|4-python|4-slot-abi)$ ]]
+   [[ ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi)$ ]]
 }
 
 ___eapi_has_S_WORKDIR_fallback() {
-   [[ ${1-${EAPI}} =~ ^(0|1|2|3)$ ]]
+   [[ ${1-${EAPI-0}} =~ ^(0|1|2|3)$ ]]
 }
 
 # VARIABLES
 
 ___eapi_has_prefix_variables() {
-   [[ ! ${1-${EAPI}} =~ ^(0|1|2)$ ||  ${FEATURES}  == * force-prefix * 
]]
+   [[ ! ${1-${EAPI-0}} =~ ^(0|1|2)$ ||  ${FEATURES}  == * force-prefix 
* ]]
 }
 
 ___eapi_has_HDEPEND() {
-   [[ ${1-${EAPI}} =~ ^(5-hdepend)$ ]]
+   [[ ${1-${EAPI-0}} =~ ^(5-hdepend)$ ]]
 }
 
 ___eapi_has_RDEPEND_DEPEND_fallback() {
-   [[ ${1-${EAPI}} =~ ^(0|1|2|3)$ ]]
+   [[ ${1-${EAPI-0}} =~ ^(0|1|2|3)$ ]]
 }
 
 # HELPERS PRESENCE
 
 ___eapi_has_dohard() {
-   [[ ${1-${EAPI}} =~ ^(0|1|2|3)$ ]]
+   [[ ${1-${EAPI-0}} =~ ^(0|1|2|3)$ ]]
 }
 
 ___eapi_has_dosed() {
-   [[ ${1-${EAPI}} =~ ^(0|1|2|3)$ ]]
+   [[ ${1-${EAPI-0}} =~ ^(0|1|2|3)$ ]]
 }
 
 ___eapi_has_einstall() {
-   [[ ${1-${EAPI}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+   [[ ${1-${EAPI-0}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
 }
 
 ___eapi_has_dohtml_deprecated() {
-   [[ ! ${1-${EAPI}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
 }
 
 ___eapi_has_docompress() {
-   [[ ! ${1-${EAPI}} =~ ^(0|1|2|3)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3)$ ]]
 }
 
 ___eapi_has_nonfatal() {
-   [[ ! ${1-${EAPI}} =~ ^(0|1|2|3)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3)$ ]]
 }
 
 ___eapi_has_doheader() {
-   [[ ! ${1-${EAPI}} =~ ^(0|1|2|3|4|4-python|4-slot-abi)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi)$ ]]
 }
 
 ___eapi_has_usex() {
-   [[ ! ${1-${EAPI}} =~ ^(0|1|2|3|4|4-python|4-slot-abi)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ ^(0|1|2|3|4|4-python|4-slot-abi)$ ]]
 }
 
 ___eapi_has_get_libdir() {
-   [[ ! ${1-${EAPI}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
 }
 
 ___eapi_has_einstalldocs() {
-   [[ ! ${1-${EAPI}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
 }
 
 ___eapi_has_eapply() {
-   [[ ! ${1-${EAPI}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
 }
 
 ___eapi_has_eapply_user() {
-   [[ ! ${1-${EAPI}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
 }
 
 ___eapi_has_in_iuse() {
-   [[ ! ${1-${EAPI}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
+   [[ ! ${1-${EAPI-0}} =~ 
^(0|1|2|3|4|4-python|4-slot-abi|5|5-hdepend|5-progress)$ ]]
 }
 
 ___eapi_has_master_repositories() {
-   [[ ${1

Re: [gentoo-portage-dev] [PATCH 16/18] Disallow helpers in global scope in EAPI 6

2015-01-16 Thread Sergei Trofimovich 
On Mon,  1 Dec 2014 22:28:34 +0100
Michał Górny mgo...@gentoo.org wrote:

 Disallow calling most of the ebuild helpers in global scope since they
 are meaningless in that context. Most of them are also prohibited by PMS
 for all EAPIs, so EAPI 6 sounds like a good moment to finally enforce
 that restriction.
 ---
  bin/eapi.sh | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
 
 diff --git a/bin/eapi.sh b/bin/eapi.sh
 index e0ade02..5ab92f4 100644
 --- a/bin/eapi.sh
 +++ b/bin/eapi.sh
 @@ -159,7 +159,7 @@ ___eapi_helpers_can_die() {
  }
  
  ___eapi_disallows_helpers_in_global_scope() {
 - [[ ${1-${EAPI}} =~ ^(4-python|5-progress)$ ]]
 + [[ ! ${1-${EAPI}} =~ ^(0|1|2|3|4|4-slot-abi|5|5-hdepend)$ ]]
  }
  
  ___eapi_unpack_is_case_sensitive() {

Looks like it cries on ebuilds with unset EAPI in main tree (attached).
Can be easily fixed in tree by adding EAPI=0, but better handle it
here: ${EAPI-0} (untested!)

It also changed binutils: it used to be slotted only in presence of USE flag.
Now it's always slotted even with USE=-multislot (at least eix thinks so).

-- 

  Sergei
 * ERROR: dev-java/ibm-jdk-bin-1.6.0.9_p1::gentoo failed (depend 
phase):
 *   use() calls are not allowed in global scope
 * 
 * Call stack:
 *   ebuild.sh, line 584:  Called source 
'/gentoo-32k/gentoo-x86/dev-java/ibm-jdk-bin/ibm-jdk-bin-1.6.0.9_p1.ebuild'
 * ERROR: dev-java/ibm-jdk-bin-1.6.0.9_p2::gentoo failed (depend 
phase):
 *   use() calls are not allowed in global scope
 *   ibm-jdk-bin-1.6.0.9_p1.ebuild, line  50:  Called use 'x86'
 * 
 * Call stack:
 *   ebuild.sh, line  47:  Called die
 * The specific snippet of code:
 *   ebuild.sh, line 584:  Called source 
'/gentoo-32k/gentoo-x86/dev-java/ibm-jdk-bin/ibm-jdk-bin-1.6.0.9_p2.ebuild'
 *  # These functions die because calls to them during the 
depend phase
 * ERROR: dev-java/ibm-jre-bin-1.6.0.9_p1::gentoo failed (depend 
phase):
 * 
 *   use() calls are not allowed in global scope
 * 
 *   ibm-jdk-bin-1.6.0.9_p2.ebuild, line  50:  Called use 'x86'
 * Call stack:
 * If you need support, post the output of `emerge --info 
'=dev-java/ibm-jdk-bin-1.6.0.9_p1::gentoo'`,
 * the complete build log and the output of `emerge -pqv 
'=dev-java/ibm-jdk-bin-1.6.0.9_p1::gentoo'`.
 *   ebuild.sh, line  47:  Called die
 * The specific snippet of code:
 * Working directory: '/usr/lib64/python3.3/site-packages'
 * ERROR: dev-java/ibm-jre-bin-1.6.0.9_p2::gentoo failed (depend 
phase):
 * S: 
'/tmp/portage-tmpdir/portage/dev-java/ibm-jdk-bin-1.6.0.9_p1/work/ibm-jdk-bin-1.6.0.9_p1'
 *   use() calls are not allowed in global scope
 * 
 *  # These functions die because calls to them during the 
depend phase
 * Call stack:
 * 
 * If you need support, post the output of `emerge --info 
'=dev-java/ibm-jdk-bin-1.6.0.9_p2::gentoo'`,
 * the complete build log and the output of `emerge -pqv 
'=dev-java/ibm-jdk-bin-1.6.0.9_p2::gentoo'`.
 *   ebuild.sh, line 584:  Called source 
'/gentoo-32k/gentoo-x86/dev-java/ibm-jre-bin/ibm-jre-bin-1.6.0.9_p2.ebuild'
 *   ebuild.sh, line 584:  Called source 
'/gentoo-32k/gentoo-x86/dev-java/ibm-jre-bin/ibm-jre-bin-1.6.0.9_p1.ebuild'
 * Working directory: '/usr/lib64/python3.3/site-packages'
 * S: 
'/tmp/portage-tmpdir/portage/dev-java/ibm-jdk-bin-1.6.0.9_p2/work/ibm-jdk-bin-1.6.0.9_p2'
 *   ibm-jre-bin-1.6.0.9_p2.ebuild, line  31:  Called use 'x86'
 *   ibm-jre-bin-1.6.0.9_p1.ebuild, line  31:  Called use 'x86'
 *   ebuild.sh, line  47:  Called die
 *   ebuild.sh, line  47:  Called die
 * The specific snippet of code:
 * The specific snippet of code:
 *  # These functions die because calls to them during the 
depend phase
 *  # These functions die because calls to them during the 
depend phase
 * 
 * 
 * If you need support, post the output of `emerge --info 
'=dev-java/ibm-jre-bin-1.6.0.9_p2::gentoo'`,
 * If you need support, post the output of `emerge --info 
'=dev-java/ibm-jre-bin-1.6.0.9_p1::gentoo'`,
 * the complete build log and the output of `emerge -pqv 
'=dev-java/ibm-jre-bin-1.6.0.9_p2::gentoo'`.
 * the complete build log and the output of `emerge -pqv 
'=dev-java/ibm-jre-bin-1.6.0.9_p1::gentoo'`.
 * Working directory:

[gentoo-portage-dev] Re: [PATCH] emake: pass EXTRA_EMAKE after user's $@, not before

2014-09-14 Thread Sergei Trofimovich 
On Fri, 25 Jul 2014 01:11:47 +0300
Sergei Trofimovich sly...@gentoo.org wrote:

ping :]

 Patch allows easier override or ebuild things, like
 #foo-.ebuild:
 src_compile() {
 emake V=1
 }
 
 EXTRA_EMAKE=V=0 emerge =foo-
 
 That way it's easier to eyeball upstream builds for new warnings.
 
 This behaviour is more in line with EXTRA_ECONF variable.
 
 Signed-off-by: Sergei Trofimovich sly...@gentoo.org
 ---
  bin/ebuild-helpers/emake | 4 ++--
  bin/phase-helpers.sh | 8 
  2 files changed, 6 insertions(+), 6 deletions(-)
 
 diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
 index 69d836f..4618053 100755
 --- a/bin/ebuild-helpers/emake
 +++ b/bin/ebuild-helpers/emake
 @@ -13,7 +13,7 @@ source 
 ${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}/isolated-functions.sh
  
  if [[ $PORTAGE_QUIET != 1 ]] ; then
   (
 - for arg in ${MAKE:-make} $MAKEOPTS $EXTRA_EMAKE $@ ; do
 + for arg in ${MAKE:-make} $MAKEOPTS $@ $EXTRA_EMAKE ; do
   [[ ${arg} == * * ]] \
printf '%s'  ${arg} \
   || printf %s  ${arg}
 @@ -22,7 +22,7 @@ if [[ $PORTAGE_QUIET != 1 ]] ; then
   ) 2
  fi
  
 -${MAKE:-make} ${MAKEOPTS} ${EXTRA_EMAKE} $@
 +${MAKE:-make} ${MAKEOPTS} $@ ${EXTRA_EMAKE}
  ret=$?
  [[ $ret -ne 0 ]]  __helpers_die ${0##*/} failed
  exit $ret
 diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
 index 412decb..47bd843 100644
 --- a/bin/phase-helpers.sh
 +++ b/bin/phase-helpers.sh
 @@ -617,8 +617,8 @@ einstall() {
   mandir=${ED}usr/share/man \
   sysconfdir=${ED}etc \
   ${LOCAL_EXTRA_EINSTALL} \
 - ${MAKEOPTS} ${EXTRA_EMAKE} -j1 \
 - $@ install
 + ${MAKEOPTS} -j1 \
 + $@ ${EXTRA_EMAKE} install
   fi
   ${MAKE:-make} prefix=${ED}usr \
   datadir=${ED}usr/share \
 @@ -627,8 +627,8 @@ einstall() {
   mandir=${ED}usr/share/man \
   sysconfdir=${ED}etc \
   ${LOCAL_EXTRA_EINSTALL} \
 - ${MAKEOPTS} ${EXTRA_EMAKE} -j1 \
 - $@ install || die einstall failed
 + ${MAKEOPTS} -j1 \
 + $@ ${EXTRA_EMAKE} install || die einstall failed
   else
   die no Makefile found
   fi
 -- 
 2.0.0
 


-- 

  Sergei


signature.asc
Description: PGP signature

Re: [gentoo-portage-dev] [PATCH] QA: use pngfix to find broken PNG files.

2014-07-26 Thread Sergei Trofimovich 
On Sat, 26 Jul 2014 08:50:43 +0200
Michał Górny mgo...@gentoo.org wrote:

 + if [[ -z ${next} ]] ; then
 + eqawarn QA Notice: broken .png files 
 found:

I guess you planned to plug 'next=yes' here, right?

 + fi


-- 

  Sergei


signature.asc
Description: PGP signature

[gentoo-portage-dev] [PATCH] repoman: warn when herd's email appears in maintaineremail section

2014-07-24 Thread Sergei Trofimovich 
Manuel Rüger noticed that most of haskell packages's 'metadata.xml'
contain duplicate information:
herdhaskell/herd
maintaineremailhask...@gentoo.org/email/maintainer

I've added a check against 'herds.xml's email aliases.

Now repoman warns about such redundancy:
  metadata.warning  1
   dev-haskell/text/metadata.xml: use herdhaskell/herd instead of 
maintainer 'hask...@gentoo.org'

Quick scan [1] of tree revealed a lot of non-haskell packages
having redundancy: samba, xemacs, sci, gpe, etc, etc.

[1]: https://github.com/trofi/gentoo-qa/blob/master/check_herd.sh
Run in the root tree of gentoo-x86:
gentoo-x86 $ ~/portage/gentoo-qa/check_herd.sh  | wc -l
571
# some examples:
gentoo-x86 $ ~/portage/gentoo-qa/check_herd.sh
./app-admin/haskell-updater/metadata.xml:  emailhask...@gentoo.org/email
./app-editors/xemacs/metadata.xml:emailxem...@gentoo.org/email
./app-i18n/ibus-table-chinese/metadata.xml: 
emailc...@gentoo.org/email
./app-portage/fquery/metadata.xml:  
emailhask...@gentoo.org/email
./app-text/glosung/metadata.xml:  emailtheol...@gentoo.org/email
... # a lot of haskell
./dev-lang/tcl/metadata.xml:emailtc...@gentoo.org/email
./dev-libs/Ice/metadata.xml:emailc...@gentoo.org/email
./dev-libs/cloog/metadata.xml:  emailtoolch...@gentoo.org/email
./dev-libs/cvector/metadata.xml:emails...@gentoo.org/email
./dev-libs/iniparser/metadata.xml:  emailsa...@gentoo.org/email
./dev-libs/isl/metadata.xml:emailtoolch...@gentoo.org/email
...

Signed-off-by: Sergei Trofimovich sly...@gentoo.org
---
 bin/repoman  | 13 +
 pym/repoman/herdbase.py  | 21 +
 pym/repoman/utilities.py | 19 +++
 3 files changed, 49 insertions(+), 4 deletions(-)

diff --git a/bin/repoman b/bin/repoman
index c36ace1..74a4d44 100755
--- a/bin/repoman
+++ b/bin/repoman
@@ -1762,6 +1762,19 @@ for x in effective_scanlist:
fails[metadata.bad].append(%s/metadata.xml: 
%s % (x, e))
del e
 
+   # check if 'metadata.xml' contains redundant
+   #   
'maintaineremailsome-h...@gentoo.org/email/maintainer'
+   # email address. Instead it should contain
+   #   'herdsome-herd/herd'
+   if herd_base is not None:
+   for m_email in 
utilities.get_maintainer_emails_from_metadata(_metadata_xml):
+   herd_name = 
herd_base.herd_by_herd_email(m_email)
+   if herd_name is not None:
+   stats[metadata.warning] += 1
+   
fails[metadata.warning].append(%s/metadata.xml:
+use herd%s/herd 
instead of maintainer '%s' % \
+   (x, herd_name, m_email))
+
# Only carry out if in package directory or check forced
if xmllint_capable and not metadata_bad:
# xmlint can produce garbage output even on success, so 
only dump
diff --git a/pym/repoman/herdbase.py b/pym/repoman/herdbase.py
index c5b88ff..ae7c4e4 100644
--- a/pym/repoman/herdbase.py
+++ b/pym/repoman/herdbase.py
@@ -34,9 +34,10 @@ def _make_email(nick_name):
 
 
 class HerdBase(object):
-   def __init__(self, herd_to_emails, all_emails):
+   def __init__(self, herd_to_emails, all_emails, herd_email_to_herd):
self.herd_to_emails = herd_to_emails
self.all_emails = all_emails
+   self.herd_email_to_herd = herd_email_to_herd
 
def known_herd(self, herd_name):
return herd_name in self.herd_to_emails
@@ -47,6 +48,9 @@ class HerdBase(object):
def maintainer_in_herd(self, nick_name, herd_name):
return _make_email(nick_name) in self.herd_to_emails[herd_name]
 
+   def herd_by_herd_email(self, email):
+   return self.herd_email_to_herd.get(email)
+
 class _HerdsTreeBuilder(xml.etree.ElementTree.TreeBuilder):

Implements doctype() as required to avoid deprecation warnings with
@@ -57,6 +61,7 @@ class _HerdsTreeBuilder(xml.etree.ElementTree.TreeBuilder):
 
 def make_herd_base(filename):
herd_to_emails = dict()
+   herd_email_to_herd = dict()
all_emails = set()
 
try:
@@ -82,6 +87,11 @@ def make_herd_base(filename):
herd_name = _herd_name.text.strip()
del _herd_name
 
+   _herd_email = h.find('email')
+   herd_email = _herd_email.text.strip()
+   del _herd_email
+   herd_email_to_herd[herd_email] = herd_name
+
maintainers = h.findall('maintainer

[gentoo-portage-dev] [PATCH] repoman: warn when herd's email appears in maintaineremail section

2014-07-24 Thread Sergei Trofimovich 
Manuel Rüger noticed that most of haskell packages's 'metadata.xml'
contain duplicate information:
herdhaskell/herd
maintaineremailhask...@gentoo.org/email/maintainer

I've added a check against 'herds.xml's email aliases.

Now repoman warns about such redundancy:
  metadata.warning  1
   dev-haskell/text/metadata.xml: use herdhaskell/herd instead of 
maintainer 'hask...@gentoo.org'

Quick scan [1] of tree revealed a lot of non-haskell packages
having redundancy: samba, xemacs, sci, gpe, etc, etc.

[1]: https://github.com/trofi/gentoo-qa/blob/master/check_herd.sh
Run in the root tree of gentoo-x86:
gentoo-x86 $ ~/portage/gentoo-qa/check_herd.sh  | wc -l
571
# some examples:
gentoo-x86 $ ~/portage/gentoo-qa/check_herd.sh
./app-admin/haskell-updater/metadata.xml:  emailhask...@gentoo.org/email
./app-editors/xemacs/metadata.xml:emailxem...@gentoo.org/email
./app-i18n/ibus-table-chinese/metadata.xml: 
emailc...@gentoo.org/email
./app-portage/fquery/metadata.xml:  
emailhask...@gentoo.org/email
./app-text/glosung/metadata.xml:  emailtheol...@gentoo.org/email
... # a lot of haskell
./dev-lang/tcl/metadata.xml:emailtc...@gentoo.org/email
./dev-libs/Ice/metadata.xml:emailc...@gentoo.org/email
./dev-libs/cloog/metadata.xml:  emailtoolch...@gentoo.org/email
./dev-libs/cvector/metadata.xml:emails...@gentoo.org/email
./dev-libs/iniparser/metadata.xml:  emailsa...@gentoo.org/email
./dev-libs/isl/metadata.xml:emailtoolch...@gentoo.org/email
...

Signed-off-by: Sergei Trofimovich sly...@gentoo.org
---
 bin/repoman  | 13 +
 pym/repoman/herdbase.py  | 21 +
 pym/repoman/utilities.py | 19 +++
 3 files changed, 49 insertions(+), 4 deletions(-)

diff --git a/bin/repoman b/bin/repoman
index c36ace1..74a4d44 100755
--- a/bin/repoman
+++ b/bin/repoman
@@ -1762,6 +1762,19 @@ for x in effective_scanlist:
fails[metadata.bad].append(%s/metadata.xml: 
%s % (x, e))
del e
 
+   # check if 'metadata.xml' contains redundant
+   #   
'maintaineremailsome-h...@gentoo.org/email/maintainer'
+   # email address. Instead it should contain
+   #   'herdsome-herd/herd'
+   if herd_base is not None:
+   for m_email in 
utilities.get_maintainer_emails_from_metadata(_metadata_xml):
+   herd_name = 
herd_base.herd_by_herd_email(m_email)
+   if herd_name is not None:
+   stats[metadata.warning] += 1
+   
fails[metadata.warning].append(%s/metadata.xml:
+use herd%s/herd 
instead of maintainer '%s' % \
+   (x, herd_name, m_email))
+
# Only carry out if in package directory or check forced
if xmllint_capable and not metadata_bad:
# xmlint can produce garbage output even on success, so 
only dump
diff --git a/pym/repoman/herdbase.py b/pym/repoman/herdbase.py
index c5b88ff..ae7c4e4 100644
--- a/pym/repoman/herdbase.py
+++ b/pym/repoman/herdbase.py
@@ -34,9 +34,10 @@ def _make_email(nick_name):
 
 
 class HerdBase(object):
-   def __init__(self, herd_to_emails, all_emails):
+   def __init__(self, herd_to_emails, all_emails, herd_email_to_herd):
self.herd_to_emails = herd_to_emails
self.all_emails = all_emails
+   self.herd_email_to_herd = herd_email_to_herd
 
def known_herd(self, herd_name):
return herd_name in self.herd_to_emails
@@ -47,6 +48,9 @@ class HerdBase(object):
def maintainer_in_herd(self, nick_name, herd_name):
return _make_email(nick_name) in self.herd_to_emails[herd_name]
 
+   def herd_by_herd_email(self, email):
+   return self.herd_email_to_herd.get(email)
+
 class _HerdsTreeBuilder(xml.etree.ElementTree.TreeBuilder):

Implements doctype() as required to avoid deprecation warnings with
@@ -57,6 +61,7 @@ class _HerdsTreeBuilder(xml.etree.ElementTree.TreeBuilder):
 
 def make_herd_base(filename):
herd_to_emails = dict()
+   herd_email_to_herd = dict()
all_emails = set()
 
try:
@@ -82,6 +87,11 @@ def make_herd_base(filename):
herd_name = _herd_name.text.strip()
del _herd_name
 
+   _herd_email = h.find('email')
+   herd_email = _herd_email.text.strip()
+   del _herd_email
+   herd_email_to_herd[herd_email] = herd_name
+
maintainers = h.findall('maintainer

Re: [gentoo-portage-dev] [PATCH] Introduce depgraph._extend_slot_operator_conflicts

2014-04-05 Thread Sergei Trofimovich 
On Thu, 27 Feb 2014 20:08:49 +0100
Sebastian Luther sebastianlut...@gmx.de wrote:

 This function allows emerge to solve more slot conflicts without
 backtracking.
 
 It does this by creating more conflicts for packages with built slot
 operator dependencies. This gives more freedom to
 depgraph._solve_non_slot_operator_slot_conflicts, which is then able
 to solve conflicts it wouldn't have otherwise.
 ---

Nice patch!
For haskell packages it makes huge difference.

My typical workflow is:
- bump some package foo and it's subslot
- emerge -1 foo -j9 
- see what will get rebuilt/broken by subslot rebuild

If package would trigger 500 rebuilds portage had no chance
to find a solution with backtracks. Now sometimes it does.

Thanks!

  pym/_emerge/depgraph.py| 483 
 +++--
  pym/_emerge/resolver/output.py |   5 +-
  2 files changed, 415 insertions(+), 73 deletions(-)
 
 diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
 index 835bd6b..970a9c7 100644
 --- a/pym/_emerge/depgraph.py
 +++ b/pym/_emerge/depgraph.py
 @@ -427,6 +427,12 @@ class _dynamic_depgraph_config(object):
   # Track missed updates caused by solved conflicts.
   self._conflict_missed_update = collections.defaultdict(dict)
  
 + # Rebuilds caused by slot conflicts.
 + self._slot_conflict_rebuilds = {}
 + # Rebuilds caused by missed slot operator updates or
 + # slot conflicts.
 + self._forced_rebuilds = None
 +
   for myroot in depgraph._frozen_config.trees:
   self.sets[myroot] = _depgraph_sets()
   vardb = 
 depgraph._frozen_config.trees[myroot][vartree].dbapi
 @@ -614,6 +620,9 @@ class depgraph(object):
   Fill self._forced_rebuilds with packages that cause rebuilds.
   
  
 + if self._dynamic_config._forced_rebuilds is not None:
 + return
 +
   debug = --debug in self._frozen_config.myopts
  
   # Get all atoms that might have caused a forced rebuild.
 @@ -687,19 +696,23 @@ class depgraph(object):
   writemsg_level(\n\n,
   level=logging.DEBUG, noiselevel=-1)
  
 - self._forced_rebuilds = forced_rebuilds
 + for child, parents in 
 self._dynamic_config._slot_conflict_rebuilds.items():
 + forced_rebuilds.setdefault(child.root, 
 {}).setdefault(child, set()).update(parents)
 +
 + self._dynamic_config._forced_rebuilds = forced_rebuilds
  
   def _show_abi_rebuild_info(self):
  
 - if not self._forced_rebuilds:
 + forced_rebuilds = self._dynamic_config._forced_rebuilds
 + if not forced_rebuilds:
   return
  
   writemsg_stdout(\nThe following packages are causing 
 rebuilds:\n\n, noiselevel=-1)
  
 - for root in self._forced_rebuilds:
 - for child in self._forced_rebuilds[root]:
 + for root in forced_rebuilds:
 + for child in forced_rebuilds[root]:
   writemsg_stdout(  %s causes rebuilds for:\n % 
 (child,), noiselevel=-1)
 - for parent in 
 self._forced_rebuilds[root][child]:
 + for parent in forced_rebuilds[root][child]:
   writemsg_stdout(%s\n % (parent,), 
 noiselevel=-1)
  
   def _show_ignored_binaries(self):
 @@ -968,16 +981,16 @@ class depgraph(object):
   writemsg(line + '\n', noiselevel=-1)
   writemsg('\n', noiselevel=-1)
  
 - def _solve_non_slot_operator_slot_conflicts(self):
 +
 + def _extend_slot_operator_conflicts(self):
   
 - This function solves slot conflicts which can
 - be solved by simply choosing one of the conflicting
 - and removing all the other ones.
 - It is able to solve somewhat more complex cases where
 - conflicts can only be solved simultaniously.
 + This function searches for packages that cause
 + slot conflicts by dependening on conflict packages
 + with built slot operator deps. If such a package
 + is found an alternative package is pulled in with
 + the hope that the alternative package would not
 + cuase the slot conflict.
   
 - debug = --debug in self._frozen_config.myopts
 -
   # List all conflicts. Ignore those that involve slot operator 
 rebuilds
   # as the logic there needs special slot conflict behavior which 
 isn't
   # provided by this function.
 @@ -990,9 +1003,133 @@ class depgraph(object):
   if not conflicts:
   return
  
 - # Get a set of all conflicting packages.
 +

Re: [gentoo-portage-dev] [PATCH v2] prepman: do not compress files =128 bytes

2013-03-21 Thread Sergei Trofimovich 
On Thu, 21 Mar 2013 07:51:49 -0400
James Cloos cl...@jhcloos.com wrote:

  MF == Mike Frysinger vap...@gentoo.org writes:
 
 MF +SIZE_LIMIT='128'
 
 The limit probably should be larger, given that nothing which fits into
 one filesystem block will use less space compressed.

While it's usually true btrfs does not store things in blocks and
allocates space in byte ranges. All filesystem (data or meta)
modifications are copy-on-write thus it does not needs growth holes
by design.

-- 

  Sergei


signature.asc
Description: PGP signature