Re: [gentoo-portage-dev] [PATCH] depgraph: fix slot operator rebuild for llvm:0 to llvm:4 upgrade (bug 612772)

[Zac Medico]

On Wed, Mar 15, 2017 at 9:39 PM, Brian Dolbec  wrote:
> On Wed, 15 Mar 2017 20:23:43 -0700
> Zac Medico  wrote:
>
>> Fix check_reverse_dependencies to ignore dependencies of parent
>> packages that could be uninstalled in order to solve a blocker
>> conflict. This case is similar to the one from bug 584626, except
>> that the relevant parent package is in an older slot which is blocked
>> by a newer slot. In this case, the _upgrade_available method returns
>> False, because the package in the older slot is the highest version
>> version available for its slot. Therefore, a new _in_blocker_conflict
>> method is needed to detect parent packages that cold be uninstalled.
>> The included unit test fails without this fix.
>>
>> Since the _in_blocker_conflict method requires information that is
>> collected by the _validate_blockers method, the _validate_blockers
>> method now has to be called before the _process_slot_conflict and
>> _slot_operator_trigger_reinstalls methods.
>>
>> X-Gentoo-bug: 612772
>> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=612772
>> ---
>>  pym/_emerge/depgraph.py|  59 ---
>>  .../resolver/test_slot_operator_exclusive_slots.py | 109
>> + 2 files changed, 155 insertions(+), 13
>> deletions(-) create mode 100644
>> pym/portage/tests/resolver/test_slot_operator_exclusive_slots.py
>>
>> diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
>> index 1379b05..96b6d5f 100644
>> --- a/pym/_emerge/depgraph.py
>> +++ b/pym/_emerge/depgraph.py
>> @@ -387,7 +387,10 @@ class _dynamic_depgraph_config(object):
>>   # Contains only unsolvable Package -> Blocker edges
>>   self._unsolvable_blockers = digraph()
>>   # Contains all Blocker -> Blocked Package edges
>> - self._blocked_pkgs = digraph()
>> + # Do not initialized this until the depgraph
>   /
>   typo  -d
>
>
> Otherwise looks good :)

Thanks, merged with typo fix:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=15e67f5516e0779d2cba37704c15b42193808197
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH] depgraph: fix slot operator rebuild for llvm:0 to llvm:4 upgrade (bug 612772)

[Zac Medico]

Fix check_reverse_dependencies to ignore dependencies of parent packages
that could be uninstalled in order to solve a blocker conflict. This case
is similar to the one from bug 584626, except that the relevant parent
package is in an older slot which is blocked by a newer slot. In this
case, the _upgrade_available method returns False, because the package
in the older slot is the highest version version available for its
slot. Therefore, a new _in_blocker_conflict method is needed to detect
parent packages that cold be uninstalled. The included unit test fails
without this fix.

Since the _in_blocker_conflict method requires information that is
collected by the _validate_blockers method, the _validate_blockers
method now has to be called before the _process_slot_conflict and
_slot_operator_trigger_reinstalls methods.

X-Gentoo-bug: 612772
X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=612772
---
 pym/_emerge/depgraph.py|  59 ---
 .../resolver/test_slot_operator_exclusive_slots.py | 109 +
 2 files changed, 155 insertions(+), 13 deletions(-)
 create mode 100644 
pym/portage/tests/resolver/test_slot_operator_exclusive_slots.py

diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index 1379b05..96b6d5f 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -387,7 +387,10 @@ class _dynamic_depgraph_config(object):
# Contains only unsolvable Package -> Blocker edges
self._unsolvable_blockers = digraph()
# Contains all Blocker -> Blocked Package edges
-   self._blocked_pkgs = digraph()
+   # Do not initialized this until the depgraph _validate_blockers
+   # method is called, so that the _in_blocker_conflict method can
+   # assert that _validate_blockers has been called first.
+   self._blocked_pkgs = None
# Contains world packages that have been protected from
# uninstallation but may not have been added to the graph
# if the graph is not complete yet.
@@ -1466,9 +1469,22 @@ class depgraph(object):
 
self._solve_non_slot_operator_slot_conflicts()
 
+   if not self._validate_blockers():
+   # Blockers don't trigger the _skip_restart flag, since
+   # backtracking may solve blockers when it solves slot
+   # conflicts (or by blind luck).
+   raise self._unknown_internal_error()
+
+   # Both _process_slot_conflict and 
_slot_operator_trigger_reinstalls
+   # can call _slot_operator_update_probe, which requires that
+   # self._dynamic_config._blocked_pkgs has been initialized by a
+   # call to the _validate_blockers method.
for conflict in 
self._dynamic_config._package_tracker.slot_conflicts():
self._process_slot_conflict(conflict)
 
+   if self._dynamic_config._allow_backtracking:
+   self._slot_operator_trigger_reinstalls()
+
def _process_slot_conflict(self, conflict):
"""
Process slot conflict data to identify specific atoms which
@@ -1829,9 +1845,12 @@ class depgraph(object):
not 
self._frozen_config.excluded_pkgs.
findAtomForPackage(parent,

modified_use=self._pkg_use_enabled(parent)) and
-   
self._upgrade_available(parent)):
+   
(self._upgrade_available(parent) or
+   (parent.installed and 
self._in_blocker_conflict(parent:
# This parent may be 
irrelevant, since an
-   # update is available (see bug 
584626).
+   # update is available (see bug 
584626), or
+   # it could be uninstalled in 
order to solve
+   # a blocker conflict (bug 
612772).
continue
 
atom_set = 
InternalPackageSet(initial_atoms=(atom,),
@@ -2125,6 +2144,24 @@ class depgraph(object):
 
self._dynamic_config._need_restart = True
 
+   def _in_blocker_conflict(self, pkg):
+   """
+   Check if pkg is involved in a blocker conflict. This method
+   only works after the _validate_blockers method has been called.
+   """
+
+   if self._dynamic_config._blocked_pkgs is None:
+   raise AssertionError(
+   '_in_blocker_conflict called 

Re: [gentoo-portage-dev] [PATCH v3] movefile: support in-kernel file copying on Linux (bug 607868)

[Zac Medico]

On Wed, Mar 15, 2017 at 3:25 PM, Brian Dolbec  wrote:
> On Fri,  3 Mar 2017 18:18:50 -0800
> Zac Medico  wrote:
>
>> Perform in-kernel file copying when possible, and also support
>> reflinks and sparse files. If the optimized implementation
>> fails at runtime, gracefully fallback to a plain read/write
>> loop.
>>
>> Compile-time and run-time fallbacks are implemented, so that
>> any incompatiblities will be handled gracefully. For example,
>> if the code is compiled on a system that supports the
>> copy_file_range syscall, but at run-time an older kernel that
>> does not support this syscall is detected, it will be handled
>> gracefully. There are similar fallbacks for lack of lseek
>> SEEK_DATA and sendfile support.
>>
>> X-Gentoo-Bug: 607868
>> X-Gentoo-Bug-Url: https://bugs.gentoo.org/show_bug.cgi?id=607868
>> ---
>> [PATCH v3] Changes:
>>  * Add function documentation comments
>>  * Rename do_lseek function to do_lseek_data
>>  * Fix do_lseek_data to handle sparse blocks at EOF, and
>>fix _reflink_linux_file_copy to call ftruncate
>>  * Eliminate indirection in python copyfile function
>>  * Cleaned up error variable handling
>>  * Added lseek calls to ensure correct revovery from EINTR
>>  * Added buf != NULL call before free(buf)
>>
>>  pym/portage/tests/util/file_copy/__init__.py  |   0
>>  pym/portage/tests/util/file_copy/__test__.py  |   0
>>  pym/portage/tests/util/file_copy/test_copyfile.py |  71 
>>  pym/portage/util/file_copy/__init__.py|  36 ++
>>  pym/portage/util/movefile.py  |   4 +-
>>  setup.py  |   9 +
>>  src/portage_util_file_copy_reflink_linux.c| 385
>> ++ 7 files changed, 504 insertions(+), 1
>> deletion(-) create mode 100644
>
>
> I'd say go for it, merge.  There's been no additional feedback or review

Thanks, merged:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=8ab5c8835931fd9ec098dbf4c5f416eb32e4a3a4
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PROPOSAL] Don't split user visible messages across multiple lines

[Brian Dolbec]

On Mon, 27 Feb 2017 20:33:28 +0200
Alexandru Elisei  wrote:

> I was working on emerge --sync and my test repo configuration
> generated several warning messages. Finding the exact location where
> the warnings was generated was surprisingly difficult because the
> message string was split across three lines due to the 80 character
> line limit and I had to grep for different patterns until I found one
> that matched part of a line.
> 
> I propose that messages that are visible to the user never be split
> across multiple lines. This is also the coding style convention used
> by the linux kernel:
> https://www.kernel.org/doc/html/latest/process/coding-style.html#breaking-long-lines-and-strings
> 
> If accepted, this could go in the DEVELOPING file.
> 

That could be pretty hard to do for all messages.
Especially messages with embedded data

  eg:  "%s is missing %s required use flag..." % ('sys-apps/foo', 'bar') 


I know I don't always enforce the line length for a few characters, 
also when clarity is more important than line length.

We could also increase the max. line length to something like 120 or 130. 

-- 
Brian Dolbec 

Re: [gentoo-portage-dev] [PATCH v3] movefile: support in-kernel file copying on Linux (bug 607868)

[Brian Dolbec]

On Fri,  3 Mar 2017 18:18:50 -0800
Zac Medico  wrote:

> Perform in-kernel file copying when possible, and also support
> reflinks and sparse files. If the optimized implementation
> fails at runtime, gracefully fallback to a plain read/write
> loop.
> 
> Compile-time and run-time fallbacks are implemented, so that
> any incompatiblities will be handled gracefully. For example,
> if the code is compiled on a system that supports the
> copy_file_range syscall, but at run-time an older kernel that
> does not support this syscall is detected, it will be handled
> gracefully. There are similar fallbacks for lack of lseek
> SEEK_DATA and sendfile support.
> 
> X-Gentoo-Bug: 607868
> X-Gentoo-Bug-Url: https://bugs.gentoo.org/show_bug.cgi?id=607868
> ---
> [PATCH v3] Changes:
>  * Add function documentation comments
>  * Rename do_lseek function to do_lseek_data
>  * Fix do_lseek_data to handle sparse blocks at EOF, and
>fix _reflink_linux_file_copy to call ftruncate
>  * Eliminate indirection in python copyfile function
>  * Cleaned up error variable handling
>  * Added lseek calls to ensure correct revovery from EINTR
>  * Added buf != NULL call before free(buf)
> 
>  pym/portage/tests/util/file_copy/__init__.py  |   0
>  pym/portage/tests/util/file_copy/__test__.py  |   0
>  pym/portage/tests/util/file_copy/test_copyfile.py |  71 
>  pym/portage/util/file_copy/__init__.py|  36 ++
>  pym/portage/util/movefile.py  |   4 +-
>  setup.py  |   9 +
>  src/portage_util_file_copy_reflink_linux.c| 385
> ++ 7 files changed, 504 insertions(+), 1
> deletion(-) create mode 100644


I'd say go for it, merge.  There's been no additional feedback or review


-- 
Brian Dolbec 

Re: [gentoo-portage-dev] Re: [PATCH v2] Change how the tmp file for the commit msg is made (bug 571546)

[Brian Dolbec]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Mon, 23 May 2016 12:43:04 +0200
Alexander Berntsen  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> I see you have incorporated Zac's feedback by using shutil. You have
> also incorporated my feedback on the commit message. I forgot to say
> that the message body should also begin in the imperative mood, so it
> should be "Change" not "Changes". But that's OK, we can just change
> this when we merge it. My bad.
> 
> The patch still looks OK to my eyes. I will leave it to Brian to ack,
> as repoman has been his baby these past months. (Brian, I can merge it
> if you don't have the time, but please give an ack.)
> - -- 
> Alexander
> berna...@gentoo.org
> https://secure.plaimi.net/~alexander
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2
> 
> iQIcBAEBCgAGBQJXQt64AAoJENQqWdRUGk8BksQP/iV301h2bP32hHeAh0e+YUHH
> NmUvwx1KSAnAjaL28rTEE8LzJcy+QDTdwhW9Y9x7gUD0fWeUuQpMXEnIQ7DIVmm/
> XfNI8M9bey90VLa8VNzIkfpHqeb5asvkHSLlyVVCb/BcE/wR23peRdW4F9LAhrAs
> 0Mdz+g09Bjb5SawiTyaNig+YSid8MDWTDI+6oskB8NAhyaDqaUKGk/HfGqAG95FW
> lbXIBE4dIQ2jFdzyn+3/rXWXBVCuXL1OufQPSiIFwfsU4j957SRo/cYlDqXyyQXD
> mXXju2fU3cEk16Po+7v9wNp77xu2HdML7jgO75HdSDN5eDMJ3gyFlrlLw1Ztw9OA
> v4+IXcbgGEGguN4Tz+DuY1+htGRnjBR5C2dj6y6rZU9HbIMTzJSxI/rQQUsQJ53j
> Nd6++GJXqpskDhd+Zx1Gmhj5yV/h7Nc/YDUsT6CTsLNOzVJUy6acyhWqJRt46MAs
> 9nw1c9xqoGLhCiWvAK5PZOz4/cfEUtEawUrQg4wtoPap05qzMx8fsk3rX9J7ifrq
> rQlBVxwGQqjN9ykzDl2IkY8/W9sx8Nhe5/M5rmUCsk0KUzhcrkQNCOHdzSSU2jSM
> Yimj4y+gBnizHIVn2+sAZ4vosTXic/zVKi5meYgSM4pBRAwD4GJgQ5vV//cWTTZl
> 6QTa+hD/NQZAQd14COhc
> =9N4L
> -END PGP SIGNATURE-
> 

Sorry, merged finally

- -- 
Brian Dolbec 

-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQKTBAEBCgB9FiEEpdfHTggcxw20pKr1+70IcnWCDtgFAljJu0JfFIAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEE1
RDdDNzRFMDgxQ0M3MERCNEE0QUFGNUZCQkQwODcyNzU4MjBFRDgACgkQ+70IcnWC
DtibPg/+IyqLtruXwdmDMuUyr+CJtnPlF2k6WD9JK5xp8j8yqAnZ3VhmvWQeLzL0
Gs0PkcquVuNKie6vVA+fgoyS73ji1KGX8c5ZO0eE5zU4tGEOAV3zL15crJ3hZvSl
W4vlQlPJgGigld4ztb15nUwYn+O0w+K03uabYhSORBCHRvuDsKTB23+KYJaM9SPC
upM4xA+ThQCdF4Vc8d1MiKz12Zmyfpkpmq5h2wHPGfkq5JEf5Hct879l278lD9cl
TFrH6BXKKF92isr1bQx6u7Fmap6CT5PHmrKyipDPnI5GEqRQY2j4BuG67DFvQdBp
/OwfOKZWOC+lRkGUFVV7WT0EBWBdGTsEe4KWE8pv8wWYrZi8xxqzJPhUcbxyS+uB
42Ry1OM0XkPd2pI+S/yVAJR+QVQDspvRrkOxN850Sf1aJeyCh8h5zCpb/oY0Bhvg
siKdjoHd7/xHp0CATiMdlfYQsZJNhRs/f2ks4S+LH0MZr4VabclIG9SN19sHWR00
+TC6mvwCFoQpxDS+VNby4iWRL2mnLgCr3FYCVapbA6Mypr/I2nvfBwQwb3OYlz39
VWD/yDN18uKYk12Gm6VXRQaJImPr4H4+gfXp1zd/FdBkIV4oKpVo03Xp2qcIIzR9
fbI+hUbMVGlcthS4hbxQHJqPCp1PXD/dAoFL/f7yNK/AJiM85ic=
=mt0p
-END PGP SIGNATURE-

[gentoo-dev] slight behaviour change of clock service in openrc-0.24.1

[William Hubbs]

All,

this is a heads-up more than anything; I don't think it is major. I
wrote an entry about it in the NEWS.md file of openrc-0.21.4.

There is no action for users, so I don't think it is a newsitem, but
there might be action for developers.

In a nutshell, if you have a service that should be added to the boot
runlevel and it needs to start after the clock service, you may need to
specify "after clock" in the depend() function.

The short version of the reason for this is I had to remove "before *"
from the clock service scripts to avoid deadlocks if rc_parallel=yes [1].

William

[1] https://bugs.gentoo.org/show_bug.cgi?id=612618


signature.asc
Description: Digital signature

Re: [gentoo-dev] RFC: Pre-GLEP: Security Project

[Kristian Fiskerstrand]

On 03/15/2017 08:12 AM, Alexis Ballier wrote:
> On Tue, 14 Mar 2017 19:55:44 -0400


> 
> 
> Agreed, but I was under the impression that sometimes sec. team was
> waiting for cleanup to close a bug. If you've just done the analysis
> that it is the only thing left, just do it and close the bug, instead
> of pinging on the bug and re-do that analysis in a later pass. This
> reduces context switches and makes everything more efficient :)
> 


Indeed, although it should be noted that the amount of context switches
is reduced by using whiteboards to tag status along with version
information in summary, which is why it is important they follow
security team policies for security bugs.

In particular the whiteboard status allows for effective filtering when
creating work-lists to reduce context switching (so if you're a
maintainer starting a stablereq, feel free to update whiteboard from
ebuild to stable!)

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Pre-GLEP: Security Project

[Alexis Ballier]

On Tue, 14 Mar 2017 19:55:44 -0400
Yury German  wrote:

> > On Mar 12, 2017, at 4:14 AM, Alexis Ballier 
> > wrote:
> > 
> > 
> > Also, it'd be nice to have something more formal for sec. cleanup:
> > "After 30 days a sec. issue has been fixed, sec. team is free to
> > cleanup old vulnerable versions.". I've seen too much pings by sec.
> > team members on old bugs for this and they could have spent the same
> > amount of time simply doing it instead.  
> 
> 
> Alexis, here is a problem that I have noticed over the years.
> Everyone is short on time, but if the developers do not step in (and
> only some) and clean up the packages then we might as well make this
> another job for Security team as everyone will just leave it to
> security.
> 
> Security looks at every security bug, and needs to do a lot of things
> behind the scenes. GLSA writing, look for CVE’s if not there, assign
> them to bugs in the CVE system used for GLSA’s. If no CVE’s exist
> communicate with upstream to see if it was requested, if not
> requested request it on their behalf and work with MITRE in getting
> it assigned. When you multiply that time over the 100+ security bugs
> at any time. Cleanup is not a 5 second thing as for me typing three
> characters to have that bug be submitted with that comment. 
> 
> The maintainer also knows the package, dependencies, other bugs
> filed, etc. Removing things for your packages might be simple, but it
> is not the same across all packages and that is the reason we ask the
> Maintainers to take an active step in cleaning up.


Agreed, but I was under the impression that sometimes sec. team was
waiting for cleanup to close a bug. If you've just done the analysis
that it is the only thing left, just do it and close the bug, instead
of pinging on the bug and re-do that analysis in a later pass. This
reduces context switches and makes everything more efficient :)

Re: [gentoo-dev] Unused profiles

[Michał Górny]

W dniu 14.03.2017, wto o godzinie 17∶19 -0700, użytkownik Jack Morgan
napisał:
> 
> On 01/19/17 09:08, Michał Górny wrote:
> > (CC-ing a lot of potentially interested teams)
> > 
> > Hi, everyone.
> > 
> > I've did a quick sweep of profiles/ directory for unused profiles.
> > Unused means: not used as parent of any other profile, and not listed
> > in profiles.desc.
> > 
> > Note that the list does not include parents of the listed profiles
> > since they are still used by them ;-).
> > 
> > Please let me know which of the profiles are worth keeping, and which
> > can be killed entirely. If you prefer to keep them, please consider
> > adding them to profiles.desc so that our users can officially use them.
> > 
> > List follows
> > 
> 
> Just to follow up, for all Sparc profiles
> 
> I use two profiles..
> default/linux/sparc/13.0 *
> default/linux/sparc/13.0/desktop
> 
> I'm happy to remove these profiles...
> default/linux/sparc/13.0/desktop/gnome
> default/linux/sparc/13.0/desktop/gnome/systemd
> default/linux/sparc/13.0/developer
> 
> anyone else?

Would you do the removal or should I?

-- 
Best regards,
Michał Górny


signature.asc
Description: This is a digitally signed message part