Re: [gentoo-portage-dev] [PATCH] Fix unsupported eapi continuing with remaining checks
On 04/30/2016 09:42 AM, Brian Dolbec wrote: > From 6ead9b76de648747ae72ac572768ca1f464ff23e Mon Sep 17 00:00:00 2001 > From: Brian Dolbec > Date: Sat, 30 Apr 2016 08:55:41 -0700 > Subject: [PATCH] repoman: Fix unsupported eapi continuing with remaining > checks. bug 581600 > > Tighten up the xpkg loop and the use of the xpkg_continue bolean for invalid > pkg or ebuilds. > This still allows addtional checks for some types of errors, while completely > bypassing the > remaining checks for EAPI.unsupported. > > X-gentoo-bug: 581600 > X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=581600 > --- > > I don't know if we should add more cases for it to abandon ship ;) and > not perform more checks on the ebuilds with some of the failures > detected in these exceptions. What wre your thoughts? It seems like can_force indicates that a very severe problem has been found, but it doesn't keep track of the specific problem that triggers the state. I think it would make sense to use the qatracker to keep track of this sort of state, with certain issues having a severity level which triggers the "can't force" state. Also, as we've discussed on irc, I think we should turn the plugin check methods into generator methods that yield a series of issues. That way, plugins will not need direct access to the qatracker, since they will simply yield issues to it (including those issues that trigger the "can't force" state). -- Thanks, Zac
[gentoo-portage-dev] [PATCH] Manifest._apply_max_mtime: handle EPERM from utime (bug 582388)
Only warn if utime fails due to the Manifest parent directory
being owned by a different user, since it's not a problem
unless the repo is being prepared for distribution via rsync.
X-Gentoo-bug: 582388
X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=582388
---
pym/portage/manifest.py | 17 ++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index f696f84..fe4166c 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -1,10 +1,11 @@
-# Copyright 1999-2014 Gentoo Foundation
+# Copyright 1999-2016 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
from __future__ import unicode_literals
import errno
import io
+import logging
import re
import stat
import sys
@@ -15,7 +16,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.checksum:hashfunc_map,perform_multiple_checksums,' + \
'verify_all,_apply_hash_filter,_filter_unaccelarated_hashes',
'portage.repository.config:_find_invalid_path_char',
- 'portage.util:write_atomic',
+ 'portage.util:write_atomic,writemsg_level',
)
from portage import os
@@ -387,7 +388,17 @@ class Manifest(object):
if max_mtime is not None:
for path in preserved_stats:
- os.utime(path, (max_mtime, max_mtime))
+ try:
+ os.utime(path, (max_mtime, max_mtime))
+ except OSError as e:
+ # Even though we have write permission,
utime fails
+ # with EPERM if path is owned by a
different user.
+ # Only warn in this case, since it's
not a problem
+ # unless this repo is being prepared
for distribution
+ # via rsync.
+ writemsg_level('!!! utime(\'%s\', (%s,
%s)): %s\n' %
+ (path, max_mtime, max_mtime, e),
+ level=logging.WARNING,
noiselevel=-1)
def sign(self):
""" Sign the Manifest """
--
2.7.4
Re: [gentoo-portage-dev] [PATCH] dblink: add locks for parallel-install with blockers (bug 576888)
On 03/14/2016 11:36 AM, Brian Dolbec wrote: > On Mon, 14 Mar 2016 11:26:23 +0100 > Alexander Berntsen wrote: > >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA512 > >> I can't say much more than "ACK, probably makes sense" really. But >> please test this *a lot* before merging it. > > > I ack as well, the code looks good. I don't know enough about to be > able to critique it in detail ;). But it does look decent and the idea > of what it is doing sounds good. > > >> Regarding the merging of this patch, and th egencache patch that has >> already been released: I thought we agreed that .29 should be *only* >> the repoman merger, and then bug fixes go into a .30 where we try to >> get a stable release with the new repoman. Why was egencache merged >> anyway? Should we not merge repoman to stable ASAP before doing >> anything else? That would make .29 easier. >> - -- >> Alexander >> berna...@gentoo.org >> https://secure.plaimi.net/~alexander > > With a .29 release coming out very soon after the the .28, the .28 > would not get much more testing for the stabilization. If only the > repoman code was changed, it makes it easier to know that any bugs > submitted for .29 that re not repoman specific, apply to .28 as well. > But more that if no non-repoman bugs were filed, then that clears .28 > for stabilization. Can we merge this now? Feedback from the user who reported the issue is very positive: https://bugs.gentoo.org/show_bug.cgi?id=576786#c7 -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] news.py: Check only for major version when parsing
On 05/18/2016 01:36 AM, Alexander Berntsen wrote:
> Only check the major version of news items, as GLEP 42 specifies an
> upgrade path for them. Future revisions to news item formats may yield
> minor number increments. GLEP 42 further ensures that only
> forwards-compatible changes may incur, as incompatible changes demand a
> major version increment.
>
> Suggested-by: Ulrich Müller
> Signed-off-by: Alexander Berntsen
> ---
> pym/portage/news.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/pym/portage/news.py b/pym/portage/news.py
> index 784ba70..f52d96e 100644
> --- a/pym/portage/news.py
> +++ b/pym/portage/news.py
> @@ -270,7 +270,7 @@ class NewsItem(object):
> # Optimization to ignore regex matchines on lines that
> # will never match
> format_match = _formatRE.match(line)
> - if format_match is not None and format_match.group(1)
> != '1.0':
> + if format_match is not None and format_match.group(1)
> != '1.*':
> invalids.append((i + 1, line.rstrip('\n')))
> break
> if not line.startswith('D'):
>
I think we want to use something like this:
if (format_match is not None and
not fnmatch.fnmatch(format_match.group(1), '1.*')):
--
Thanks,
Zac
[gentoo-portage-dev] Re: [PATCH v2] Change how the tmp file for the commit msg is made (bug 571546)
On Tue, May 17, 2016 at 11:43 PM, Alexander Berntsen wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On 17/05/16 15:36, Doug Goldstein wrote: >> ping? I believe I covered all the review items but I see this is >> not in master yet. Is there anything further you need me to do? > It seems you are replying to a thread with a second version of your > patch. I don't have the initial email to this thread. I don't know > what fucked up, or where. Could you resend the patch? Sorry about this. This continues a thread that originally started on the dev-portage alias, but I don't see a mail PATCH v2 anywhere either. Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] news.py: Check only for major version when parsing
On 05/18/2016 10:04 AM, Alexander Berntsen wrote: > On 18/05/16 17:46, Zac Medico wrote: >> not fnmatch.fnmatch(format_match.group(1), '1.*')): > Wait, why "not"? Because this code searches for items which do NOT match the pattern, which is why the existing code uses a != operator. -- Thanks, Zac
[gentoo-portage-dev] Re: [PATCH v2] Change how the tmp file for the commit msg is made (bug 571546)
On 05/18/2016 11:57 AM, Doug Goldstein wrote: > On 5/18/16 11:38 AM, Zac Medico wrote: >> On Tue, May 17, 2016 at 11:43 PM, Alexander Berntsen >> wrote: >>> -BEGIN PGP SIGNED MESSAGE- >>> Hash: SHA512 >>> >>> On 17/05/16 15:36, Doug Goldstein wrote: >>>> ping? I believe I covered all the review items but I see this is >>>> not in master yet. Is there anything further you need me to do? >>> It seems you are replying to a thread with a second version of your >>> patch. I don't have the initial email to this thread. I don't know >>> what fucked up, or where. Could you resend the patch? Sorry about this. >> >> This continues a thread that originally started on the dev-portage >> alias, but I don't see a mail PATCH v2 anywhere either. >> >> Thanks, >> Zac >> > > I sent it to gentoo-portage-dev@lists.gentoo.org as requested. I can > resend it. > It was probably during a certain window of time a couple months back when no mails were reaching the list. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH v2] news.py: Check only for major version when parsing
On 05/19/2016 03:12 AM, Alexander Berntsen wrote:
> Only check the major version of news items, as GLEP 42 specifies an
> upgrade path for them. Future revisions to news item formats may yield
> minor number increments. GLEP 42 further ensures that only
> forwards-compatible changes may incur, as incompatible changes demand a
> major version increment.
>
> Suggested-by: Ulrich Müller
> Signed-off-by: Alexander Berntsen
> ---
> pym/portage/news.py | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/pym/portage/news.py b/pym/portage/news.py
> index 784ba70..ea1b947 100644
> --- a/pym/portage/news.py
> +++ b/pym/portage/news.py
> @@ -9,6 +9,7 @@ __all__ = ["NewsManager", "NewsItem", "DisplayRestriction",
> "DisplayInstalledRestriction",
> "count_unread_news", "display_news_notifications"]
>
> +import fnmatch
> import io
> import logging
> import os as _os
> @@ -270,7 +271,8 @@ class NewsItem(object):
> # Optimization to ignore regex matchines on lines that
> # will never match
> format_match = _formatRE.match(line)
> - if format_match is not None and format_match.group(1)
> != '1.0':
> + if (format_match is not None and
> + not
> fnmatch.fnmatch(format_match.group(1), '1.*')):
> invalids.append((i + 1, line.rstrip('\n')))
> break
> if not line.startswith('D'):
>
Looks good.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v2 2/2] portage.package.ebuild.config: Always export filtered USE_EXPAND vars
On 05/21/2016 12:15 AM, Michał Górny wrote: > Ensure that all USE_EXPAND variables are always exported with filtered > USE flags inside, even if none of those flags are declared in IUSE. > This is the behavior required for EAPI 5+ by the PMS. > > Since the behavior for earlier EAPIs is left undefined and having > different behavior would be confusing to users, apply it in earlier > EAPIs as well. The patch looks correct. However, I think it's arguable that the existing portage behavior makes more sense that what PMS dictates, because why should USE_EXPAND have any effect on a package which doesn't declare the corresponding flags in IUSE? Anyway, given the invasive nature of the proposed change, it's probably a good idea to have the council weigh in on this. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 2/2] portage.util.configparser: Commonize portable config file reading routine
On 05/22/2016 01:41 AM, Michał Górny wrote: > --- > pym/portage/_sets/__init__.py | 29 ++- > pym/portage/repository/config.py | 38 ++--- > pym/portage/util/_desktop_entry.py | 20 ++--- > pym/portage/util/configparser.py | 57 > +- > 4 files changed, 64 insertions(+), 80 deletions(-) These patches look good. Note that the _native_kwargs function is not really needed anymore, because it was just a workaround for this issue which only affected python 2.6.4 and earlier: http://bugs.python.org/issue4978 -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH v2 2/2] Colorize packages in world_sets (bug 583164)
On 05/26/2016 02:01 AM, Alexander Berntsen wrote: > Looks OK to me, but I'll set Zac ACK it too, since he was the one > commenting on the bug. Looks good to me. ACK. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 1/3] portage.package.ebuild.config: Move FEATURES=no* handling there
On 05/21/2016 11:56 PM, Michał Górny wrote:
> diff --git a/pym/portage/package/ebuild/config.py
> b/pym/portage/package/ebuild/config.py
> index 45b7d08..fcc7ce5 100644
> --- a/pym/portage/package/ebuild/config.py
> +++ b/pym/portage/package/ebuild/config.py
> @@ -1773,6 +1773,16 @@ class config(object):
> # setcpv triggers lazy instantiation of things like
> _use_manager.
> _eapi_cache.clear()
>
> + # Prepare the final value of INSTALL_MASK
> + install_mask = self["INSTALL_MASK"].split()
> + if 'nodoc' in self.features:
> + install_mask.append("/usr/share/doc")
> + if 'noinfo' in self.features:
> + install_mask.append("/usr/share/info")
> + if 'noman' in self.features:
> + install_mask.append("/usr/share/man")
> + self["INSTALL_MASK"] = ' '.join(install_mask)
> +
> def _grab_pkg_env(self, penv, container, protected_keys=None):
> if protected_keys is None:
> protected_keys = ()
>
I'm concerned that these values can be appended more than once, causing
the variable to contain duplicate values, because setcpv only calls
reset when the has_changed variable is True. So, we only want to call
this code when has_changed is True. In fact, this code can go
immediately after the reset call here:
if has_changed:
self.reset(keeping_pkg=1)
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH 1/3] portage.package.ebuild.config: Move FEATURES=no* handling there
On 06/12/2016 12:19 AM, Zac Medico wrote:
> On 05/21/2016 11:56 PM, Michał Górny wrote:
>> diff --git a/pym/portage/package/ebuild/config.py
>> b/pym/portage/package/ebuild/config.py
>> index 45b7d08..fcc7ce5 100644
>> --- a/pym/portage/package/ebuild/config.py
>> +++ b/pym/portage/package/ebuild/config.py
>> @@ -1773,6 +1773,16 @@ class config(object):
>> # setcpv triggers lazy instantiation of things like
>> _use_manager.
>> _eapi_cache.clear()
>>
>> +# Prepare the final value of INSTALL_MASK
>> +install_mask = self["INSTALL_MASK"].split()
>> +if 'nodoc' in self.features:
>> +install_mask.append("/usr/share/doc")
>> +if 'noinfo' in self.features:
>> +install_mask.append("/usr/share/info")
>> +if 'noman' in self.features:
>> +install_mask.append("/usr/share/man")
>> +self["INSTALL_MASK"] = ' '.join(install_mask)
>> +
>> def _grab_pkg_env(self, penv, container, protected_keys=None):
>> if protected_keys is None:
>> protected_keys = ()
>>
>
> I'm concerned that these values can be appended more than once, causing
> the variable to contain duplicate values, because setcpv only calls
> reset when the has_changed variable is True. So, we only want to call
> this code when has_changed is True. In fact, this code can go
> immediately after the reset call here:
>
> if has_changed:
> self.reset(keeping_pkg=1)
>
Actually, it's more tricky than that, because we need to account for
both global FEATURES settings and package.env FEATURES settings, and my
above statements do not account for the global settings.
Also need to consider the case where these features are enabled
globally, and then *disabled* via package.env!
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v2] Move INSTALL_MASK handling into merging
On 05/22/2016 01:21 AM, Michał Górny wrote:
> + def _is_install_masked(self, relative_path):
> + ret = False
> + for pattern in self.settings.install_mask:
> + # absolute path pattern
> + if pattern.startswith('/'):
> + # match either exact path or one of parent dirs
> + # the latter is done via matching pattern/*
> + if (fnmatch.fnmatch(relative_path, pattern[1:])
> + or
> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
> + ret = True
> + break
> + # filename
> + else:
> + if
> fnmatch.fnmatch(os.path.basename(relative_path), pattern):
> + ret = True
> + break
> + return ret
> +
This is a hot spot, so it should use a pre-compiled regular expression,
using | to join the results of fnmatch.translate(pattern) calls for each
pattern.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v2] Move INSTALL_MASK handling into merging
On 05/22/2016 01:21 AM, Michał Górny wrote: > Introduce a new logic for INSTALL_MASK handling in merging code, > replacing the old code that removed matching files and directories > from imagedir in bash. The new code actually ignores matching files > on-the-fly while testing for file collisions and merging files. > The files are still written to CONTENTS, and output using "###" zing > to indicate being masked, yet are not actually merged to the filesystem. Since collision-protect relies on existing files in its collision test, install-masked files are no longer going to trigger collisions. Then, since the install-masked files are still written to CONTENTS, it's possible for the unmerge of one package to unmerge colliding files that belong to another package! There are a number of ways to solve this problem. For example, we could have the unmerge code ignore any files in CONTENTS that match the INSTALL_MASK value that was used at merge time. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK
On 05/21/2016 11:56 PM, Michał Górny wrote:
> Allow INSTALL_MASK patterns to start with '-' to indicate that
> a specific match is to be excluded from being masked. In this case,
> the last matching pattern determines whether the file is actually
> filtered out or kept.
> ---
> pym/portage/dbapi/vartree.py | 10 ++
> 1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> index 8e5ac43..d02d850 100644
> --- a/pym/portage/dbapi/vartree.py
> +++ b/pym/portage/dbapi/vartree.py
> @@ -3690,19 +3690,21 @@ class dblink(object):
> def _is_install_masked(self, relative_path):
> ret = False
> for pattern in self.settings.install_mask:
> + # if pattern starts with -, possibly exclude this path
> + pat_res = not pattern.startswith('-')
> + if not pat_res:
> + pattern = pattern[1:]
> # absolute path pattern
> if pattern.startswith('/'):
> # match either exact path or one of parent dirs
> # the latter is done via matching pattern/*
> if (fnmatch.fnmatch(relative_path, pattern[1:])
> or
> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
> - ret = True
> - break
> + ret = pat_res
> # filename
> else:
> if
> fnmatch.fnmatch(os.path.basename(relative_path), pattern):
> - ret = True
> - break
> + ret = pat_res
> return ret
>
> def treewalk(self, srcroot, destroot, inforoot, myebuild, cleanup=0,
>
In order to implement this with pre-compiled patterns, we can use a
separate regular expression to represent all of the exclusions.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK
On 06/12/2016 02:31 AM, Michał Górny wrote:
> Dnia 12 czerwca 2016 11:20:36 CEST, Zac Medico
> napisał(a):
>> On 05/21/2016 11:56 PM, Michał Górny wrote:
>>> Allow INSTALL_MASK patterns to start with '-' to indicate that
>>> a specific match is to be excluded from being masked. In this case,
>>> the last matching pattern determines whether the file is actually
>>> filtered out or kept.
>>> ---
>>> pym/portage/dbapi/vartree.py | 10 ++
>>> 1 file changed, 6 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/pym/portage/dbapi/vartree.py
>> b/pym/portage/dbapi/vartree.py
>>> index 8e5ac43..d02d850 100644
>>> --- a/pym/portage/dbapi/vartree.py
>>> +++ b/pym/portage/dbapi/vartree.py
>>> @@ -3690,19 +3690,21 @@ class dblink(object):
>>> def _is_install_masked(self, relative_path):
>>> ret = False
>>> for pattern in self.settings.install_mask:
>>> + # if pattern starts with -, possibly exclude this path
>>> + pat_res = not pattern.startswith('-')
>>> + if not pat_res:
>>> + pattern = pattern[1:]
>>> # absolute path pattern
>>> if pattern.startswith('/'):
>>> # match either exact path or one of parent dirs
>>> # the latter is done via matching pattern/*
>>> if (fnmatch.fnmatch(relative_path, pattern[1:])
>>> or
>>> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
>>> - ret = True
>>> - break
>>> + ret = pat_res
>>> # filename
>>> else:
>>> if
>>> fnmatch.fnmatch(os.path.basename(relative_path), pattern):
>>> - ret = True
>>> - break
>>> + ret = pat_res
>>> return ret
>>>
>>> def treewalk(self, srcroot, destroot, inforoot, myebuild,
>> cleanup=0,
>>>
>>
>> In order to implement this with pre-compiled patterns, we can use a
>> separate regular expression to represent all of the exclusions.
>
> That won't work since exclusive and inclusive patterns can be stacked, and
> for this to work they are applied in order.
>
> E.g. /usr/foo -/usr/foo/bar /usr/foo/bar/baz
>
> Should remove all foo subdirectories except for bar, and also baz inside bar.
Hmm, I suppose there could be an optimized implementation to handle
INSTALL_MASK settings where there are no such ordering constraints.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v2] Move INSTALL_MASK handling into merging
On 06/12/2016 02:28 AM, Michał Górny wrote: > Dnia 12 czerwca 2016 11:10:55 CEST, Zac Medico > napisał(a): >> On 05/22/2016 01:21 AM, Michał Górny wrote: >>> Introduce a new logic for INSTALL_MASK handling in merging code, >>> replacing the old code that removed matching files and directories >>> from imagedir in bash. The new code actually ignores matching files >>> on-the-fly while testing for file collisions and merging files. >>> The files are still written to CONTENTS, and output using "###" zing >>> to indicate being masked, yet are not actually merged to the >> filesystem. >> >> Since collision-protect relies on existing files in its collision test, >> install-masked files are no longer going to trigger collisions. Then, >> since the install-masked files are still written to CONTENTS, it's >> possible for the unmerge of one package to unmerge colliding files that >> belong to another package! >> >> There are a number of ways to solve this problem. For example, we could >> have the unmerge code ignore any files in CONTENTS that match the >> INSTALL_MASK value that was used at merge time. > > Hmm, thinking about this more widely (i.e. actually thinking rather than > mimicking the old behavior), I think it would be better to actually use the > original file set for collision-protect. This will make it possible to detect > collisions that would otherwise be hidden via INSTALL_MASK. Even then, we have to carefully consider how the absence of installed files affects the collision test. It's safest for the unmerge code to be aware of the merge time INSTALL_MASK setting, and not try to unmerge the install-masked files. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK
On 06/12/2016 03:03 AM, Michał Górny wrote:
> Dnia 12 czerwca 2016 11:43:35 CEST, Zac Medico
> napisał(a):
>> On 06/12/2016 02:31 AM, Michał Górny wrote:
>>> Dnia 12 czerwca 2016 11:20:36 CEST, Zac Medico
>> napisał(a):
>>>> On 05/21/2016 11:56 PM, Michał Górny wrote:
>>>>> Allow INSTALL_MASK patterns to start with '-' to indicate that
>>>>> a specific match is to be excluded from being masked. In this case,
>>>>> the last matching pattern determines whether the file is actually
>>>>> filtered out or kept.
>>>>> ---
>>>>> pym/portage/dbapi/vartree.py | 10 ++
>>>>> 1 file changed, 6 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/pym/portage/dbapi/vartree.py
>>>> b/pym/portage/dbapi/vartree.py
>>>>> index 8e5ac43..d02d850 100644
>>>>> --- a/pym/portage/dbapi/vartree.py
>>>>> +++ b/pym/portage/dbapi/vartree.py
>>>>> @@ -3690,19 +3690,21 @@ class dblink(object):
>>>>> def _is_install_masked(self, relative_path):
>>>>> ret = False
>>>>> for pattern in self.settings.install_mask:
>>>>> + # if pattern starts with -, possibly exclude this path
>>>>> + pat_res = not pattern.startswith('-')
>>>>> + if not pat_res:
>>>>> + pattern = pattern[1:]
>>>>> # absolute path pattern
>>>>> if pattern.startswith('/'):
>>>>> # match either exact path or one of parent dirs
>>>>> # the latter is done via matching pattern/*
>>>>> if (fnmatch.fnmatch(relative_path, pattern[1:])
>>>>> or
>>>>> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
>>>>> - ret = True
>>>>> - break
>>>>> + ret = pat_res
>>>>> # filename
>>>>> else:
>>>>> if
>>>>> fnmatch.fnmatch(os.path.basename(relative_path), pattern):
>>>>> - ret = True
>>>>> - break
>>>>> + ret = pat_res
>>>>> return ret
>>>>>
>>>>> def treewalk(self, srcroot, destroot, inforoot, myebuild,
>>>> cleanup=0,
>>>>>
>>>>
>>>> In order to implement this with pre-compiled patterns, we can use a
>>>> separate regular expression to represent all of the exclusions.
>>>
>>> That won't work since exclusive and inclusive patterns can be
>> stacked, and for this to work they are applied in order.
>>>
>>> E.g. /usr/foo -/usr/foo/bar /usr/foo/bar/baz
>>>
>>> Should remove all foo subdirectories except for bar, and also baz
>> inside bar.
>>
>> Hmm, I suppose there could be an optimized implementation to handle
>> INSTALL_MASK settings where there are no such ordering constraints.
>
> Is this really worth the effort? I'm using the patch for a while and I/O's
> the bottleneck, not CPU. If you really insist on optimizing this, i guess we
> could precompile all patterns separately.
Since it's not going to make any difference when INSTALL_MASK is not
used, I guess it's fine to leave it unoptimized for now.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v2] Move INSTALL_MASK handling into merging
On 06/12/2016 03:05 AM, Michał Górny wrote: > Dnia 12 czerwca 2016 11:49:26 CEST, Zac Medico > napisał(a): >> On 06/12/2016 02:28 AM, Michał Górny wrote: >>> Dnia 12 czerwca 2016 11:10:55 CEST, Zac Medico >> napisał(a): >>>> On 05/22/2016 01:21 AM, Michał Górny wrote: >>>>> Introduce a new logic for INSTALL_MASK handling in merging code, >>>>> replacing the old code that removed matching files and directories >>>>> from imagedir in bash. The new code actually ignores matching files >>>>> on-the-fly while testing for file collisions and merging files. >>>>> The files are still written to CONTENTS, and output using "###" >> zing >>>>> to indicate being masked, yet are not actually merged to the >>>> filesystem. >>>> >>>> Since collision-protect relies on existing files in its collision >> test, >>>> install-masked files are no longer going to trigger collisions. >> Then, >>>> since the install-masked files are still written to CONTENTS, it's >>>> possible for the unmerge of one package to unmerge colliding files >> that >>>> belong to another package! >>>> >>>> There are a number of ways to solve this problem. For example, we >> could >>>> have the unmerge code ignore any files in CONTENTS that match the >>>> INSTALL_MASK value that was used at merge time. >>> >>> Hmm, thinking about this more widely (i.e. actually thinking rather >> than mimicking the old behavior), I think it would be better to >> actually use the original file set for collision-protect. This will >> make it possible to detect collisions that would otherwise be hidden >> via INSTALL_MASK. >> >> Even then, we have to carefully consider how the absence of installed >> files affects the collision test. It's safest for the unmerge code to >> be >> aware of the merge time INSTALL_MASK setting, and not try to unmerge >> the >> install-masked files. > > But then it wouldn't unmerge the newly masked files as well. I'm suggesting to save the INSTALL_MASK setting from merge time, so any new INSTALL_MASK settings since then are irrelevant. > Getting this right will require a lot of effort, and we're less likely to > screw something up if we keep it simple. You can use exactly the same pattern matching code for merge and unmerge. It's worthwhile, given the say that collision-protect relies on file existence in the collision test. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH v2] Move INSTALL_MASK handling into merging
On 05/22/2016 01:21 AM, Michał Górny wrote:
> diff --git a/pym/portage/package/ebuild/config.py
> b/pym/portage/package/ebuild/config.py
> index fcc7ce5..9d13703 100644
> --- a/pym/portage/package/ebuild/config.py
> +++ b/pym/portage/package/ebuild/config.py
> @@ -1774,14 +1774,14 @@ class config(object):
> _eapi_cache.clear()
>
> # Prepare the final value of INSTALL_MASK
> - install_mask = self["INSTALL_MASK"].split()
> + install_mask = self.get("INSTALL_MASK", "").split()
> if 'nodoc' in self.features:
> install_mask.append("/usr/share/doc")
> if 'noinfo' in self.features:
> install_mask.append("/usr/share/info")
> if 'noman' in self.features:
> install_mask.append("/usr/share/man")
> - self["INSTALL_MASK"] = ' '.join(install_mask)
> + self.install_mask = install_mask
>
> def _grab_pkg_env(self, penv, container, protected_keys=None):
> if protected_keys is None:
>
The config.reset method should reset self.install_mask to match the
global INSTALL_MASK setting.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v2] Move INSTALL_MASK handling into merging
On 06/12/2016 01:29 PM, Zac Medico wrote:
> On 05/22/2016 01:21 AM, Michał Górny wrote:
>> diff --git a/pym/portage/package/ebuild/config.py
>> b/pym/portage/package/ebuild/config.py
>> index fcc7ce5..9d13703 100644
>> --- a/pym/portage/package/ebuild/config.py
>> +++ b/pym/portage/package/ebuild/config.py
>> @@ -1774,14 +1774,14 @@ class config(object):
>> _eapi_cache.clear()
>>
>> # Prepare the final value of INSTALL_MASK
>> -install_mask = self["INSTALL_MASK"].split()
>> +install_mask = self.get("INSTALL_MASK", "").split()
>> if 'nodoc' in self.features:
>> install_mask.append("/usr/share/doc")
>> if 'noinfo' in self.features:
>> install_mask.append("/usr/share/info")
>> if 'noman' in self.features:
>> install_mask.append("/usr/share/man")
>> -self["INSTALL_MASK"] = ' '.join(install_mask)
>> +self.install_mask = install_mask
>>
>> def _grab_pkg_env(self, penv, container, protected_keys=None):
>> if protected_keys is None:
>>
>
> The config.reset method should reset self.install_mask to match the
> global INSTALL_MASK setting.
>
For the benefit of those who may not be as familiar with the config
class, here is some of the related irc discussion:
mgorny: config.setcpv put's the config into per-package state,
and config.reset reverts it to global state
i see
so how does that affect me? ;-D
because yout set self.install_mask in config.setcpv
so you also have to handle global setting of that attribute
every attribute has global and per-package state
for example the usemask attribute is similar
initially it has global use.mask settings
hmm, so you mean i can't rely on both being combined properly
in .setcpv() ?
hmm, but wait, do i need to care about global state at all?
yes, that's what config.reset is for
it transitions back to global state
but is there any case when i need the global state?
i don't think the mask is used without packages
if that's the case, the you can set it to None whien the
config is in global mode
careful though, setcpv returns early in some cases
and you set self.install_mask at the very end of that method
so it's not guaranteed to execute
so you have end up with self.install_mask having the global
setting after a setcpv call
which is fine if there's no package.env setting for that package
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v2] Move INSTALL_MASK handling into merging
On 06/12/2016 01:43 PM, Zac Medico wrote:
> On 06/12/2016 01:29 PM, Zac Medico wrote:
>> On 05/22/2016 01:21 AM, Michał Górny wrote:
>>> diff --git a/pym/portage/package/ebuild/config.py
>>> b/pym/portage/package/ebuild/config.py
>>> index fcc7ce5..9d13703 100644
>>> --- a/pym/portage/package/ebuild/config.py
>>> +++ b/pym/portage/package/ebuild/config.py
>>> @@ -1774,14 +1774,14 @@ class config(object):
>>> _eapi_cache.clear()
>>>
>>> # Prepare the final value of INSTALL_MASK
>>> - install_mask = self["INSTALL_MASK"].split()
>>> + install_mask = self.get("INSTALL_MASK", "").split()
>>> if 'nodoc' in self.features:
>>> install_mask.append("/usr/share/doc")
>>> if 'noinfo' in self.features:
>>> install_mask.append("/usr/share/info")
>>> if 'noman' in self.features:
>>> install_mask.append("/usr/share/man")
>>> - self["INSTALL_MASK"] = ' '.join(install_mask)
>>> + self.install_mask = install_mask
>>>
>>> def _grab_pkg_env(self, penv, container, protected_keys=None):
>>> if protected_keys is None:
>>>
>>
>> The config.reset method should reset self.install_mask to match the
>> global INSTALL_MASK setting.
>>
>
> For the benefit of those who may not be as familiar with the config
> class, here is some of the related irc discussion:
>
> mgorny: config.setcpv put's the config into per-package state,
> and config.reset reverts it to global state
> i see
> so how does that affect me? ;-D
> because yout set self.install_mask in config.setcpv
> so you also have to handle global setting of that attribute
> every attribute has global and per-package state
> for example the usemask attribute is similar
> initially it has global use.mask settings
> hmm, so you mean i can't rely on both being combined properly
> in .setcpv() ?
> hmm, but wait, do i need to care about global state at all?
> yes, that's what config.reset is for
> it transitions back to global state
> but is there any case when i need the global state?
> i don't think the mask is used without packages
> if that's the case, the you can set it to None whien the
> config is in global mode
> careful though, setcpv returns early in some cases
> and you set self.install_mask at the very end of that method
> so it's not guaranteed to execute
> so you have end up with self.install_mask having the global
> setting after a setcpv call
> which is fine if there's no package.env setting for that package
>
mgorny: also, you should copy self.install_mask in the clone
part of the config constructor
--
Thanks,
Zac
[gentoo-portage-dev] Re: [PATCH] repoman: typo in docstring s/dependancy/dependency/
On 06/18/2016 10:01 AM, Sergei Trofimovich wrote:
> From: Sergei Trofimovich
>
> Signed-off-by: Sergei Trofimovich
> ---
> repoman/pym/repoman/modules/scan/depend/profile.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/repoman/pym/repoman/modules/scan/depend/profile.py
> b/repoman/pym/repoman/modules/scan/depend/profile.py
> index a714a93..cf3d9a8 100644
> --- a/repoman/pym/repoman/modules/scan/depend/profile.py
> +++ b/repoman/pym/repoman/modules/scan/depend/profile.py
> @@ -51,7 +51,7 @@ class ProfileDependsChecks(ScanBase):
> self.repo_metadata = kwargs.get('repo_metadata')
>
> def check(self, **kwargs):
> - '''Perform profile dependant dependancy checks
> + '''Perform profile dependant dependency checks
>
> @param arches:
> @param pkg: Package in which we check (object).
>
Thanks, applied:
https://gitweb.gentoo.org/proj/portage.git/commit/?id=279b51a20dc180fb10252e5250236e04a248ea67
--
Thanks,
Zac
[gentoo-portage-dev] Re: [PATCH v2] repoman/modules/.../profile.py: typo in docstring s/dependancy/dependency/
On 06/18/2016 12:27 PM, Sergei Trofimovich wrote:
> From: Sergei Trofimovich
>
> Signed-off-by: Sergei Trofimovich
> ---
> V2: updated subject
> repoman/pym/repoman/modules/scan/depend/profile.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/repoman/pym/repoman/modules/scan/depend/profile.py
> b/repoman/pym/repoman/modules/scan/depend/profile.py
> index a714a93..cf3d9a8 100644
> --- a/repoman/pym/repoman/modules/scan/depend/profile.py
> +++ b/repoman/pym/repoman/modules/scan/depend/profile.py
> @@ -51,7 +51,7 @@ class ProfileDependsChecks(ScanBase):
> self.repo_metadata = kwargs.get('repo_metadata')
>
> def check(self, **kwargs):
> - '''Perform profile dependant dependancy checks
> + '''Perform profile dependant dependency checks
>
> @param arches:
> @param pkg: Package in which we check (object).
>
I've already pushed the first patch:
https://gitweb.gentoo.org/proj/portage.git/commit/?id=279b51a20dc180fb10252e5250236e04a248ea67
--
Thanks,
Zac
[gentoo-portage-dev] Re: [PATCH v2] repoman: new QA error: slot operator under '||' alternative
On 06/18/2016 12:27 PM, Sergei Trofimovich wrote:
> + if in_any_of and atom.slot_operator:
> + qatracker.add_error("dependency.badslotop",
> relative_path +
> + ": %s: '%s' uses ':=' slot operator under '||'
> dep clause." %
> + (mytype, atom))
It should be like this:
if in_any_of and atom.slot_operator == '=':
Because the case of atom.slot_operator == '*' is not forbidden.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH] repoman: new QA error: slot operator under '||' alternative
On 06/18/2016 01:38 PM, Brian Dolbec wrote: > Zac, what do you think? I don't see any problems except for the need to do atom.slot_operator == '=', as mentioned in my previous reply. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] unmerge-backup: check BUILD_TIME of existing binary package (bug 586410)
Fix the dblink._quickpkg_dblink method to search for a binary package
having identical BUILD_TIME to the installed instance.
X-Gentoo-Bug: 586410
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=586410
---
pym/portage/dbapi/vartree.py | 15 +++
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
index bfbe356..28ae584 100644
--- a/pym/portage/dbapi/vartree.py
+++ b/pym/portage/dbapi/vartree.py
@@ -5231,12 +5231,19 @@ class dblink(object):
def _quickpkg_dblink(self, backup_dblink, background, logfile):
+ build_time = backup_dblink.getfile('BUILD_TIME')
+ try:
+ build_time = long(build_time.strip())
+ except ValueError:
+ build_time = 0
+
trees = QueryCommand.get_db()[self.settings["EROOT"]]
bintree = trees["bintree"]
- binpkg_path = bintree.getname(backup_dblink.mycpv)
- if os.path.exists(binpkg_path) and \
- catsplit(backup_dblink.mycpv)[1] not in
bintree.invalids:
- return os.EX_OK
+
+ for binpkg in reversed(
+ bintree.dbapi.match('={}'.format(backup_dblink.mycpv))):
+ if binpkg.build_time == build_time:
+ return os.EX_OK
self.lockdb()
try:
--
2.7.4
[gentoo-portage-dev] [PATCH] repoman: fix KeyError during manifest generation (bug 585388)
Fix Scanner not to override portdbapi._aux_cache_keys when generating
manifests, since spawn_nofetch requires additional keys.
Fixes: 87e32d4863f3 ("repoman: fix KeyError...during manifest generation (bug
569942)")
X-Gentoo-Bug: 585388
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=585388
---
repoman/pym/repoman/modules/commit/manifest.py | 91 +-
repoman/pym/repoman/scanner.py | 12 +++-
2 files changed, 53 insertions(+), 50 deletions(-)
diff --git a/repoman/pym/repoman/modules/commit/manifest.py
b/repoman/pym/repoman/modules/commit/manifest.py
index 475dae4..b338a5b 100644
--- a/repoman/pym/repoman/modules/commit/manifest.py
+++ b/repoman/pym/repoman/modules/commit/manifest.py
@@ -33,55 +33,52 @@ class Manifest(object):
@returns: dictionary
'''
self.generated_manifest = False
- self.digest_only = self.options.mode != 'manifest-check' \
- and self.options.digest == 'y'
- if self.options.mode in ("manifest", 'commit', 'fix') or
self.digest_only:
- failed = False
- self.auto_assumed = set()
- fetchlist_dict = portage.FetchlistDict(
- checkdir, self.repoman_settings, self.portdb)
- if self.options.mode == 'manifest' and
self.options.force:
- self._discard_dist_digests(checkdir,
fetchlist_dict)
- self.repoman_settings["O"] = checkdir
- try:
- self.generated_manifest = digestgen(
- mysettings=self.repoman_settings,
myportdb=self.portdb)
- except portage.exception.PermissionDenied as e:
- self.generated_manifest = False
- writemsg_level(
- "!!! Permission denied: '%s'\n" % (e,),
- level=logging.ERROR, noiselevel=-1)
+ failed = False
+ self.auto_assumed = set()
+ fetchlist_dict = portage.FetchlistDict(
+ checkdir, self.repoman_settings, self.portdb)
+ if self.options.mode == 'manifest' and self.options.force:
+ self._discard_dist_digests(checkdir, fetchlist_dict)
+ self.repoman_settings["O"] = checkdir
+ try:
+ self.generated_manifest = digestgen(
+ mysettings=self.repoman_settings,
myportdb=self.portdb)
+ except portage.exception.PermissionDenied as e:
+ self.generated_manifest = False
+ writemsg_level(
+ "!!! Permission denied: '%s'\n" % (e,),
+ level=logging.ERROR, noiselevel=-1)
- if not self.generated_manifest:
- writemsg_level(
- "Unable to generate manifest.",
- level=logging.ERROR, noiselevel=-1)
- failed = True
+ if not self.generated_manifest:
+ writemsg_level(
+ "Unable to generate manifest.",
+ level=logging.ERROR, noiselevel=-1)
+ failed = True
- if self.options.mode == "manifest":
- if not failed and self.options.force and
self.auto_assumed and \
- 'assume-digests' in
self.repoman_settings.features:
- # Show which digests were assumed
despite the --force option
- # being given. This output will already
have been shown by
- # digestgen() if assume-digests is not
enabled, so only show
- # it here if assume-digests is enabled.
- pkgs = list(fetchlist_dict)
- pkgs.sort()
- portage.writemsg_stdout(
- " digest.assumed %s" %
- portage.output.colorize(
- "WARN",
str(len(self.auto_assumed)).rjust(18)) + "\n")
- for cpv in pkgs:
- fetchmap = fetchlist_dict[cpv]
- pf = portage.catsplit(cpv)[1]
- for distfile in
sorted(fetchmap):
- if distfile in
self
[gentoo-portage-dev] [PATCH 2/2] depgraph: fix missed llvm update (bug 584626)
Fix check_reverse_dependencies to ignore dependencies of parent packages for which updates are desirable, and add required _want_update_pkg support for DependencyArg parents. This solves a missed llvm update by ignoring a reverse dependency from the installed instance of clang, since an update to a newer version of clang is desirable. In order to cope with this change, there are 2 existing unit tests that require larger --backtrack settings in order to succeed. X-Gentoo-Bug: 584626 X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=584626 --- pym/_emerge/depgraph.py| 40 -- .../soname/test_slot_conflict_reinstall.py | 1 + .../tests/resolver/test_slot_conflict_rebuild.py | 2 +- .../resolver/test_slot_operator_reverse_deps.py| 6 +++- 4 files changed, 44 insertions(+), 5 deletions(-) diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py index f78f08d..ef3cc5d 100644 --- a/pym/_emerge/depgraph.py +++ b/pym/_emerge/depgraph.py @@ -1822,6 +1822,22 @@ class depgraph(object): # necessarily relevant. continue + if (not self._frozen_config.excluded_pkgs. + findAtomForPackage(parent, + modified_use=self._pkg_use_enabled(parent)) and + self._upgrade_available(parent)): + want_update = False + for grandparent in self._dynamic_config.digraph.\ + parent_nodes(parent): + if self._want_update_pkg(grandparent, parent): + want_update = True + break + + if want_update: + # This parent may be irrelevant, since an + # update is available (see bug 584626). + continue + atom_set = InternalPackageSet(initial_atoms=(atom,), allow_repo=True) if not atom_set.findAtomForPackage(candidate_pkg, @@ -2113,6 +2129,19 @@ class depgraph(object): self._dynamic_config._need_restart = True + def _upgrade_available(self, pkg): + """ + Detect cases where an upgrade of the given package is available + within the same slot. + """ + available_pkg = None + for available_pkg in self._iter_similar_available(pkg, + pkg.slot_atom): + if available_pkg >= pkg: + return True + + return False + def _downgrade_probe(self, pkg): """ Detect cases where a downgrade of the given package is considered @@ -5382,9 +5411,14 @@ class depgraph(object): # should have been masked before it was selected raise - depth = parent.depth or 0 - if isinstance(depth, int): - depth += 1 + if isinstance(parent, Package): + depth = parent.depth or 0 + if isinstance(depth, int): + depth += 1 + elif parent.reset_depth: + depth = 0 + else: + depth = self._UNREACHABLE_DEPTH if arg_atoms: for arg, atom in arg_atoms: diff --git a/pym/portage/tests/resolver/soname/test_slot_conflict_reinstall.py b/pym/portage/tests/resolver/soname/test_slot_conflict_reinstall.py index 40e6995..f474761 100644 --- a/pym/portage/tests/resolver/soname/test_slot_conflict_reinstall.py +++ b/pym/portage/tests/resolver/soname/test_slot_conflict_reinstall.py @@ -89,6 +89,7 @@ class SonameSlotConflictReinstallTestCase(TestCase): "--ignore-soname-deps": "n", "--update": True, "--usepkgonly": True, + "--backtrack": 10, }, success = True, mergelist = [ diff --git a/pym/portage/tests/resolver/test_slot_conflict_rebuild.py b/pym/portage/tests/resolver/test_slot_conflict_rebuild.py index b39eaf0..2dfa79c 100644 --- a/pym/portage/tests/
[gentoo-portage-dev] [PATCH 1/2] Add a unit test which reproduces bug 584626
X-Gentoo-Bug: 584626
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=584626
---
.../resolver/test_slot_operator_reverse_deps.py| 109 +
1 file changed, 109 insertions(+)
create mode 100644
pym/portage/tests/resolver/test_slot_operator_reverse_deps.py
diff --git a/pym/portage/tests/resolver/test_slot_operator_reverse_deps.py
b/pym/portage/tests/resolver/test_slot_operator_reverse_deps.py
new file mode 100644
index 000..72879f8
--- /dev/null
+++ b/pym/portage/tests/resolver/test_slot_operator_reverse_deps.py
@@ -0,0 +1,109 @@
+# Copyright 2016 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+from portage.tests import TestCase
+from portage.tests.resolver.ResolverPlayground import (
+ ResolverPlayground,
+ ResolverPlaygroundTestCase,
+)
+
+class SlotOperatorReverseDepsTestCase(TestCase):
+
+ def testSlotOperatorReverseDeps(self):
+
+ ebuilds = {
+
+ "media-libs/mesa-11.2.2" : {
+ "EAPI": "6",
+ "SLOT": "0",
+ "RDEPEND": ">=sys-devel/llvm-3.6.0:="
+ },
+
+ "sys-devel/clang-3.7.1-r100" : {
+ "EAPI": "6",
+ "SLOT": "0/3.7",
+ "RDEPEND": "~sys-devel/llvm-3.7.1"
+ },
+
+ "sys-devel/clang-3.8.0-r100" : {
+ "EAPI": "6",
+ "SLOT": "0/3.8",
+ "RDEPEND": "~sys-devel/llvm-3.8.0"
+ },
+
+ "sys-devel/llvm-3.7.1-r2" : {
+ "EAPI": "6",
+ "SLOT": "0/3.7.1",
+ "PDEPEND": "=sys-devel/clang-3.7.1-r100"
+ },
+
+ "sys-devel/llvm-3.8.0-r2" : {
+ "EAPI": "6",
+ "SLOT": "0/3.8.0",
+ "PDEPEND": "=sys-devel/clang-3.8.0-r100"
+ },
+
+ }
+
+ installed = {
+
+ "media-libs/mesa-11.2.2" : {
+ "EAPI": "6",
+ "SLOT": "0",
+ "RDEPEND": ">=sys-devel/llvm-3.6.0:0/3.7.1="
+ },
+
+ "sys-devel/clang-3.7.1-r100" : {
+ "EAPI": "6",
+ "SLOT": "0/3.7",
+ "RDEPEND": "~sys-devel/llvm-3.7.1"
+ },
+
+ "sys-devel/llvm-3.7.1-r2" : {
+ "EAPI": "6",
+ "SLOT": "0/3.7.1",
+ "PDEPEND": "=sys-devel/clang-3.7.1-r100"
+ },
+
+ }
+
+ world = ["media-libs/mesa"]
+
+ test_cases = (
+
+ # Test bug #584626, where an llvm update is missed due
to
+ # the check_reverse_dependencies function seeing that
+ # updating llvm will break a dependency of the installed
+ # version of clang (though a clang update is available).
+ ResolverPlaygroundTestCase(
+ ["@world"],
+ options = {"--update": True, "--deep": True},
+ success = True,
+ mergelist = [],
+ ),
+
+ ResolverPlaygroundTestCase(
+ ["@world"],
+ options = {
+ "--update": True,
+ "--deep": True,
+ "--ignore-built-slot-operator-deps":
"y",
+ },
+ success = True,
+ mergelist = [
+ 'sys-devel/llvm-3.8.0-r2',
+ 'sys-devel/clang-3.8.0-r100',
+ ],
+ ),
+
+ )
+
+ playground = ResolverPlayground(ebuilds=ebuilds,
+ installed=installed, world=world, debug=False)
+ try:
+ for test_case in test_cases:
+ playground.run_TestCase(test_case)
+ self.assertEqual(test_case.test_success, True,
+ test_case.fail_msg)
+ finally:
+ playground.cleanup()
--
2.7.4
[gentoo-portage-dev] Re: [PATCH 2/2] depgraph: fix missed llvm update (bug 584626)
On 06/23/2016 12:38 AM, Zac Medico wrote: > @@ -2113,6 +2129,19 @@ class depgraph(object): > > self._dynamic_config._need_restart = True > > + def _upgrade_available(self, pkg): > + """ > + Detect cases where an upgrade of the given package is available > + within the same slot. > + """ > + available_pkg = None > + for available_pkg in self._iter_similar_available(pkg, > + pkg.slot_atom): > + if available_pkg >= pkg: > + return True > + > + return False > + I've fixed this method to use the > operator instead of the >= operator. The latest version is in my branch here: https://github.com/zmedico/portage/tree/bug_584626 I've also opened this pull request so that travis runs the tests: https://github.com/gentoo/portage/pull/29 -- Thanks, Zac
[gentoo-portage-dev] Re: [PATCH 2/2] depgraph: fix missed llvm update (bug 584626)
On 06/23/2016 12:38 AM, Zac Medico wrote: > diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py > index f78f08d..ef3cc5d 100644 > --- a/pym/_emerge/depgraph.py > +++ b/pym/_emerge/depgraph.py > @@ -1822,6 +1822,22 @@ class depgraph(object): > # necessarily relevant. > continue > > + if (not > self._frozen_config.excluded_pkgs. > + findAtomForPackage(parent, > + > modified_use=self._pkg_use_enabled(parent)) and > + > self._upgrade_available(parent)): > + want_update = False > + for grandparent in > self._dynamic_config.digraph.\ > + parent_nodes(parent): > + if > self._want_update_pkg(grandparent, parent): > + want_update = > True > + break > + > + if want_update: > + # This parent may be > irrelevant, since an > + # update is available > (see bug 584626). > + continue > + I've simplified this as follows: if (not self._too_deep(parent.depth) and not self._frozen_config.excluded_pkgs. findAtomForPackage(parent, modified_use=self._pkg_use_enabled(parent)) and self._upgrade_available(parent)): # This parent may be irrelevant, since an # update is available (see bug 584626). continue -- Thanks, Zac
[gentoo-portage-dev] [PATCH 2/2 v2] depgraph: fix missed llvm update (bug 584626)
Fix check_reverse_dependencies to ignore dependencies of parent packages
for which updates are desirable. This solves a missed llvm update by
ignoring a reverse dependency from the installed instance of clang,
since an update to a newer version of clang is desirable.
In order to cope with this change, there are 2 existing unit tests
that require larger --backtrack settings in order to succeed.
X-Gentoo-Bug: 584626
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=584626
---
[PATCH 2/2 v2]
* use > operator instead of >= in _upgrade_available
* simplify check_reverse_dependencies changes, using _too_deep
pym/_emerge/depgraph.py | 21 +
.../resolver/soname/test_slot_conflict_reinstall.py | 1 +
.../tests/resolver/test_slot_conflict_rebuild.py| 2 +-
.../resolver/test_slot_operator_reverse_deps.py | 6 +-
4 files changed, 28 insertions(+), 2 deletions(-)
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index f78f08d..c1c37b4 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -1822,6 +1822,15 @@ class depgraph(object):
# necessarily relevant.
continue
+ if (not self._too_deep(parent.depth) and
+ not
self._frozen_config.excluded_pkgs.
+ findAtomForPackage(parent,
+
modified_use=self._pkg_use_enabled(parent)) and
+
self._upgrade_available(parent)):
+ # This parent may be
irrelevant, since an
+ # update is available (see bug
584626).
+ continue
+
atom_set =
InternalPackageSet(initial_atoms=(atom,),
allow_repo=True)
if not
atom_set.findAtomForPackage(candidate_pkg,
@@ -2113,6 +2122,18 @@ class depgraph(object):
self._dynamic_config._need_restart = True
+ def _upgrade_available(self, pkg):
+ """
+ Detect cases where an upgrade of the given package is available
+ within the same slot.
+ """
+ for available_pkg in self._iter_similar_available(pkg,
+ pkg.slot_atom):
+ if available_pkg > pkg:
+ return True
+
+ return False
+
def _downgrade_probe(self, pkg):
"""
Detect cases where a downgrade of the given package is
considered
diff --git a/pym/portage/tests/resolver/soname/test_slot_conflict_reinstall.py
b/pym/portage/tests/resolver/soname/test_slot_conflict_reinstall.py
index 40e6995..f474761 100644
--- a/pym/portage/tests/resolver/soname/test_slot_conflict_reinstall.py
+++ b/pym/portage/tests/resolver/soname/test_slot_conflict_reinstall.py
@@ -89,6 +89,7 @@ class SonameSlotConflictReinstallTestCase(TestCase):
"--ignore-soname-deps": "n",
"--update": True,
"--usepkgonly": True,
+ "--backtrack": 10,
},
success = True,
mergelist = [
diff --git a/pym/portage/tests/resolver/test_slot_conflict_rebuild.py
b/pym/portage/tests/resolver/test_slot_conflict_rebuild.py
index b39eaf0..2dfa79c 100644
--- a/pym/portage/tests/resolver/test_slot_conflict_rebuild.py
+++ b/pym/portage/tests/resolver/test_slot_conflict_rebuild.py
@@ -91,7 +91,7 @@ class SlotConflictRebuildTestCase(TestCase):
# upgrade and we don't want to trigger unnecessary
rebuilds.
ResolverPlaygroundTestCase(
["@world"],
- options = {"--update": True, "--deep": True},
+ options = {"--update": True, "--deep": True,
"--backtrack": 4},
success = True,
mergelist = ["app-misc/D-2", "app-misc/E-0"]),
diff --git a/pym/portage/tests/resolver/test_slot_operator_reverse_deps.py
b/pym/portage/tests/resolver/test_slot_operator_reverse_deps.py
index 72879f8..ce614a4 100644
--- a/pym/portage/tests/resolver/test_slot_operator_reverse_deps.py
+++ b/pym/portage/tests/resolver/test_slot_operator_reverse_deps.py
@@ -79,7 +79,11 @@ class SlotOperatorReverseDepsTestCase(TestCase):
["@world"],
options = {"--update": True, "--deep": True},
success = True,
-
Re: [gentoo-portage-dev] [PATCH] ebuild: Extend helper-in-global-scope ban to all EAPIs
On 05/20/2016 03:27 PM, Michał Górny wrote: > On Thu, 5 May 2016 23:10:43 +0200 > Michał Górny wrote: > >> Make helper calls in global scope fatal in all supported EAPIs since >> this is the behavior required by PMS and all major offenders are fixed >> already. >> --- >> bin/eapi.sh | 4 >> bin/ebuild.sh | 30 ++ >> 2 files changed, 6 insertions(+), 28 deletions(-) > > Merged now. > Seems we might need to revert this patch: https://forums.gentoo.org/viewtopic-t-1047248.html -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] repoman: Make LIVEVCS.* checks fatal
On 06/28/2016 11:30 AM, Michał Górny wrote: > Make LIVEVCS.* checks fatal to prevent people from committing ebuilds > using live eclasses instead of package.masking them afterwards by QA. > --- > repoman/pym/repoman/qa_data.py | 2 -- > 1 file changed, 2 deletions(-) > > diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py > index 48ab389..738368e 100644 > --- a/repoman/pym/repoman/qa_data.py > +++ b/repoman/pym/repoman/qa_data.py > @@ -270,8 +270,6 @@ qawarnings = set(( > "repo.eapi.deprecated", > "usage.obsolete", > "upstream.workaround", > - "LIVEVCS.stable", > - "LIVEVCS.unmasked", > "IUSE.rubydeprecated", > )) > > Looks good. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH] Add emerge --autounmask-continue option (bug 582624)
This option will cause emerge to automatically apply autounmask changes
to configuration files, and continue to execute the specified command.
If the dependency calculation is not entirely successful, then emerge
will simply abort without modifying any configuration files.
This sort of behavior can be very useful in a continuous integration
setting, where the emerge invocation might be inside of a container that
is later discarded (so there is no threat of negative consequences).
It's also safe for general use, when combined with the --ask option.
X-Gentoo-Bug: 582624
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=582624
---
man/emerge.1| 14 +-
pym/_emerge/actions.py | 28 ++--
pym/_emerge/depgraph.py | 28
pym/_emerge/main.py | 9 +
pym/portage/tests/emerge/test_simple.py | 5 +
5 files changed, 73 insertions(+), 11 deletions(-)
diff --git a/man/emerge.1 b/man/emerge.1
index bfa2f73..40be14f 100644
--- a/man/emerge.1
+++ b/man/emerge.1
@@ -1,4 +1,4 @@
-.TH "EMERGE" "1" "Feb 2016" "Portage VERSION" "Portage"
+.TH "EMERGE" "1" "Jul 2016" "Portage VERSION" "Portage"
.SH "NAME"
emerge \- Command\-line interface to the Portage system
.SH "SYNOPSIS"
@@ -361,6 +361,18 @@ the specified configuration file(s), or enable the
\fBEMERGE_DEFAULT_OPTS\fR variable may be used to
disable this option by default in \fBmake.conf\fR(5).
.TP
+.BR "\-\-autounmask\-continue [ y | n ]"
+Automatically apply autounmask changes to configuration
+files, and continue to execute the specified command. If
+the dependency calculation is not entirely successful, then
+emerge will simply abort without modifying any configuration
+files.
+\fBWARNING:\fR
+This option is intended to be used only with great caution,
+since it is possible for it to make nonsensical configuration
+changes which may lead to system breakage. Therefore, it is
+advisable to use \fB\-\-ask\fR together with this option.
+.TP
.BR "\-\-autounmask\-only [ y | n ]"
Instead of doing any package building, just unmask
packages and generate package.use settings as necessary
diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
index 2ca7902..ff22687 100644
--- a/pym/_emerge/actions.py
+++ b/pym/_emerge/actions.py
@@ -96,8 +96,22 @@ if sys.hexversion >= 0x300:
else:
_unicode = unicode
-def action_build(settings, trees, mtimedb,
- myopts, myaction, myfiles, spinner):
+def action_build(emerge_config, trees=DeprecationWarning,
+ mtimedb=DeprecationWarning, myopts=DeprecationWarning,
+ myaction=DeprecationWarning, myfiles=DeprecationWarning, spinner=None):
+
+ if not isinstance(emerge_config, _emerge_config):
+ warnings.warn("_emerge.actions.action_build() now expects "
+ "an _emerge_config instance as the first parameter",
+ DeprecationWarning, stacklevel=2)
+ emerge_config = load_emerge_config(
+ action=myaction, args=myfiles, trees=trees, opts=myopts)
+ adjust_configs(emerge_config.opts, emerge_config.trees)
+
+ settings, trees, mtimedb = emerge_config
+ myopts = emerge_config.opts
+ myaction = emerge_config.action
+ myfiles = emerge_config.args
if '--usepkgonly' not in myopts:
old_tree_timestamp_warn(settings['PORTDIR'], settings)
@@ -327,6 +341,11 @@ def action_build(settings, trees, mtimedb,
display_missing_pkg_set(root_config, e.value)
return 1
+ if success and mydepgraph.need_config_reload():
+ load_emerge_config(emerge_config=emerge_config)
+ adjust_configs(emerge_config.opts, emerge_config.trees)
+ settings, trees, mtimedb = emerge_config
+
if "--autounmask-only" in myopts:
mydepgraph.display_problems()
return 0
@@ -3230,10 +3249,7 @@ def run_action(emerge_config):
except OSError:
writemsg("Please install eselect to use
this feature.\n",
noiselevel=-1)
- retval = action_build(emerge_config.target_config.settings,
- emerge_config.trees,
emerge_config.target_config.mtimedb,
- emerge_config.opts, emerge_config.action,
- emerge_config.args, spinner)
+ retval = action_build(emerge_config, spinner=spinner)
post_emerge(emerge_config.action, emerge_config.opts,
emerge_config.args, emerge_config.target_config.root,
emerge_config.trees,
emerge_config.target_config.mtimedb, retval)
diff --git a/pym/_emerge/depgraph.py b/pym/_e
Re: [gentoo-portage-dev] [PATCH] Add emerge --autounmask-continue option (bug 582624)
On 07/01/2016 03:29 AM, Alexander Berntsen wrote: > The patch itself looks OK, but I think that this option is a bad idea > and design, and that the extra complexity isn't warranted. I know > users have asked for something similar several times, but thankfully > the users aren't the developers. It's an extremely useful option in certain contexts (especially continuous integration). I have a wrapper script that does this, but it's tricky to emulate this behavior with existing options, since without it there's no way to know whether or not the dependency calculation was completely successful (then you have to check if an config changes were made, apply them, and waste time repeating the dependency calculation all over again without knowing whether or not it will succeed). The patch is really much less complex than I had imagined before I started writing it. I expect it to be quite maintainable. > But if you genuinely think this is a good idea, and someone else on > the team does too, I won't oppose it. We should make sure that we > strongly discourage its usage for regular users. Perhaps your > suggested manpage addition already does -- I don't know. Yeah, I think the warning message that I've put in the man patch is pretty good: > This option is intended to be used only with great caution, > since it is possible for it to make nonsensical configuration > changes which may lead to system breakage. Therefore, it is > advisable to use ---ask together with this option. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] Add emerge --autounmask-continue option (bug 582624)
On 07/01/2016 05:45 AM, Francesco Riosa wrote: > Hi, > thanks for your continuous work, just an implementation detail, it may > be possible to avoid a new option reusing the current autounmask one? > > instead of > --autounmask [ y | n ] > --autounmask-only [ y | n ] > --autounmask-write [ y | n ] > --autounmask-continue [ y | n ] > > something like: > --autounmask [ y | n | only | write | continue] > > especially if these options make no sense used together That's an interesting idea, but my experience is that boolean options lead to simpler code, so I would prefer to stick with booleans. -- Thanks, Zac
Re: [gentoo-portage-dev] Re: [PATCH] Add emerge --autounmask-continue option (bug 582624)
On 07/01/2016 09:42 AM, Duncan wrote: > Zac Medico posted on Fri, 01 Jul 2016 08:35:26 -0700 as excerpted: > >>> But if you genuinely think this is a good idea, and someone else on the >>> team does too, I won't oppose it. We should make sure that we strongly >>> discourage its usage for regular users. Perhaps your suggested manpage >>> addition already does -- I don't know. >> >> Yeah, I think the warning message that I've put in the man patch is >> pretty good: >> >>> This option is intended to be used only with great caution, >>> since it is possible for it to make nonsensical configuration changes >>> which may lead to system breakage. Therefore, it is advisable to use >>> ---ask together with this option. > > Perhaps rename the option so it makes perfectly clear the possible > consequences? Something like --autounmask-breakme, or --auto-breakme ? My experience with my wrapper script that gives similar behavior is that it practically always "just works". It's fabulous for continuous integration (aka tinderbox) settings. However, as with self-driving cars, it deserves caution. > Or alternatively, if there are other arguably dangerous options now or > possible in the future, put them all under another option, --breakme, > such that if that option isn't there, the otherwise dangerous options > only print a warning and die. > > Then people can read the manpage if they really want to know what it > does, but people who haven't, aren't as likely to blunder into it due to > the stereotypical "rm -rf .*" type advice. It's simply not as risky as you're making it out to be. If it's a production system, use --ask. Honestly, people who can't be exposed to options like this should not have root access. -- Thanks, Zac
[gentoo-portage-dev] Re: [PATCH] Add emerge --autounmask-continue option (bug 582624)
On 07/01/2016 12:37 AM, Zac Medico wrote: > @@ -327,6 +341,11 @@ def action_build(settings, trees, mtimedb, > display_missing_pkg_set(root_config, e.value) > return 1 > > + if success and mydepgraph.need_config_reload(): > + load_emerge_config(emerge_config=emerge_config) > + adjust_configs(emerge_config.opts, emerge_config.trees) > + settings, trees, mtimedb = emerge_config > + The Package instances inside the depgraph are still going to reference the old RootConfig instances, so I'll update the patch to solve that somehow. -- Thanks, Zac
[gentoo-portage-dev] [PATCH v2] Add emerge --autounmask-continue option (bug 582624)
This option will cause emerge to automatically apply autounmask changes
to configuration files, and continue to execute the specified command.
If the dependency calculation is not entirely successful, then emerge
will simply abort without modifying any configuration files.
This sort of behavior can be very useful in a continuous integration
setting, where the emerge invocation might be inside of a container that
is later discarded (so there is no threat of negative consequences).
It's also safe for general use, when combined with the --ask option.
X-Gentoo-Bug: 582624
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=582624
---
[PATCH v2] changes:
* fix depgraph to update USE state of Package instances to be consistent
with config file changes
* fix load_emerge_config to update existing RootConfig instances
in-place, so that the update propagates globally (to depgraph and the
Package instances it contains)
* include a use configuration change in the unit test, and assert that
it is applied correctly
man/emerge.1| 14 +++-
pym/_emerge/actions.py | 36 +++--
pym/_emerge/depgraph.py | 40 +
pym/_emerge/main.py | 9
pym/portage/tests/emerge/test_simple.py | 15 +
5 files changed, 102 insertions(+), 12 deletions(-)
diff --git a/man/emerge.1 b/man/emerge.1
index bfa2f73..40be14f 100644
--- a/man/emerge.1
+++ b/man/emerge.1
@@ -1,4 +1,4 @@
-.TH "EMERGE" "1" "Feb 2016" "Portage VERSION" "Portage"
+.TH "EMERGE" "1" "Jul 2016" "Portage VERSION" "Portage"
.SH "NAME"
emerge \- Command\-line interface to the Portage system
.SH "SYNOPSIS"
@@ -361,6 +361,18 @@ the specified configuration file(s), or enable the
\fBEMERGE_DEFAULT_OPTS\fR variable may be used to
disable this option by default in \fBmake.conf\fR(5).
.TP
+.BR "\-\-autounmask\-continue [ y | n ]"
+Automatically apply autounmask changes to configuration
+files, and continue to execute the specified command. If
+the dependency calculation is not entirely successful, then
+emerge will simply abort without modifying any configuration
+files.
+\fBWARNING:\fR
+This option is intended to be used only with great caution,
+since it is possible for it to make nonsensical configuration
+changes which may lead to system breakage. Therefore, it is
+advisable to use \fB\-\-ask\fR together with this option.
+.TP
.BR "\-\-autounmask\-only [ y | n ]"
Instead of doing any package building, just unmask
packages and generate package.use settings as necessary
diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
index 2ca7902..1dc2b0d 100644
--- a/pym/_emerge/actions.py
+++ b/pym/_emerge/actions.py
@@ -96,8 +96,22 @@ if sys.hexversion >= 0x300:
else:
_unicode = unicode
-def action_build(settings, trees, mtimedb,
- myopts, myaction, myfiles, spinner):
+def action_build(emerge_config, trees=DeprecationWarning,
+ mtimedb=DeprecationWarning, myopts=DeprecationWarning,
+ myaction=DeprecationWarning, myfiles=DeprecationWarning, spinner=None):
+
+ if not isinstance(emerge_config, _emerge_config):
+ warnings.warn("_emerge.actions.action_build() now expects "
+ "an _emerge_config instance as the first parameter",
+ DeprecationWarning, stacklevel=2)
+ emerge_config = load_emerge_config(
+ action=myaction, args=myfiles, trees=trees, opts=myopts)
+ adjust_configs(emerge_config.opts, emerge_config.trees)
+
+ settings, trees, mtimedb = emerge_config
+ myopts = emerge_config.opts
+ myaction = emerge_config.action
+ myfiles = emerge_config.args
if '--usepkgonly' not in myopts:
old_tree_timestamp_warn(settings['PORTDIR'], settings)
@@ -327,6 +341,11 @@ def action_build(settings, trees, mtimedb,
display_missing_pkg_set(root_config, e.value)
return 1
+ if success and mydepgraph.need_config_reload():
+ load_emerge_config(emerge_config=emerge_config)
+ adjust_configs(emerge_config.opts, emerge_config.trees)
+ settings, trees, mtimedb = emerge_config
+
if "--autounmask-only" in myopts:
mydepgraph.display_problems()
return 0
@@ -2384,7 +2403,13 @@ def load_emerge_config(emerge_config=None, **kargs):
settings = root_trees["vartree"].settings
settings._init_dirs()
setconfig = load_default_config(settings, root_trees)
- root_trees["root_config"] = RootConfig(settings, root_trees,
setconfig)
+ root_config = RootConfig(settings, root_trees, setconfig)
+ if "root_config" in root_trees:
+ # Propagate changes to the exist
[gentoo-portage-dev] Re: [PATCH] Add emerge --autounmask-continue option (bug 582624)
On 07/01/2016 03:46 PM, Zac Medico wrote: > On 07/01/2016 12:37 AM, Zac Medico wrote: >> @@ -327,6 +341,11 @@ def action_build(settings, trees, mtimedb, >> display_missing_pkg_set(root_config, e.value) >> return 1 >> >> +if success and mydepgraph.need_config_reload(): >> +load_emerge_config(emerge_config=emerge_config) >> +adjust_configs(emerge_config.opts, emerge_config.trees) >> +settings, trees, mtimedb = emerge_config >> + > > The Package instances inside the depgraph are still going to reference > the old RootConfig instances, so I'll update the patch to solve that > somehow. Fixed in v2 by making load_emerge_config update existing RootConfig instances in-place. -- Thanks, Zac
Re: [gentoo-portage-dev] Re: [PATCH] Add emerge --autounmask-continue option (bug 582624)
On 07/04/2016 05:16 AM, Alexander Berntsen wrote: > Looks OK. But you've found a few bugs already. Maybe you'll find more. > I'd appreciate if you hold off until the end of the week before > pushing it confidently. I'm extremely confident in v2 of the patch. I don't expect that we'll find any bugs in it, and even if we do find bugs, I think it's even less likely that those bugs would affect anyone who's not using --autounmask-continue. -- Thanks, Zac
Re: [gentoo-portage-dev] Re: [PATCH] Add emerge --autounmask-continue option (bug 582624)
On 07/04/2016 05:17 AM, Alexander Berntsen wrote: > Never mind. I see that it's already pushed. I guess this is where I > continue to argue my case for "Pushed as [commit hash]" emails. I wasn't aware that there was an argument about that. I'll be happy to send pushed emails. -- Thanks, Zac
Re: [gentoo-portage-dev] Re: [PATCH] Add emerge --autounmask-continue option (bug 582624)
On 07/02/2016 12:31 PM, Brian Dolbec wrote: > On Fri, 1 Jul 2016 22:40:47 -0700 > Zac Medico wrote: > >> On 07/01/2016 03:46 PM, Zac Medico wrote: >>> On 07/01/2016 12:37 AM, Zac Medico wrote: >>>> @@ -327,6 +341,11 @@ def action_build(settings, trees, mtimedb, >>>>display_missing_pkg_set(root_config, >>>> e.value) return 1 >>>> >>>> + if success and mydepgraph.need_config_reload(): >>>> + >>>> load_emerge_config(emerge_config=emerge_config) >>>> + adjust_configs(emerge_config.opts, >>>> emerge_config.trees) >>>> + settings, trees, mtimedb = emerge_config >>>> + >>> >>> The Package instances inside the depgraph are still going to >>> reference the old RootConfig instances, so I'll update the patch to >>> solve that somehow. >> >> Fixed in v2 by making load_emerge_config update existing RootConfig >> instances in-place. > > looks fine to me Thanks, pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=e2d88ef3ff5958571ef2da5f69e8756522390f0f -- Thanks, Zac
[gentoo-portage-dev] [PATCH] portage.cache: write md5 instead of mtime (bug 568934)
Change cache modules to write md5 in cache entries, instead of mtime.
Since portage-2.2.27, the relevant cache modules have had the ability
to read cache entries containing either md5 or mtime, therefore this
change is backward-compatible with portage-2.2.27 and later.
Also, fix the reconstruct_eclasses function to raise CacheCorruption
when the specified chf_type is md5 and the cache entry contains mtime
data. This is needed so that the cache module chf_types attributes can
list md5 before mtime, without having mtime data be incorrectly
interpreted as md5 data.
X-Gentoo-Bug: 568934
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=568934
---
pym/portage/cache/anydbm.py| 4 ++--
pym/portage/cache/flat_hash.py | 4 ++--
pym/portage/cache/sqlite.py| 4 ++--
pym/portage/cache/template.py | 19 +++
4 files changed, 21 insertions(+), 10 deletions(-)
diff --git a/pym/portage/cache/anydbm.py b/pym/portage/cache/anydbm.py
index 80d24e5..88d85b0 100644
--- a/pym/portage/cache/anydbm.py
+++ b/pym/portage/cache/anydbm.py
@@ -36,8 +36,8 @@ from portage.cache import cache_errors
class database(fs_template.FsBased):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
autocommits = True
cleanse_keys = True
diff --git a/pym/portage/cache/flat_hash.py b/pym/portage/cache/flat_hash.py
index cca0f10..3a899c0 100644
--- a/pym/portage/cache/flat_hash.py
+++ b/pym/portage/cache/flat_hash.py
@@ -163,5 +163,5 @@ class md5_database(database):
class mtime_md5_database(database):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
diff --git a/pym/portage/cache/sqlite.py b/pym/portage/cache/sqlite.py
index 32e4076..69150f6 100644
--- a/pym/portage/cache/sqlite.py
+++ b/pym/portage/cache/sqlite.py
@@ -18,8 +18,8 @@ if sys.hexversion >= 0x300:
class database(fs_template.FsBased):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
autocommits = False
synchronous = False
diff --git a/pym/portage/cache/template.py b/pym/portage/cache/template.py
index a7c6de0..021c706 100644
--- a/pym/portage/cache/template.py
+++ b/pym/portage/cache/template.py
@@ -310,6 +310,18 @@ def serialize_eclasses(eclass_dict, chf_type='mtime',
paths=True):
for k, v in sorted(eclass_dict.items(), key=_keysorter))
+def _md5_deserializer(md5):
+ if len(md5) != 32:
+ raise ValueError('expected 32 hex digits')
+ return md5
+
+
+_chf_deserializers = {
+ 'md5': _md5_deserializer,
+ 'mtime': long,
+}
+
+
def reconstruct_eclasses(cpv, eclass_string, chf_type='mtime', paths=True):
"""returns a dict when handed a string generated by
serialize_eclasses"""
eclasses = eclass_string.rstrip().lstrip().split("\t")
@@ -317,9 +329,7 @@ def reconstruct_eclasses(cpv, eclass_string,
chf_type='mtime', paths=True):
# occasionally this occurs in the fs backends. they suck.
return {}
- converter = _unicode
- if chf_type == 'mtime':
- converter = long
+ converter = _chf_deserializers.get(chf_type, lambda x: x)
if paths:
if len(eclasses) % 3 != 0:
@@ -340,6 +350,7 @@ def reconstruct_eclasses(cpv, eclass_string,
chf_type='mtime', paths=True):
raise cache_errors.CacheCorruption(cpv,
"_eclasses_ was of invalid len %i" % len(eclasses))
except ValueError:
- raise cache_errors.CacheCorruption(cpv, "_eclasses_ mtime
conversion to long failed")
+ raise cache_errors.CacheCorruption(cpv,
+ "_eclasses_ not valid for chf_type {}".format(chf_type))
del eclasses
return d
--
2.7.4
[gentoo-portage-dev] [PATCH v2] portage.cache: write md5 instead of mtime (bug 568934)
Change cache modules to write md5 in cache entries, instead of mtime.
Since portage-2.2.27, the relevant cache modules have had the ability
to read cache entries containing either md5 or mtime, therefore this
change is backward-compatible with portage-2.2.27 and later.
Also fix the reconstruct_eclasses function to raise CacheCorruption
when the specified chf_type is md5 and the cache entry contains mtime
data, and optimize __getitem__ to skip reconstruct_eclasses calls when
the entry appears to have a different chf_type.
X-Gentoo-Bug: 568934
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=568934
---
[PATCH v2] adds a __getitem__ optimization to skip reconstruct_eclasses
calls when the entry appears to have a different chf_type
pym/portage/cache/anydbm.py| 4 ++--
pym/portage/cache/flat_hash.py | 4 ++--
pym/portage/cache/sqlite.py| 4 ++--
pym/portage/cache/template.py | 23 +++
4 files changed, 25 insertions(+), 10 deletions(-)
diff --git a/pym/portage/cache/anydbm.py b/pym/portage/cache/anydbm.py
index 80d24e5..88d85b0 100644
--- a/pym/portage/cache/anydbm.py
+++ b/pym/portage/cache/anydbm.py
@@ -36,8 +36,8 @@ from portage.cache import cache_errors
class database(fs_template.FsBased):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
autocommits = True
cleanse_keys = True
diff --git a/pym/portage/cache/flat_hash.py b/pym/portage/cache/flat_hash.py
index cca0f10..3a899c0 100644
--- a/pym/portage/cache/flat_hash.py
+++ b/pym/portage/cache/flat_hash.py
@@ -163,5 +163,5 @@ class md5_database(database):
class mtime_md5_database(database):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
diff --git a/pym/portage/cache/sqlite.py b/pym/portage/cache/sqlite.py
index 32e4076..69150f6 100644
--- a/pym/portage/cache/sqlite.py
+++ b/pym/portage/cache/sqlite.py
@@ -18,8 +18,8 @@ if sys.hexversion >= 0x300:
class database(fs_template.FsBased):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
autocommits = False
synchronous = False
diff --git a/pym/portage/cache/template.py b/pym/portage/cache/template.py
index a7c6de0..24d8f8f 100644
--- a/pym/portage/cache/template.py
+++ b/pym/portage/cache/template.py
@@ -54,6 +54,10 @@ class database(object):
if self.serialize_eclasses and "_eclasses_" in d:
for chf_type in chf_types:
+ if '_%s_' % chf_type not in d:
+ # Skip the reconstruct_eclasses call,
since this
+ # entry appears to have a different
chf_type.
+ continue
try:
d["_eclasses_"] =
reconstruct_eclasses(cpv, d["_eclasses_"],
chf_type,
paths=self.store_eclass_paths)
@@ -310,6 +314,18 @@ def serialize_eclasses(eclass_dict, chf_type='mtime',
paths=True):
for k, v in sorted(eclass_dict.items(), key=_keysorter))
+def _md5_deserializer(md5):
+ if len(md5) != 32:
+ raise ValueError('expected 32 hex digits')
+ return md5
+
+
+_chf_deserializers = {
+ 'md5': _md5_deserializer,
+ 'mtime': long,
+}
+
+
def reconstruct_eclasses(cpv, eclass_string, chf_type='mtime', paths=True):
"""returns a dict when handed a string generated by
serialize_eclasses"""
eclasses = eclass_string.rstrip().lstrip().split("\t")
@@ -317,9 +333,7 @@ def reconstruct_eclasses(cpv, eclass_string,
chf_type='mtime', paths=True):
# occasionally this occurs in the fs backends. they suck.
return {}
- converter = _unicode
- if chf_type == 'mtime':
- converter = long
+ converter = _chf_deserializers.get(chf_type, lambda x: x)
if paths:
if len(eclasses) % 3 != 0:
@@ -340,6 +354,7 @@ def reconstruct_eclasses(cpv, eclass_string,
chf_type='mtime', paths=True):
raise cache_errors.CacheCorruption(cpv,
"_eclasses_ was of invalid len %i" % len(eclasses))
except ValueError:
- raise cache_errors.CacheCorruption(cpv, "_eclasses_ mtime
conversion to long failed")
+ raise cache_errors.CacheCorruption(cpv,
+ "_eclasses_ not valid for chf_type {}".format(chf_type))
del eclasses
return d
--
2.7.4
[gentoo-portage-dev] [PATCH v3] portage.cache: write md5 instead of mtime (bug 568934)
Change cache modules to write md5 in cache entries, instead of mtime.
Since portage-2.2.27, the relevant cache modules have had the ability
to read cache entries containing either md5 or mtime, therefore this
change is backward-compatible with portage-2.2.27 and later.
Also fix the reconstruct_eclasses function to raise CacheCorruption
when the specified chf_type is md5 and the cache entry contains mtime
data, and optimize __getitem__ to skip reconstruct_eclasses calls when
the entry appears to have a different chf_type.
X-Gentoo-Bug: 568934
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=568934
---
[PATCH v3] fixes the __getitem__ optimization to ensure that
CacheCorruption is raised if a cache entry does not contain a
recognized chf_type
pym/portage/cache/anydbm.py| 4 ++--
pym/portage/cache/flat_hash.py | 4 ++--
pym/portage/cache/sqlite.py| 4 ++--
pym/portage/cache/template.py | 26 ++
4 files changed, 28 insertions(+), 10 deletions(-)
diff --git a/pym/portage/cache/anydbm.py b/pym/portage/cache/anydbm.py
index 80d24e5..88d85b0 100644
--- a/pym/portage/cache/anydbm.py
+++ b/pym/portage/cache/anydbm.py
@@ -36,8 +36,8 @@ from portage.cache import cache_errors
class database(fs_template.FsBased):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
autocommits = True
cleanse_keys = True
diff --git a/pym/portage/cache/flat_hash.py b/pym/portage/cache/flat_hash.py
index cca0f10..3a899c0 100644
--- a/pym/portage/cache/flat_hash.py
+++ b/pym/portage/cache/flat_hash.py
@@ -163,5 +163,5 @@ class md5_database(database):
class mtime_md5_database(database):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
diff --git a/pym/portage/cache/sqlite.py b/pym/portage/cache/sqlite.py
index 32e4076..69150f6 100644
--- a/pym/portage/cache/sqlite.py
+++ b/pym/portage/cache/sqlite.py
@@ -18,8 +18,8 @@ if sys.hexversion >= 0x300:
class database(fs_template.FsBased):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
autocommits = False
synchronous = False
diff --git a/pym/portage/cache/template.py b/pym/portage/cache/template.py
index a7c6de0..d292eed 100644
--- a/pym/portage/cache/template.py
+++ b/pym/portage/cache/template.py
@@ -54,6 +54,10 @@ class database(object):
if self.serialize_eclasses and "_eclasses_" in d:
for chf_type in chf_types:
+ if '_%s_' % chf_type not in d:
+ # Skip the reconstruct_eclasses call,
since this
+ # entry appears to have a different
chf_type.
+ continue
try:
d["_eclasses_"] =
reconstruct_eclasses(cpv, d["_eclasses_"],
chf_type,
paths=self.store_eclass_paths)
@@ -62,6 +66,9 @@ class database(object):
raise
else:
break
+ else:
+ raise cache_errors.CacheCorruption(cpv,
+ 'entry does not contain a recognized
chf_type')
elif "_eclasses_" not in d:
d["_eclasses_"] = {}
@@ -310,6 +317,18 @@ def serialize_eclasses(eclass_dict, chf_type='mtime',
paths=True):
for k, v in sorted(eclass_dict.items(), key=_keysorter))
+def _md5_deserializer(md5):
+ if len(md5) != 32:
+ raise ValueError('expected 32 hex digits')
+ return md5
+
+
+_chf_deserializers = {
+ 'md5': _md5_deserializer,
+ 'mtime': long,
+}
+
+
def reconstruct_eclasses(cpv, eclass_string, chf_type='mtime', paths=True):
"""returns a dict when handed a string generated by
serialize_eclasses"""
eclasses = eclass_string.rstrip().lstrip().split("\t")
@@ -317,9 +336,7 @@ def reconstruct_eclasses(cpv, eclass_string,
chf_type='mtime', paths=True):
# occasionally this occurs in the fs backends. they suck.
return {}
- converter = _unicode
- if chf_type == 'mtime':
- converter = long
+ converter = _chf_deserializers.get(chf_type, lambda x: x)
if paths:
if len(eclasses) % 3 != 0:
@@ -340,6 +357,7 @@ def reconstruct_eclasses(cpv, eclass_string,
chf_type='mtime', paths=True):
raise cache_errors.CacheCorruption(cpv,
"_eclasses_ was of invalid len %i" % len(eclasses))
except ValueError:
- raise cache_errors.CacheCorruption
Re: [gentoo-portage-dev] [PATCH v3] portage.cache: write md5 instead of mtime (bug 568934)
On 07/12/2016 06:59 AM, Alexander Berntsen wrote: > The _chf_deserializers and _md5_deserializer stuff looks rather > overengineered. That stuff is not strictly required after the addition of the intelligent reconstruct_eclasses skipping in __getitem__. However, it's still good to have because it protects against a subtle misbehavior of reconstruct_eclasses, where it's called with chf_type='md5' and produces an invalid data structure containing mtime strings (rather than mtime ints). > I don't know what the reconstruct_eclasses skipping > entails (the comment makes it seem like "skip this because apparently > it's different lol who knows -_o_-"). The _eclasses_data contains either md5 or mtime. It's a waste of time to try to call reconstruct_eclasses with chf_type='md5' when eclasses contains mtime data (and it would also produce an invalid data structure in the absence of the _chf_deserializers and _md5_deserializer stuff). So, it's nice to take the presence of _md5_ or _mtime_ in the cache entry as a hint about whether _eclasses_ contains md5 or mtime data. > The rest of the patch lgtm. I'll add the insights that I have discussed above as comments the patch. -- Thanks, Zac
[gentoo-portage-dev] [PATCH v4] portage.cache: write md5 instead of mtime (bug 568934)
Change cache modules to write md5 in cache entries, instead of mtime.
Since portage-2.2.27, the relevant cache modules have had the ability
to read cache entries containing either md5 or mtime, therefore this
change is backward-compatible with portage-2.2.27 and later.
Also fix the reconstruct_eclasses function to raise CacheCorruption
when the specified chf_type is md5 and the cache entry contains mtime
data, and optimize __getitem__ to skip reconstruct_eclasses calls when
the entry appears to have a different chf_type.
X-Gentoo-Bug: 568934
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=568934
---
[PATCH v4] adds some comments to clarify the purposes of the __getitem__
optimization and _md5_deserializer stuff
pym/portage/cache/anydbm.py| 4 ++--
pym/portage/cache/flat_hash.py | 4 ++--
pym/portage/cache/sqlite.py| 4 ++--
pym/portage/cache/template.py | 36
4 files changed, 38 insertions(+), 10 deletions(-)
diff --git a/pym/portage/cache/anydbm.py b/pym/portage/cache/anydbm.py
index 80d24e5..88d85b0 100644
--- a/pym/portage/cache/anydbm.py
+++ b/pym/portage/cache/anydbm.py
@@ -36,8 +36,8 @@ from portage.cache import cache_errors
class database(fs_template.FsBased):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
autocommits = True
cleanse_keys = True
diff --git a/pym/portage/cache/flat_hash.py b/pym/portage/cache/flat_hash.py
index cca0f10..3a899c0 100644
--- a/pym/portage/cache/flat_hash.py
+++ b/pym/portage/cache/flat_hash.py
@@ -163,5 +163,5 @@ class md5_database(database):
class mtime_md5_database(database):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
diff --git a/pym/portage/cache/sqlite.py b/pym/portage/cache/sqlite.py
index 32e4076..69150f6 100644
--- a/pym/portage/cache/sqlite.py
+++ b/pym/portage/cache/sqlite.py
@@ -18,8 +18,8 @@ if sys.hexversion >= 0x300:
class database(fs_template.FsBased):
- validation_chf = 'mtime'
- chf_types = ('mtime', 'md5')
+ validation_chf = 'md5'
+ chf_types = ('md5', 'mtime')
autocommits = False
synchronous = False
diff --git a/pym/portage/cache/template.py b/pym/portage/cache/template.py
index a7c6de0..8662d85 100644
--- a/pym/portage/cache/template.py
+++ b/pym/portage/cache/template.py
@@ -54,6 +54,15 @@ class database(object):
if self.serialize_eclasses and "_eclasses_" in d:
for chf_type in chf_types:
+ if '_%s_' % chf_type not in d:
+ # Skip the reconstruct_eclasses call,
since it's
+ # a waste of time if it contains a
different chf_type
+ # than the current one. In the past, it
was possible
+ # for reconstruct_eclasses called with
chf_type='md5'
+ # to "successfully" return invalid data
here, because
+ # it was unable to distinguish between
md5 data and
+ # mtime data.
+ continue
try:
d["_eclasses_"] =
reconstruct_eclasses(cpv, d["_eclasses_"],
chf_type,
paths=self.store_eclass_paths)
@@ -62,6 +71,9 @@ class database(object):
raise
else:
break
+ else:
+ raise cache_errors.CacheCorruption(cpv,
+ 'entry does not contain a recognized
chf_type')
elif "_eclasses_" not in d:
d["_eclasses_"] = {}
@@ -310,6 +322,23 @@ def serialize_eclasses(eclass_dict, chf_type='mtime',
paths=True):
for k, v in sorted(eclass_dict.items(), key=_keysorter))
+def _md5_deserializer(md5):
+ """
+ Without this validation, it's possible for reconstruct_eclasses to
+ mistakenly interpret mtime data as md5 data, and return an invalid
+ data structure containing strings where ints are expected.
+ """
+ if len(md5) != 32:
+ raise ValueError('expected 32 hex digits')
+ return md5
+
+
+_chf_deserializers = {
+ 'md5': _md5_deserializer,
+ 'mtime': long,
+}
+
+
def reconstruct_eclasses(cpv, eclass_string, chf_type='mtime', paths=True):
"""returns a dict when handed a string generated by
serialize_eclasses"""
eclasses = eclass_string.rstrip().lstrip().split("\t")
@@ -317,9 +346,7 @@ def reconstruct_eclasses(cpv, eclass_string,
c
Re: [gentoo-portage-dev] [PATCH v4] portage.cache: write md5 instead of mtime (bug 568934)
On 07/13/2016 04:16 AM, Alexander Berntsen wrote: > LGTM. > > Pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=9abbda7d054761ae6c333d3e6d420632b9658b6d -- Thanks, Zac
[gentoo-portage-dev] [PATCH] GitSync.update: respect sync-depth (bug 552814)
Fix updates to respect sync-depth (previously it was only respected
for clone operations). Since the default merge strategy typically
fails when the the depth is not unlimited, use `git fetch` followed
by `git reset --hard`.
X-Gentoo-Bug: 552814
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=552814
---
pym/portage/sync/modules/git/git.py | 22 +-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/pym/portage/sync/modules/git/git.py
b/pym/portage/sync/modules/git/git.py
index 02eeb16..62c24f6 100644
--- a/pym/portage/sync/modules/git/git.py
+++ b/pym/portage/sync/modules/git/git.py
@@ -83,7 +83,21 @@ class GitSync(NewBase):
git_cmd_opts += " --quiet"
if
self.repo.module_specific_options.get('sync-git-pull-extra-opts'):
git_cmd_opts += " %s" %
self.repo.module_specific_options['sync-git-pull-extra-opts']
- git_cmd = "%s pull%s" % (self.bin_command, git_cmd_opts)
+ if self.repo.sync_depth is None:
+ git_cmd = "%s pull%s" % (self.bin_command, git_cmd_opts)
+ else:
+ # Since the default merge strategy typically fails when
+ # the depth is not unlimited, use `git fetch` followed
by
+ # `git reset --hard`.
+ remote_branch = portage._unicode_decode(
+ subprocess.check_output([self.bin_command,
'rev-parse',
+ '--abbrev-ref', '--symbolic-full-name',
'@{upstream}'],
+
cwd=portage._unicode_encode(self.repo.location))).rstrip('\n')
+
+ git_cmd_opts += " --depth %d" % self.repo.sync_depth
+ git_cmd = "%s fetch %s%s" % (self.bin_command,
+ remote_branch.partition('/')[0], git_cmd_opts)
+
writemsg_level(git_cmd + "\n")
rev_cmd = [self.bin_command, "rev-list", "--max-count=1",
"HEAD"]
@@ -93,6 +107,12 @@ class GitSync(NewBase):
exitcode = portage.process.spawn_bash("cd %s ; exec %s" % (
portage._shell_quote(self.repo.location),
git_cmd),
**self.spawn_kwargs)
+
+ if exitcode == os.EX_OK and self.repo.sync_depth is not None:
+ exitcode = subprocess.call(
+ [self.bin_command, 'reset', '--hard',
remote_branch],
+ cwd=portage._unicode_encode(self.repo.location))
+
if exitcode != os.EX_OK:
msg = "!!! git pull error in %s" % self.repo.location
self.logger(self.xterm_titles, msg)
--
2.7.4
[gentoo-portage-dev] [PATCH v2] GitSync.update: respect sync-depth (bug 552814)
Fix updates to respect sync-depth (previously it was only respected
for clone operations). Since the default merge strategy typically
fails when the the depth is not unlimited, use `git fetch` followed
by `git reset --hard`.
X-Gentoo-Bug: 552814
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=552814
---
[PATCH v2] fixes the git reset call to use --quiet when appropriate
pym/portage/sync/modules/git/git.py | 27 +--
1 file changed, 25 insertions(+), 2 deletions(-)
diff --git a/pym/portage/sync/modules/git/git.py
b/pym/portage/sync/modules/git/git.py
index 02eeb16..09257f3 100644
--- a/pym/portage/sync/modules/git/git.py
+++ b/pym/portage/sync/modules/git/git.py
@@ -79,11 +79,26 @@ class GitSync(NewBase):
'''
git_cmd_opts = ""
- if self.settings.get("PORTAGE_QUIET") == "1":
+ quiet = self.settings.get("PORTAGE_QUIET") == "1"
+ if quiet:
git_cmd_opts += " --quiet"
if
self.repo.module_specific_options.get('sync-git-pull-extra-opts'):
git_cmd_opts += " %s" %
self.repo.module_specific_options['sync-git-pull-extra-opts']
- git_cmd = "%s pull%s" % (self.bin_command, git_cmd_opts)
+ if self.repo.sync_depth is None:
+ git_cmd = "%s pull%s" % (self.bin_command, git_cmd_opts)
+ else:
+ # Since the default merge strategy typically fails when
+ # the depth is not unlimited, use `git fetch` followed
by
+ # `git reset --hard`.
+ remote_branch = portage._unicode_decode(
+ subprocess.check_output([self.bin_command,
'rev-parse',
+ '--abbrev-ref', '--symbolic-full-name',
'@{upstream}'],
+
cwd=portage._unicode_encode(self.repo.location))).rstrip('\n')
+
+ git_cmd_opts += " --depth %d" % self.repo.sync_depth
+ git_cmd = "%s fetch %s%s" % (self.bin_command,
+ remote_branch.partition('/')[0], git_cmd_opts)
+
writemsg_level(git_cmd + "\n")
rev_cmd = [self.bin_command, "rev-list", "--max-count=1",
"HEAD"]
@@ -93,6 +108,14 @@ class GitSync(NewBase):
exitcode = portage.process.spawn_bash("cd %s ; exec %s" % (
portage._shell_quote(self.repo.location),
git_cmd),
**self.spawn_kwargs)
+
+ if exitcode == os.EX_OK and self.repo.sync_depth is not None:
+ reset_cmd = [self.bin_command, 'reset', '--hard',
remote_branch]
+ if quiet:
+ reset_cmd.append('--quiet')
+ exitcode = subprocess.call(reset_cmd,
+ cwd=portage._unicode_encode(self.repo.location))
+
if exitcode != os.EX_OK:
msg = "!!! git pull error in %s" % self.repo.location
self.logger(self.xterm_titles, msg)
--
2.7.4
Re: [gentoo-portage-dev] [PATCH v2] GitSync.update: respect sync-depth (bug 552814)
On 07/14/2016 10:08 AM, Brian Dolbec wrote: > looks fine Pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=84413bb1dd9df322568ce25efc5b7854a43d03c7 -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH v2] GitSync.update: respect sync-depth (bug 552814)
On 07/14/2016 11:43 AM, Zac Medico wrote: > On 07/14/2016 10:08 AM, Brian Dolbec wrote: >> looks fine > > Pushed: > > https://gitweb.gentoo.org/proj/portage.git/commit/?id=84413bb1dd9df322568ce25efc5b7854a43d03c7 > Now optimized to use `git reset --merge`: https://gitweb.gentoo.org/proj/portage.git/commit/?id=55aef9bf297ef8cbf29921acb454449d01313818 -- Thanks, Zac
[gentoo-portage-dev] [PATCH] emerge: add --fuzzy-search and --search-similarity (bug 65566)
Add --fuzzy-search option, and --search-similarity option to adjust
the minimum similarity for search results (defaults to 80%).
X-Gentoo-bug: 65566
X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=65566
---
man/emerge.1 | 14 ++
pym/_emerge/actions.py | 7 +--
pym/_emerge/main.py| 32 +++-
pym/_emerge/search.py | 26 --
4 files changed, 74 insertions(+), 5 deletions(-)
diff --git a/man/emerge.1 b/man/emerge.1
index da1d852..7442220 100644
--- a/man/emerge.1
+++ b/man/emerge.1
@@ -565,6 +565,14 @@ packages (fetch things from SRC_URI based upon USE
setting).
Instead of doing any package building, just perform fetches for all
packages (fetch everything in SRC_URI regardless of USE setting).
.TP
+.BR "\-\-fuzzy\-search [ y | n ]"
+Enable or disable fuzzy search for search actions. When fuzzy search
+is enabled, a result is returned if it is sufficiently similar to the
+search string, without requiring an exact match. This option is enabled
+by default. Fuzzy search does not support regular expressions, therefore
+it is automatically disabled for regular expression searches. Fuzzy
+search is slightly slower than non\-fuzzy search.
+.TP
.BR "\-\-getbinpkg [ y | n ] (\-g short option)"
Using the server and location defined in \fIPORTAGE_BINHOST\fR (see
\fBmake.conf\fR(5)), portage will download the information from each binary
@@ -874,6 +882,12 @@ enabled by default. The search index needs to be
regenerated by
to \fBEMERGE_DEFAULT_OPTS\fR (see \fBmake.conf\fR(5)) and later
overridden via the command line.
.TP
+.BR "\-\-search\-similarity PERCENTAGE"
+Set the minimum similarity percentage (a floating-point number between
+0 and 100). Search results with similarity percentages lower than this
+are discarded (default: \'80\'). This option has no effect unless the
+\fB\-\-fuzzy\-search\fR option is enabled.
+.TP
.BR "\-\-select [ y | n ] (\-w short option)"
Add specified packages to the world set (inverse of
\fB\-\-oneshot\fR). This is useful if you want to
diff --git a/pym/_emerge/actions.py b/pym/_emerge/actions.py
index 1dc2b0d..6704afc 100644
--- a/pym/_emerge/actions.py
+++ b/pym/_emerge/actions.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2015 Gentoo Foundation
+# Copyright 1999-2016 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
from __future__ import division, print_function, unicode_literals
@@ -1974,7 +1974,10 @@ def action_search(root_config, myopts, myfiles, spinner):
spinner, "--searchdesc" in myopts,
"--quiet" not in myopts, "--usepkg" in myopts,
"--usepkgonly" in myopts,
- search_index = myopts.get("--search-index", "y") != "n")
+ search_index=myopts.get("--search-index", "y") != "n",
+ search_similarity=myopts.get("--search-similarity"),
+ fuzzy=myopts.get("--fuzzy-search") != "n",
+ )
for mysearch in myfiles:
try:
searchinstance.execute(mysearch)
diff --git a/pym/_emerge/main.py b/pym/_emerge/main.py
index 0e672a2..eae1954 100644
--- a/pym/_emerge/main.py
+++ b/pym/_emerge/main.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2015 Gentoo Foundation
+# Copyright 1999-2016 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
from __future__ import print_function
@@ -141,6 +141,7 @@ def insert_optional_args(args):
'--deselect' : y_or_n,
'--binpkg-respect-use' : y_or_n,
'--fail-clean' : y_or_n,
+ '--fuzzy-search' : y_or_n,
'--getbinpkg': y_or_n,
'--getbinpkgonly': y_or_n,
'--jobs' : valid_integers,
@@ -458,6 +459,11 @@ def parse_opts(tmpcmdline, silent=False):
"choices" : true_y_or_n
},
+ "--fuzzy-search": {
+ "help": "Enable or disable fuzzy search",
+ "choices": true_y_or_n
+ },
+
"--ignore-built-slot-operator-deps": {
"help": "Ignore the slot/sub-slot := operator parts of
dependencies that have "
"been recorded when packages where built. This
option is intended "
@@ -658,6 +664,12 @@ def parse_opts(tmpcmdline, silent=False):
"choices": y_or_n
},
+ "--search-similarity": {
+ "help": ("Set minimum similarity percentage for fuzzy
seach "
+ "(a floating-point number between 0 and 100)"),
+ "action": "store"
+ },
+
"--select": {
"shortopt" : "-w",
Re: [gentoo-portage-dev] [PATCH] emerge: add --search-fuzzy and --search-fuzzy-cutoff options (bug 65566)
On 04/08/2016 04:33 AM, Alexander Berntsen wrote: > On 08/04/16 08:21, Zac Medico wrote: >> Reverse? You want it to measure dissimilarity? Not sure what you >> mean. > Sorry, I meant reverse the *docs* to mean "find things that are at > least 50% similar" rather than "cut off things that aren't above the > 0.5 threshold". I.e. use an inclusive sentence. I feel that this is > more clear. > >> I just want it to fail if the input is invalid. > Yes, I just realised you checked if it were <=, not just <. I think > this is a bad idea. It's easily missed -- I just missed it last time > around. I would suggest to make it fail early, rather than set it to > 0.0 which you then set to None. Just set it to None immediately. > I've just sent "[PATCH] emerge: add --fuzzy-search and --search-similarity (bug 65566)" which hopefully accounts for all of the previous feedback. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] depgraph._serialize_tasks: improve runtime cycle handling (bug 590514)
Previously, it was possible for _serialize_tasks to count some
dependencies of a runtime cycle as part of that cycle, leading to
sub-optimal merge order for these dependencies because they got
grouped together with the cycle in the overall merge order. Fix
it to separate these dependencies from the cycle, and merge them
earlier.
X-Gentoo-Bug: 590514
X-Gentoo-Bug-url: https://bugs.gentoo.org/show_bug.cgi?id=590514
---
pym/_emerge/depgraph.py| 50 +++
.../resolver/test_runtime_cycle_merge_order.py | 72 ++
2 files changed, 98 insertions(+), 24 deletions(-)
create mode 100644 pym/portage/tests/resolver/test_runtime_cycle_merge_order.py
diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py
index fc957f5..26037ad 100644
--- a/pym/_emerge/depgraph.py
+++ b/pym/_emerge/depgraph.py
@@ -7415,36 +7415,38 @@ class depgraph(object):
selected_nodes = set()
if gather_deps(ignore_priority,
mergeable_nodes,
selected_nodes, node):
- # When selecting
asap_nodes, we need to ensure
- # that we haven't
selected a large runtime cycle
- # that is obviously
sub-optimal. This will be
- # obvious if any of the
non-asap selected_nodes
- # is a leaf node when
medium_soft deps are
- # ignored.
- if prefer_asap and
asap_nodes and \
-
len(selected_nodes) > 1:
- for node in
selected_nodes.difference(
-
asap_nodes):
- if not
mygraph.child_nodes(node,
-
ignore_priority =
-
DepPriorityNormalRange.ignore_medium_soft):
-
selected_nodes = None
-
break
- if selected_nodes:
- if
smallest_cycle is None or \
-
len(selected_nodes) < len(smallest_cycle):
-
smallest_cycle = selected_nodes
+ if smallest_cycle is
None or \
+
len(selected_nodes) < len(smallest_cycle):
+ smallest_cycle
= selected_nodes
selected_nodes = smallest_cycle
- if selected_nodes and debug:
- writemsg("\nruntime cycle
digraph (%s nodes):\n\n" %
- (len(selected_nodes),),
noiselevel=-1)
+ if selected_nodes is not None:
cycle_digraph = mygraph.copy()
cycle_digraph.difference_update([x for x in
cycle_digraph if x not
in selected_nodes])
- cycle_digraph.debug_print()
- writemsg("\n", noiselevel=-1)
+
+ leaves =
cycle_digraph.leaf_nodes()
+ if leaves:
+ # NOTE: This case
should only be triggered when
+ # prefer_asap is True,
since otherwise these
+ # leaves would have
been selected to merge
+ # before this point.
Since these "leaves" may
+ # actually have some
low-priority dependencies
+ # that we have
intentionally ignored, select
+ # only one node here
Re: [gentoo-portage-dev] [PATCH] depgraph._serialize_tasks: improve runtime cycle handling (bug 590514)
On 08/07/2016 08:14 AM, Brian Dolbec wrote: > As usual, we are not resolver guru's, but the code changes look decent > enough :) > > LGTM > Thanks, pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=6412205462671735f6e8b3196a780bc4b0d6a077 -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] emerge: add --fuzzy-search and --search-similarity (bug 65566)
On 08/08/2016 02:22 AM, Alexander Berntsen wrote: > LGTM. Thanks for addressing my previous concerns. Thanks, pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=b7f0937a811ef4859333ccc45f8076dca109dc2f -- Thanks, Zac
[gentoo-portage-dev] [PATCH] repoman: fix erroneous LICENSE.syntax (bug 591184)
X-Gentoo-bug: 591184
X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591184
---
.../pym/repoman/modules/scan/depend/_depend_checks.py| 16 +++-
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/repoman/pym/repoman/modules/scan/depend/_depend_checks.py
b/repoman/pym/repoman/modules/scan/depend/_depend_checks.py
index 3f6c93e..f0ae863 100644
--- a/repoman/pym/repoman/modules/scan/depend/_depend_checks.py
+++ b/repoman/pym/repoman/modules/scan/depend/_depend_checks.py
@@ -27,7 +27,7 @@ def check_slotop(depstr, is_valid_flag, badsyntax, mytype,
token_class=portage.dep.Atom)
except portage.exception.InvalidDependString as e:
my_dep_tree = None
- badsyntax.append(str(e))
+ badsyntax.append((mytype, str(e)))
def _traverse_tree(dep_tree, in_any_of):
# leaf
@@ -67,7 +67,7 @@ def _depend_checks(ebuild, pkg, portdb, qatracker,
repo_metadata):
"java-pkg-opt-2" in ebuild.inherited,
inherited_wxwidgets_eclass = "wxwidgets" in ebuild.inherited
# operator_tokens = set(["||", "(", ")"])
- type_list, badsyntax = [], []
+ badsyntax = []
for mytype in Package._dep_keys + ("LICENSE", "PROPERTIES", "PROVIDE"):
mydepstr = ebuild.metadata[mytype]
@@ -83,7 +83,7 @@ def _depend_checks(ebuild, pkg, portdb, qatracker,
repo_metadata):
is_valid_flag=pkg.iuse.is_valid_flag,
token_class=token_class)
except portage.exception.InvalidDependString as e:
atoms = None
- badsyntax.append(str(e))
+ badsyntax.append((mytype, str(e)))
if atoms and mytype.endswith("DEPEND"):
if runtime and \
@@ -156,13 +156,11 @@ def _depend_checks(ebuild, pkg, portdb, qatracker,
repo_metadata):
check_missingslot(atom, mytype, ebuild.eapi,
portdb, qatracker,
ebuild.relative_path, ebuild.metadata)
- type_list.extend([mytype] * (len(badsyntax) - len(type_list)))
-
if runtime:
check_slotop(mydepstr, pkg.iuse.is_valid_flag,
badsyntax, mytype, qatracker,
ebuild.relative_path)
- for m, b in zip(type_list, badsyntax):
+ for m, b in badsyntax:
if m.endswith("DEPEND"):
qacat = "dependency.syntax"
else:
@@ -171,9 +169,9 @@ def _depend_checks(ebuild, pkg, portdb, qatracker,
repo_metadata):
qacat, "%s: %s: %s" % (ebuild.relative_path, m, b))
# data required for some other tests
- badlicsyntax = len([z for z in type_list if z == "LICENSE"])
- badprovsyntax = len([z for z in type_list if z == "PROVIDE"])
- baddepsyntax = len(type_list) != badlicsyntax + badprovsyntax
+ badlicsyntax = len([z for z in badsyntax if z[0] == "LICENSE"])
+ badprovsyntax = len([z for z in badsyntax if z[0] == "PROVIDE"])
+ baddepsyntax = len(badsyntax) != badlicsyntax + badprovsyntax
badlicsyntax = badlicsyntax > 0
#badprovsyntax = badprovsyntax > 0
--
2.7.4
Re: [gentoo-portage-dev] [PATCH] repoman: fix erroneous LICENSE.syntax (bug 591184)
On 08/14/2016 08:18 AM, Brian Dolbec wrote: > On Sat, 13 Aug 2016 16:00:59 -0700 > Zac Medico wrote: > >> X-Gentoo-bug: 591184 >> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591184 >> --- >> .../pym/repoman/modules/scan/depend/_depend_checks.py| 16 >> +++- 1 file changed, 7 insertions(+), 9 deletions(-) > > looks good, thanks for the fast fix Zac. > Pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=99d8cdc0e9b011d39ae8537169ce2ca70a4e5f83 -- Thanks, Zac
[gentoo-portage-dev] [PATCH] Scheduler._terminate_tasks: purge _running_tasks (bug 425554)
Fix the _terminate_tasks method to purge unstarted tasks from self._running_tasks, so that they don't keep the main loop running indefinitely. X-Gentoo-bug: 425554 X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=425554 --- pym/_emerge/Scheduler.py | 15 ++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/pym/_emerge/Scheduler.py b/pym/_emerge/Scheduler.py index 97b826a..71fe75f 100644 --- a/pym/_emerge/Scheduler.py +++ b/pym/_emerge/Scheduler.py @@ -328,7 +328,20 @@ class Scheduler(PollScheduler): def _terminate_tasks(self): self._status_display.quiet = True for task in list(self._running_tasks.values()): - task.cancel() + if task.isAlive(): + # This task should keep the main loop running until + # it has had an opportunity to clean up after itself. + # Rely on its exit hook to remove it from + # self._running_tasks when it has finished cleaning up. + task.cancel() + else: + # This task has been waiting to be started in one of + # self._task_queues which are all cleared below. It + # will never be started, so purged it from + # self._running_tasks so that it won't keep the main + # loop running. + del self._running_tasks[id(task)] + for q in self._task_queues.values(): q.clear() -- 2.7.4
[gentoo-portage-dev] [PATCH] EventLoop: add run_until_complete method (bug 591760)
This emulates the asyncio.AbstractEventLoop.run_until_complete(future) interface, which will make it possible to reduce latency in situations where it is desirable for a loop to exit at the earliest opportunity. X-Gentoo-bug: 591760 X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760 --- pym/portage/tests/ebuild/test_ipc_daemon.py | 23 ++- pym/portage/util/_eventloop/EventLoop.py| 17 - 2 files changed, 30 insertions(+), 10 deletions(-) diff --git a/pym/portage/tests/ebuild/test_ipc_daemon.py b/pym/portage/tests/ebuild/test_ipc_daemon.py index 835f51f..68f139a 100644 --- a/pym/portage/tests/ebuild/test_ipc_daemon.py +++ b/pym/portage/tests/ebuild/test_ipc_daemon.py @@ -1,4 +1,4 @@ -# Copyright 2010-2015 Gentoo Foundation +# Copyright 2010-2016 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 import tempfile @@ -16,6 +16,7 @@ from portage.util import ensure_dirs from portage.util._async.ForkProcess import ForkProcess from portage.util._async.TaskScheduler import TaskScheduler from portage.util._eventloop.global_event_loop import global_event_loop +from portage.util.futures.futures import Future from _emerge.SpawnProcess import SpawnProcess from _emerge.EbuildBuildDir import EbuildBuildDir from _emerge.EbuildIpcDaemon import EbuildIpcDaemon @@ -140,19 +141,23 @@ class IpcDaemonTestCase(TestCase): build_dir.unlock() shutil.rmtree(tmpdir) - def _timeout_callback(self): - self._timed_out = True + def _timeout_callback(self, task_scheduler): + task_scheduler.cancel() + self._exit_callback(task_scheduler) + + def _exit_callback(self, task_scheduler): + if not self._run_done.done(): + self._run_done.set_result(True) def _run(self, event_loop, task_scheduler, timeout): - self._timed_out = False - timeout_id = event_loop.timeout_add(timeout, self._timeout_callback) + self._run_done = Future() + timeout_id = event_loop.timeout_add(timeout, + self._timeout_callback, task_scheduler) + task_scheduler.addExitListener(self._exit_callback) try: task_scheduler.start() - while not self._timed_out and task_scheduler.poll() is None: - event_loop.iteration() - if self._timed_out: - task_scheduler.cancel() + event_loop.run_until_complete(self._run_done) task_scheduler.wait() finally: event_loop.source_remove(timeout_id) diff --git a/pym/portage/util/_eventloop/EventLoop.py b/pym/portage/util/_eventloop/EventLoop.py index 8095400..8f13de3 100644 --- a/pym/portage/util/_eventloop/EventLoop.py +++ b/pym/portage/util/_eventloop/EventLoop.py @@ -1,4 +1,4 @@ -# Copyright 1999-2014 Gentoo Foundation +# Copyright 1999-2016 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 from __future__ import division @@ -577,6 +577,21 @@ class EventLoop(object): del self._poll_event_handlers[f] return True + def run_until_complete(self, future): + """ + Run until the Future is done. + + @type future: asyncio.Future + @param future: a Future to wait for + @rtype: object + @return: the Future's result + @raise: the Future's exception + """ + while not future.done(): + self.iteration() + + return future.result() + _can_poll_device = None def can_poll_device(): -- 2.7.4
Re: [gentoo-portage-dev] [PATCH] EventLoop: add run_until_complete method (bug 591760)
On 08/22/2016 03:18 AM, Alexander Berntsen wrote: > On 21/08/16 00:18, Zac Medico wrote: >> This emulates the >> asyncio.AbstractEventLoop.run_until_complete(future) interface > What's that, and why are we emulating it instead of just using it? We will be able to use it with python 3.4 and later, but not with older versions of python. Also, we have some more work to do first: * Migrate all internal use of the EventLoop.iteration method to the new run_until_complete(future) method, and remove the EventLoop.iteration method (or make it private as long as it's needed to implement run_until_complete for older python versions). * Implement all EventLoop methods using asyncio.AbstractEventLoop methods (but keep existing implementations for use with older python). >> which will make it possible to reduce latency in situations where it >> is desirable for a loop to exit at the earliest opportunity. > It makes it possible -- okay. But does it? I'm reluctant to change > things willy nilly, and add SLOCs, without a tangible benefit. Having a migration path to asyncio is the most tangible benefit, since it allows us to rely on a standard library instead of our own internal event loop implementation. I mention latency since it's a clear advantage of the run_until_complete API (at least in theory). -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] Scheduler._terminate_tasks: purge _running_tasks (bug 425554)
On 08/22/2016 03:14 AM, Alexander Berntsen wrote: > Woah. I remember this bug. Nice. LGTM. > Pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=d54a795615ccb769a25a0f8d6cc15ba930ec428f -- Thanks, Zac
[gentoo-portage-dev] [PATCH v2] EventLoop: add run_until_complete method (bug 591760)
This emulates the asyncio.AbstractEventLoop.run_until_complete(future) interface, which will make it possible to reduce latency in situations where it is desirable for a loop to exit at the earliest opportunity. The most tangible benefit of this change is that it provides a migration path to asyncio, which will allow us to rely on a standard library instead of our own internal event loop implementation. In order to migrate to asyncio, more work is planned: * Migrate all internal use of the EventLoop.iteration method to the new run_until_complete(future) method, and remove the EventLoop.iteration method (or make it private as long as it's needed to implement run_until_complete for older python versions). * Implement all EventLoop methods using asyncio.AbstractEventLoop methods (but keep existing implementations for use with older python). X-Gentoo-bug: 591760 X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760 --- [PATCH v2] only updates the commit message to provide more context pym/portage/tests/ebuild/test_ipc_daemon.py | 23 ++- pym/portage/util/_eventloop/EventLoop.py| 17 - 2 files changed, 30 insertions(+), 10 deletions(-) diff --git a/pym/portage/tests/ebuild/test_ipc_daemon.py b/pym/portage/tests/ebuild/test_ipc_daemon.py index 835f51f..68f139a 100644 --- a/pym/portage/tests/ebuild/test_ipc_daemon.py +++ b/pym/portage/tests/ebuild/test_ipc_daemon.py @@ -1,4 +1,4 @@ -# Copyright 2010-2015 Gentoo Foundation +# Copyright 2010-2016 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 import tempfile @@ -16,6 +16,7 @@ from portage.util import ensure_dirs from portage.util._async.ForkProcess import ForkProcess from portage.util._async.TaskScheduler import TaskScheduler from portage.util._eventloop.global_event_loop import global_event_loop +from portage.util.futures.futures import Future from _emerge.SpawnProcess import SpawnProcess from _emerge.EbuildBuildDir import EbuildBuildDir from _emerge.EbuildIpcDaemon import EbuildIpcDaemon @@ -140,19 +141,23 @@ class IpcDaemonTestCase(TestCase): build_dir.unlock() shutil.rmtree(tmpdir) - def _timeout_callback(self): - self._timed_out = True + def _timeout_callback(self, task_scheduler): + task_scheduler.cancel() + self._exit_callback(task_scheduler) + + def _exit_callback(self, task_scheduler): + if not self._run_done.done(): + self._run_done.set_result(True) def _run(self, event_loop, task_scheduler, timeout): - self._timed_out = False - timeout_id = event_loop.timeout_add(timeout, self._timeout_callback) + self._run_done = Future() + timeout_id = event_loop.timeout_add(timeout, + self._timeout_callback, task_scheduler) + task_scheduler.addExitListener(self._exit_callback) try: task_scheduler.start() - while not self._timed_out and task_scheduler.poll() is None: - event_loop.iteration() - if self._timed_out: - task_scheduler.cancel() + event_loop.run_until_complete(self._run_done) task_scheduler.wait() finally: event_loop.source_remove(timeout_id) diff --git a/pym/portage/util/_eventloop/EventLoop.py b/pym/portage/util/_eventloop/EventLoop.py index 8095400..8f13de3 100644 --- a/pym/portage/util/_eventloop/EventLoop.py +++ b/pym/portage/util/_eventloop/EventLoop.py @@ -1,4 +1,4 @@ -# Copyright 1999-2014 Gentoo Foundation +# Copyright 1999-2016 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 from __future__ import division @@ -577,6 +577,21 @@ class EventLoop(object): del self._poll_event_handlers[f] return True + def run_until_complete(self, future): + """ + Run until the Future is done. + + @type future: asyncio.Future + @param future: a Future to wait for + @rtype: object + @return: the Future's result + @raise: the Future's exception + """ + while not future.done(): + self.iteration() + + return future.result() + _can_poll_device = None def can_poll_device(): -- 2.7.4
Re: [gentoo-portage-dev] [PATCH v2] EventLoop: add run_until_complete method (bug 591760)
On 08/23/2016 08:40 AM, Alexander Berntsen wrote: > OK, LGTM. > Pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=995f0f983386e2a82dbce65d4366ee7f58f59138 -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] news: Support News-Item-Format 2.0
On 09/02/2016 01:49 PM, Mike Gilbert wrote:
> Display-If-Installed already supported EAPI 5 atoms.
> Use fnmatch for wildcard support on Display-If-Profile.
> ---
> pym/portage/news.py | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/pym/portage/news.py b/pym/portage/news.py
> index 177f9db..d6d2f07 100644
> --- a/pym/portage/news.py
> +++ b/pym/portage/news.py
> @@ -271,7 +271,7 @@ class NewsItem(object):
> # will never match
> format_match = _formatRE.match(line)
> if (format_match is not None and
> - not
> fnmatch.fnmatch(format_match.group(1), '1.*')):
> + not
> fnmatch.fnmatch(format_match.group(1), '[12].*')):
> invalids.append((i + 1, line.rstrip('\n')))
> break
> if not line.startswith('D'):
> @@ -325,7 +325,7 @@ class DisplayProfileRestriction(DisplayRestriction):
> self.profile = profile
>
> def checkRestriction(self, **kwargs):
> - if self.profile == kwargs['profile']:
> + if fnmatch.fnmatch(kwargs['profile'], self.profile):
> return True
> return False
>
>
Looks good.
If we wanted to be more strict about the input that we accept, we could
limit the profile wildcard match to so that it only works if the format
is 2.* and only supports a terminal /* since that's all that the spec
says is supported.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v3] news: Support News-Item-Format 2.0
On 09/04/2016 10:04 AM, Mike Gilbert wrote:
> Validate Display-If-Installed with EAPI 0 or 5.
> Add support for trailing wildcard matching for Display-If-Profile.
>
> Bug: https://bugs.gentoo.org/577372
> ---
> pym/portage/news.py | 42 ++
> 1 file changed, 30 insertions(+), 12 deletions(-)
>
> diff --git a/pym/portage/news.py b/pym/portage/news.py
> index 177f9db..fa6fb00 100644
> --- a/pym/portage/news.py
> +++ b/pym/portage/news.py
> @@ -266,14 +266,24 @@ class NewsItem(object):
> f.close()
> self.restrictions = {}
> invalids = []
> + news_format = None
> +
> + # Look for News-Item-Format
> for i, line in enumerate(lines):
> - # Optimization to ignore regex matchines on lines that
> - # will never match
> format_match = _formatRE.match(line)
> - if (format_match is not None and
> - not
> fnmatch.fnmatch(format_match.group(1), '1.*')):
> + if format_match is not None:
> + news_format = format_match.group(1)
> + if fnmatch.fnmatch(news_format, '[12].*'):
> + break
> invalids.append((i + 1, line.rstrip('\n')))
> - break
> +
> + if news_format is None:
> + invalids.append((0, 'News-Item-Format unspecified'))
> +
> + # Parse the rest
> + for i, line in enumerate(lines):
> + # Optimization to ignore regex matchines on lines that
s/matchines/matches/
> + # will never match
> if not line.startswith('D'):
> continue
> restricts = { _installedRE :
> DisplayInstalledRestriction,
> @@ -282,13 +292,14 @@ class NewsItem(object):
> for regex, restriction in restricts.items():
> match = regex.match(line)
> if match:
> - restrict =
> restriction(match.groups()[0].strip())
> + restrict =
> restriction(match.groups()[0].strip(), news_format)
> if not restrict.isValid():
> invalids.append((i + 1,
> line.rstrip("\n")))
> else:
> self.restrictions.setdefault(
> id(restriction),
> []).append(restrict)
> continue
> +
> if invalids:
> self._valid = False
> msg = []
> @@ -321,13 +332,14 @@ class DisplayProfileRestriction(DisplayRestriction):
> if the user is running a specific profile.
> """
>
> - def __init__(self, profile):
> + def __init__(self, profile, news_format):
> self.profile = profile
> + self.format = news_format
>
> def checkRestriction(self, **kwargs):
> - if self.profile == kwargs['profile']:
> - return True
> - return False
> + if fnmatch.fnmatch(self.format, '2.*') and
> self.profile.endswith('/*'):
> + return (kwargs['profile'].startswith(self.profile[:-1]))
Maybe we should raise an exception if a wildcard is used but the format
doesn't allow it? We could also raise an exception for any unsupported
wildcards that do not occur at the end of the string.
> + return (kwargs['profile'] == self.profile)
>
> class DisplayKeywordRestriction(DisplayRestriction):
> """
> @@ -335,8 +347,9 @@ class DisplayKeywordRestriction(DisplayRestriction):
> if the user is running a specific keyword.
> """
>
> - def __init__(self, keyword):
> + def __init__(self, keyword, news_format):
> self.keyword = keyword
> + self.format = news_format
>
> def checkRestriction(self, **kwargs):
> if kwargs['config'].get('ARCH', '') == self.keyword:
> @@ -349,10 +362,15 @@ class DisplayInstalledRestriction(DisplayRestriction):
> if the user has that item installed.
> """
>
> - def __init__(self, atom):
> + def __init__(self, atom, news_format):
> self.atom = atom
> + self.format = news_format
>
> def isValid(self):
> + if fnmatch.fnmatch(self.format, '1.*'):
> + return isvalidatom(self.atom, eapi='0')
We might want to check the existing news items to make sure that they
are all conformant here.
> + if fnmatch.fnmatch(self.format, '2.*'):
> + return isvalidatom(self.a
Re: [gentoo-portage-dev] [PATCH v4] news: Support News-Item-Format 2.0
On 09/07/2016 02:03 PM, Mike Gilbert wrote:
> Validate Display-If-Installed with EAPI 0 or 5.
> Add support for trailing wildcard matching for Display-If-Profile.
>
> Bug: https://bugs.gentoo.org/577372
> ---
> pym/portage/news.py | 50 ++
> 1 file changed, 38 insertions(+), 12 deletions(-)
>
> diff --git a/pym/portage/news.py b/pym/portage/news.py
> index 177f9db..e53e905 100644
> --- a/pym/portage/news.py
> +++ b/pym/portage/news.py
> @@ -197,6 +197,7 @@ _formatRE = re.compile("News-Item-Format:\s*([^\s]*)\s*$")
> _installedRE = re.compile("Display-If-Installed:(.*)\n")
> _profileRE = re.compile("Display-If-Profile:(.*)\n")
> _keywordRE = re.compile("Display-If-Keyword:(.*)\n")
> +_bad_wc_RE = re.compile(r'.*([^/]\*|\*.)')
Maybe it's better if we use a regex that defines a valid profile string:
_valid_profile_RE = re.compile(r'^[^*]+(/\*)?$')
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v5] news: Support News-Item-Format 2.0
On 09/07/2016 10:31 PM, Mike Gilbert wrote: > Validate Display-If-Installed with EAPI 0 or 5. > Add support for trailing wildcard matching for Display-If-Profile. > > Bug: https://bugs.gentoo.org/577372 > --- > pym/portage/news.py | 50 ++ > 1 file changed, 38 insertions(+), 12 deletions(-) LGTM, thanks for addressing all of my concerns. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 3/3] news: skip parsing if News-Item-Format is unspecified
Merged: https://gitweb.gentoo.org/proj/portage.git/commit/?id=0cb7426796b2fb082f9176b7b08d08b21b5c86bb https://gitweb.gentoo.org/proj/portage.git/commit/?id=67fcbccd1b60b599e4e5dcc97f2959164ba6a7eb https://gitweb.gentoo.org/proj/portage.git/commit/?id=216297a9a7e91e48b74ef796b6642fd8c15085fc -- Thanks, Zac
Re: [gentoo-portage-dev] [Patch] 1/1 New commitmsgfile "cat/pkg: " auto-substitution
On 09/15/2016 12:50 PM, Brian Dolbec wrote:
>
> From 08f6d22b366c8dc2968573ebc4fd4f0401ea67b7 Mon Sep 17 00:00:00 2001
> From: Brian Dolbec
> Date: Thu, 15 Sep 2016 12:45:10 -0700
> Subject: [PATCH] repoman: Update man page and commitmsgfile option
>
> Use .lower() on the leading message 9 charachters of hte message text for the
> substitution
> test.
> Update the man page for this new templating capability.
> ---
> repoman/man/repoman.1 | 6 +-
> repoman/pym/repoman/actions.py | 2 +-
> 2 files changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/repoman/man/repoman.1 b/repoman/man/repoman.1
> index 4a9122e..8df3207 100644
> --- a/repoman/man/repoman.1
> +++ b/repoman/man/repoman.1
> @@ -109,7 +109,11 @@ Behave as if no package.mask entries exist (not allowed
> with commit mode)
> Adds a commit message via the command line
> .TP
> \fB-M\fR, \fB--commitmsgfile\fR
> -Adds a commit message from the specified file
> +Adds a commit message from the specified file. This option also will perform
> +an automatic text substitution of a leading "cat/pkg: " string (upper or
> lower
> +case) with the actual category/package prefix as defined by the required
> message
> +format. Use this option for templating a common commit message for
> multiple +package updates.
> .TP
> \fB-V\fR, \fB--version\fR
> Show version info
> diff --git a/repoman/pym/repoman/actions.py
> b/repoman/pym/repoman/actions.py index 0534c29..9763c26 100644
> --- a/repoman/pym/repoman/actions.py
> +++ b/repoman/pym/repoman/actions.py
> @@ -108,7 +108,7 @@ class Actions(object):
> "
> --commitmsgfile='%s'\n" % self.options.commitmsgfile) else:
> raise
> - if commitmessage[:9] in ("cat/pkg: ",
> "CAT/PKG: "):
> + if commitmessage[:9].lower() in ("cat/pkg: "):
This is a string-in-string containment check unless you add a comma to
make it a tuple like ("cat/pkg: ",).
> commitmessage = self.msg_prefix() +
> commitmessage[9:]
> if not commitmessage or not commitmessage.strip():
>
--
Thanks,
Zac
Re: [gentoo-portage-dev] [Patch] 1/1 New commitmsgfile "cat/pkg: " auto-substitution
On 09/15/2016 03:27 PM, Brian Dolbec wrote:
> On Thu, 15 Sep 2016 14:47:31 -0700
> Zac Medico wrote:
>
>> On 09/15/2016 12:50 PM, Brian Dolbec wrote:
>>>
>>> From 08f6d22b366c8dc2968573ebc4fd4f0401ea67b7 Mon Sep 17 00:00:00
>>> 2001 From: Brian Dolbec
>>> Date: Thu, 15 Sep 2016 12:45:10 -0700
>>> Subject: [PATCH] repoman: Update man page and commitmsgfile option
>>>
>>> Use .lower() on the leading message 9 charachters of hte message
>>> text for the substitution test.
>>> Update the man page for this new templating capability.
>>> ---
>>> repoman/man/repoman.1 | 6 +-
>>> repoman/pym/repoman/actions.py | 2 +-
>>> 2 files changed, 6 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/repoman/man/repoman.1 b/repoman/man/repoman.1
>>> index 4a9122e..8df3207 100644
>>> --- a/repoman/man/repoman.1
>>> +++ b/repoman/man/repoman.1
>>> @@ -109,7 +109,11 @@ Behave as if no package.mask entries exist
>>> (not allowed with commit mode) Adds a commit message via the
>>> command line .TP
>>> \fB-M\fR, \fB--commitmsgfile\fR
>>> -Adds a commit message from the specified file
>>> +Adds a commit message from the specified file. This option also
>>> will perform +an automatic text substitution of a leading "cat/pkg:
>>> " string (upper or lower +case) with the actual category/package
>>> prefix as defined by the required message +format. Use this option
>>> for templating a common commit message for multiple +package
>>> updates. .TP
>>> \fB-V\fR, \fB--version\fR
>>> Show version info
>>> diff --git a/repoman/pym/repoman/actions.py
>>> b/repoman/pym/repoman/actions.py index 0534c29..9763c26 100644
>>> --- a/repoman/pym/repoman/actions.py
>>> +++ b/repoman/pym/repoman/actions.py
>>> @@ -108,7 +108,7 @@ class Actions(object):
>>> "
>>> --commitmsgfile='%s'\n" % self.options.commitmsgfile) else:
>>> raise
>>> - if commitmessage[:9] in ("cat/pkg: ",
>>> "CAT/PKG: "):
>>> + if commitmessage[:9].lower() in ("cat/pkg:
>>> "):
>>
>> This is a string-in-string containment check unless you add a comma to
>> make it a tuple like ("cat/pkg: ",).
>>
>
> Doh, I removed that comma without realizing, thanks
Otherwise, looks good.
--
Thanks,
Zac
Re: [gentoo-portage-dev] [PATCH v2] AbstractEbuildProcess: disable ipc_daemon under Windows Subsystem for Linux
On Thu, Sep 22, 2016 at 8:23 PM, Kerin Millar wrote: > Duly updated to use any instead of ==, as recommended by Brian Dolbec. > > -- > Kerin Millar My first choice would be to use a small test case to detect when ipc is broken, and disable it dynamically. A good example of such a test is the can_poll_device function here: https://gitweb.gentoo.org/proj/portage.git/tree/pym/portage/util/_eventloop/EventLoop.py?h=portage-2.3.1#n597 If it's not possible to use a test similar to the above, maybe it's best to use /proc/version or /proc/sys/kernel/osrelease as mentioned in the following issue: https://github.com/Microsoft/BashOnWindows/issues/423 Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] ebuild: look for ccache symlinks in /usr/lib/ccache/bin
On Fri, Sep 23, 2016 at 7:55 PM, Mike Gilbert wrote:
> ccache-config has been installing the symlinks there since Nov 2014.
>
> Bug: https://bugs.gentoo.org/509956
> Bug: https://bugs.gentoo.org/594982
> ---
> pym/portage/package/ebuild/prepare_build_dirs.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/pym/portage/package/ebuild/prepare_build_dirs.py
> b/pym/portage/package/ebuild/prepare_build_dirs.py
> index 7fdac74..808e77d 100644
> --- a/pym/portage/package/ebuild/prepare_build_dirs.py
> +++ b/pym/portage/package/ebuild/prepare_build_dirs.py
> @@ -154,7 +154,7 @@ def _prepare_features_dirs(mysettings):
>
> features_dirs = {
> "ccache":{
> - "path_dir": "%s/usr/%s/ccache/bin" % (EPREFIX,
> libdir),
> + "path_dir": "%s/usr/lib/ccache/bin" % EPREFIX,
> "basedir_var":"CCACHE_DIR",
>
> "default_dir":os.path.join(mysettings["PORTAGE_TMPDIR"], "ccache"),
> "always_recurse":False},
> --
> 2.10.0
The /usr/lib/ccache/bin code lives in doebuild_environment now:
https://gitweb.gentoo.org/proj/portage.git/commit/?id=96338b8a2add3ab5f7f77fef41570807374771e3
[gentoo-portage-dev] [PATCH] ebuild.sh: start phases in temporary HOME if available (bug 595028)
This will avoid undesirable interactions with the python sitedir,
as reported in bug 574002. Since the temporary HOME is not guaranteed
to exist for some phases, use PORTAGE_PYM_PATH as a fallback. Also,
use PORTAGE_PYM_PATH inside __dyn_clean, since HOME is removed there.
X-Gentoo-Bug: 595028
X-Gentoo-Bug-URL: https://bugs.gentoo.org/595028
---
bin/ebuild.sh | 11 ---
bin/phase-functions.sh | 8 +---
2 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 5b3146d..4a431b4 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -167,9 +167,14 @@ export SANDBOX_ON=0
# Ensure that $PWD is sane whenever possible, to protect against
# exploitation of insecure search path for python -c in ebuilds.
-# See bug #239560 and bug #469338.
-cd "${PORTAGE_PYM_PATH}" || \
- die "PORTAGE_PYM_PATH does not exist: '${PORTAGE_PYM_PATH}'"
+# See bug #239560, bug #469338, and bug #595028.
+if [[ -d ${HOME} ]] then
+ # Use portage's temporary HOME directory if available.
+ cd "${HOME}" || die
+else
+ cd "${PORTAGE_PYM_PATH}" || \
+ die "PORTAGE_PYM_PATH does not exist: '${PORTAGE_PYM_PATH}'"
+fi
#if no perms are specified, dirs/files will have decent defaults
#(not secretive, but not stupid)
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index d75f43f..50f0fdb 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -261,6 +261,11 @@ __dyn_clean() {
chflags -R nosunlnk,nouunlnk "${PORTAGE_BUILDDIR}" 2>/dev/null
fi
+ # Some kernels, such as Solaris, return EINVAL when an attempt
+ # is made to remove the current working directory.
+ cd "${PORTAGE_PYM_PATH}" || \
+ die "PORTAGE_PYM_PATH does not exist: '${PORTAGE_PYM_PATH}'"
+
rm -rf "${PORTAGE_BUILDDIR}/image" "${PORTAGE_BUILDDIR}/homedir"
rm -f "${PORTAGE_BUILDDIR}/.installed"
@@ -288,9 +293,6 @@ __dyn_clean() {
# result in it wiping the users distfiles directory (bad).
rm -rf "${PORTAGE_BUILDDIR}/distdir"
- # Some kernels, such as Solaris, return EINVAL when an attempt
- # is made to remove the current working directory.
- cd "$PORTAGE_BUILDDIR"/../..
rmdir "$PORTAGE_BUILDDIR" 2>/dev/null
true
--
2.7.4
Re: [gentoo-portage-dev] [PATCH v2] AbstractEbuildProcess: disable ipc_daemon under Windows Subsystem for Linux
On Thu, Sep 22, 2016 at 8:48 PM, Zac Medico wrote: > On Thu, Sep 22, 2016 at 8:23 PM, Kerin Millar wrote: >> Duly updated to use any instead of ==, as recommended by Brian Dolbec. >> >> -- >> Kerin Millar > > My first choice would be to use a small test case to detect when ipc > is broken, and disable it dynamically. A good example of such a test > is the can_poll_device function here: > > https://gitweb.gentoo.org/proj/portage.git/tree/pym/portage/util/_eventloop/EventLoop.py?h=portage-2.3.1#n597 > > If it's not possible to use a test similar to the above, maybe it's > best to use /proc/version or /proc/sys/kernel/osrelease as mentioned > in the following issue: > > https://github.com/Microsoft/BashOnWindows/issues/423 > > Thanks, > Zac I've started playing around with WSL, and I've discovered that portage's ipc actually works if we use fcntl.flock instead of fcntl.lockf!!! Simply set _default_lock_fn = fcntl.flock in pym/portage/locks.py, and watch the tests succeed: $ pym/portage/tests/runTests.py pym/portage/tests/ebuild/test_ipc_daemon.py testIpcDaemon (portage.tests.ebuild.test_ipc_daemon.IpcDaemonTestCase) ... ok -- Ran 1 test in 1.282s OK So, I'm thinking that we should add a dynamic test inside pym/portage/locks.py which checks for a broken fcntl.lockf, and falls back to fcntl.flock in that case. Thanks, Zac
[gentoo-portage-dev] [PATCH] locks: use fcntl.flock if fcntl.lockf is broken (bug 595146)
This is needed for Windows Subsystem for Linux (WSL), as well as older versions of PyPy. X-Gentoo-bug: 595146 X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=595146 --- pym/portage/locks.py | 59 +--- 1 file changed, 52 insertions(+), 7 deletions(-) diff --git a/pym/portage/locks.py b/pym/portage/locks.py index 42ff1e3..a4564cf 100644 --- a/pym/portage/locks.py +++ b/pym/portage/locks.py @@ -8,8 +8,9 @@ __all__ = ["lockdir", "unlockdir", "lockfile", "unlockfile", \ import errno import fcntl -import platform +import multiprocessing import sys +import tempfile import time import warnings @@ -27,17 +28,61 @@ if sys.hexversion >= 0x300: HARDLINK_FD = -2 _HARDLINK_POLL_LATENCY = 3 # seconds -_default_lock_fn = fcntl.lockf - -if platform.python_implementation() == 'PyPy': - # workaround for https://bugs.pypy.org/issue747 - _default_lock_fn = fcntl.flock # Used by emerge in order to disable the "waiting for lock" message # so that it doesn't interfere with the status display. _quiet = False +_lock_fn = None + + +def _get_lock_fn(): + """ + Returns fcntl.lockf if proven to work, and otherwise returns fcntl.flock. + On some platforms fcntl.lockf is known to be broken. + """ + global _lock_fn + if _lock_fn is not None: + return _lock_fn + + def _test_lock(fd, lock_path): + os.close(fd) + try: + with open(lock_path, 'a') as f: + fcntl.lockf(f.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB) + except EnvironmentError as e: + if e.errno == errno.EAGAIN: + # Parent process holds lock, as expected. + sys.exit(0) + + # Something went wrong. + sys.exit(1) + + fd, lock_path = tempfile.mkstemp() + try: + try: + fcntl.lockf(fd, fcntl.LOCK_EX) + except EnvironmentError: + pass + else: + proc = multiprocessing.Process(target=_test_lock, + args=(fd, lock_path)) + proc.start() + proc.join() + if proc.exitcode == os.EX_OK: + # Use fcntl.lockf becase the test passed. + _lock_fn = fcntl.lockf + return _lock_fn + finally: + os.close(fd) + os.unlink(lock_path) + + # Fall back to fcntl.flock. + _lock_fn = fcntl.flock + return _lock_fn + + _open_fds = set() def _close_fds(): @@ -146,7 +191,7 @@ def lockfile(mypath, wantnewlockfile=0, unlinkfile=0, # try for a non-blocking lock, if it's held, throw a message # we're waiting on lockfile and use a blocking attempt. - locking_method = portage._eintr_func_wrapper(_default_lock_fn) + locking_method = portage._eintr_func_wrapper(_get_lock_fn()) try: if "__PORTAGE_TEST_HARDLINK_LOCKS" in os.environ: raise IOError(errno.ENOSYS, "Function not implemented") -- 2.7.4
Re: [gentoo-portage-dev] [PATCH] ebuild.sh: start phases in temporary HOME if available (bug 595028)
On Mon, Sep 26, 2016 at 8:49 AM, Alexander Berntsen wrote: > No real opinion -- not familiar enough with this to suggest alternative > solutions. Patch itself looks OK. > > -- > Alexander > berna...@gentoo.org > https://secure.plaimi.net/~alexander > Pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=120deae6d7e63a35fc068aa92ff2e4263c98e5b6 Thanks, Zac
Re: [gentoo-portage-dev] [PATCH v2] AbstractEbuildProcess: disable ipc_daemon under Windows Subsystem for Linux
On Mon, Sep 26, 2016 at 3:58 PM, Kerin Millar wrote: > On Sun, 25 Sep 2016 22:49:59 -0700 > Zac Medico wrote: > >> On Thu, Sep 22, 2016 at 8:48 PM, Zac Medico wrote: >> > On Thu, Sep 22, 2016 at 8:23 PM, Kerin Millar wrote: >> >> Duly updated to use any instead of ==, as recommended by Brian Dolbec. >> >> >> >> -- >> >> Kerin Millar >> > >> > My first choice would be to use a small test case to detect when ipc >> > is broken, and disable it dynamically. A good example of such a test >> > is the can_poll_device function here: >> > >> > https://gitweb.gentoo.org/proj/portage.git/tree/pym/portage/util/_eventloop/EventLoop.py?h=portage-2.3.1#n597 >> > >> > If it's not possible to use a test similar to the above, maybe it's >> > best to use /proc/version or /proc/sys/kernel/osrelease as mentioned >> > in the following issue: >> > >> > https://github.com/Microsoft/BashOnWindows/issues/423 >> > >> > Thanks, >> > Zac >> >> I've started playing around with WSL, and I've discovered that >> portage's ipc actually works if we use fcntl.flock instead of >> fcntl.lockf!!! Simply set _default_lock_fn = fcntl.flock in >> pym/portage/locks.py, and watch the tests succeed: >> >> $ pym/portage/tests/runTests.py pym/portage/tests/ebuild/test_ipc_daemon.py >> testIpcDaemon (portage.tests.ebuild.test_ipc_daemon.IpcDaemonTestCase) ... ok >> >> -- >> Ran 1 test in 1.282s >> >> OK > > How strange! While the ebuild-ipc helper waits for a response, it uses non-blocking lock calls to poll for liveliness, so it's critical that the locking api be usable here. We've seen a similar issue in the past with PyPy, where fcntl.lockf was broken while fcntl.flock worked just fine. Thanks, Zac
[gentoo-portage-dev] Re: [PATCH] locks: use fcntl.flock if fcntl.lockf is broken (bug 595146)
On 09/25/2016 11:44 PM, Zac Medico wrote: > This is needed for Windows Subsystem for Linux (WSL), as well as > older versions of PyPy. > > X-Gentoo-bug: 595146 > X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=595146 > --- > pym/portage/locks.py | 59 > +--- > 1 file changed, 52 insertions(+), 7 deletions(-) Any feedback on this? I think we should merge it, for inclusion in portage-2.3.2. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] setup.py: enable libc bindings optionally (bug 594744)
Since the libc bindings don't support cross-compilation, add
an option to enable them conditially. The ebuild can use this
to enable the libc bindings by adding the following code to
the python_prepare_all function:
if use native-extensions; then
printf "[build_ext]\nportage-ext-modules=true" >> \
setup.cfg || die
fi
X-Gentoo-Bug: 594744
X-Gentoo-Bug-URL: https://bugs.gentoo.org/594744
---
.travis.yml | 1 +
setup.py| 20
2 files changed, 21 insertions(+)
diff --git a/.travis.yml b/.travis.yml
index c098c4d..ded5893 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -10,6 +10,7 @@ python:
install: "pip install lxml"
script:
+- printf "[build_ext]\nportage-ext-modules=true" >> setup.cfg
- ./setup.py test
- ./setup.py install --root=/tmp/install-root
# prevent repoman tests from trying to fetch metadata.xsd
diff --git a/setup.py b/setup.py
index e900aaa..e1d21a0 100755
--- a/setup.py
+++ b/setup.py
@@ -6,6 +6,7 @@ from __future__ import print_function
from distutils.core import setup, Command, Extension
from distutils.command.build import build
+from distutils.command.build_ext import build_ext as _build_ext
from distutils.command.build_scripts import build_scripts
from distutils.command.clean import clean
from distutils.command.install import install
@@ -624,6 +625,24 @@ def get_manpages():
yield [os.path.join('$mandir', topdir, 'man%s'
% g), mans]
+class build_ext(_build_ext):
+ user_options = _build_ext.user_options + [
+('portage-ext-modules', None,
+ "enable portage's C/C++ extensions (cross-compling is not
supported)"),
+ ]
+
+ boolean_options = _build_ext.boolean_options + [
+ 'portage-ext-modules'
+ ]
+
+ def initialize_options(self):
+ _build_ext.initialize_options(self)
+ self.portage_ext_modules = None
+
+ def run(self):
+ if self.portage_ext_modules:
+ _build_ext.run(self)
+
setup(
name = 'portage',
version = '2.3.1',
@@ -651,6 +670,7 @@ setup(
cmdclass = {
'build': x_build,
+ 'build_ext': build_ext,
'build_man': build_man,
'build_scripts': x_build_scripts,
'build_scripts_bin': x_build_scripts_bin,
--
2.7.4
Re: [gentoo-portage-dev] Re: [PATCH] locks: use fcntl.flock if fcntl.lockf is broken (bug 595146)
On 10/01/2016 09:30 PM, Brian Dolbec wrote: > On Sat, 1 Oct 2016 17:08:46 -0700 > Zac Medico wrote: > >> On 09/25/2016 11:44 PM, Zac Medico wrote: >>> This is needed for Windows Subsystem for Linux (WSL), as well as >>> older versions of PyPy. >>> >>> X-Gentoo-bug: 595146 >>> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=595146 >>> --- >>> pym/portage/locks.py | 59 >>> +--- 1 file >>> changed, 52 insertions(+), 7 deletions(-) >> >> Any feedback on this? I think we should merge it, for inclusion in >> portage-2.3.2. > > > Sorry, been out all day, and again tomorrow... > > Yeah, looks fine Thanks, merged: https://gitweb.gentoo.org/proj/portage.git/commit/?id=5ef5fbaab88de47d4dbab333661d3525261d7633 -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] setup.py: enable libc bindings optionally (bug 594744)
On 10/05/2016 06:11 AM, Alexander Berntsen wrote: > On 02/10/16 05:03, Zac Medico wrote: >> The ebuild can use this > Shouldn't "this" be documented somewhere more useful than in the commit msg? The new --portage-ext-modules option shows up int the output of `./setup.py build_ext --help`. I suppose the DEVELOPING document will be a good place to document it, as well as here: https://wiki.gentoo.org/wiki/Project:Portage#Testing_Portage I have not even tested building and using the libc bindings directly from a checkout with PYTHONPATH and PATH overrides, so I'll have to try that. > Also, the msg should mention that the libc bindings are optional in the > first place. Okay, will do. -- Thanks, Zac
[gentoo-portage-dev] [PATCH v2] setup.py: enable libc bindings optionally (bug 594744)
The libc bindings are optional, since ctypes is used as a fallback when
they are not available. The libc bindings do not support cross-
compilation, therefore it is useful to be able to build them
conditionally. This patch adds an option to enable them conditionally,
which the ebuild can use by adding the following code to the
python_prepare_all function:
if use native-extensions; then
printf "[build_ext]\nportage-ext-modules=true" >> \
setup.cfg || die
fi
X-Gentoo-Bug: 594744
X-Gentoo-Bug-URL: https://bugs.gentoo.org/594744
---
[PATCH v2] adds a "Native Extensions" section to the README file, and
updates the commit message to note that the libc bindings are optional
because ctypes is used as a fallback.
.travis.yml | 1 +
README | 19 +++
setup.py| 21 +
3 files changed, 41 insertions(+)
diff --git a/.travis.yml b/.travis.yml
index c098c4d..ded5893 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -10,6 +10,7 @@ python:
install: "pip install lxml"
script:
+- printf "[build_ext]\nportage-ext-modules=true" >> setup.cfg
- ./setup.py test
- ./setup.py install --root=/tmp/install-root
# prevent repoman tests from trying to fetch metadata.xsd
diff --git a/README b/README
index 5e78842..311d036 100644
--- a/README
+++ b/README
@@ -13,6 +13,25 @@ Dependencies
Python and Bash should be the only hard dependencies. Python 2.7 is the
minimum supported version.
+Native Extensions
+=
+
+Portage includes some optional native extensions which can be built
+in the source tree by running the following command:
+
+python setup.py build_ext --inplace --portage-ext-modules
+
+The following setup.cfg settings can be used to enable building of
+native extensions for all invocations of the build_ext command (the
+build_ext command is invoked automatically by other build commands):
+
+ [build_ext]
+ portage-ext-modules=true
+
+Currently, the native extensions only include libc bindings which are
+used to validate LC_CTYPE and LC_COLLATE behavior for EAPI 6. If the
+native extensions have not been built, then portage will use ctypes
+instead.
Licensing and Legalese
===
diff --git a/setup.py b/setup.py
index e900aaa..e2c6b6e 100755
--- a/setup.py
+++ b/setup.py
@@ -6,6 +6,7 @@ from __future__ import print_function
from distutils.core import setup, Command, Extension
from distutils.command.build import build
+from distutils.command.build_ext import build_ext as _build_ext
from distutils.command.build_scripts import build_scripts
from distutils.command.clean import clean
from distutils.command.install import install
@@ -624,6 +625,25 @@ def get_manpages():
yield [os.path.join('$mandir', topdir, 'man%s'
% g), mans]
+class build_ext(_build_ext):
+ user_options = _build_ext.user_options + [
+ ('portage-ext-modules', None,
+"enable portage's C/C++ extensions (cross-compling is not
supported)"),
+ ]
+
+ boolean_options = _build_ext.boolean_options + [
+ 'portage-ext-modules',
+ ]
+
+ def initialize_options(self):
+ _build_ext.initialize_options(self)
+ self.portage_ext_modules = None
+
+ def run(self):
+ if self.portage_ext_modules:
+ _build_ext.run(self)
+
+
setup(
name = 'portage',
version = '2.3.1',
@@ -651,6 +671,7 @@ setup(
cmdclass = {
'build': x_build,
+ 'build_ext': build_ext,
'build_man': build_man,
'build_scripts': x_build_scripts,
'build_scripts_bin': x_build_scripts_bin,
--
2.7.4
Re: [gentoo-portage-dev] [PATCH v2] setup.py: enable libc bindings optionally (bug 594744)
On 10/06/2016 03:51 AM, Alexander Berntsen wrote: > LGTM. > Thanks, pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=c6b72853a6de8b71d1b5cd789a12aa934f7ce7c4 -- Thanks, Zac
Re: [gentoo-portage-dev] Leadership election
On 10/07/2016 08:09 AM, Alexander Berntsen wrote: > Friends, > > It's that time of year. I have been lead developer for a year, so we > need to have another election. > > I've not been very good, nor attentive enough. So if you want to elect > someone else than me, I can understand that. If nobody else wants the > role, I can probably continue to reject, and even sometimes accept, > Zac's patches too; I don't mind. > > I suggest we just nominate and vote in this thread. That's what we did > last time around. Unless someone has something against this, let's > just go ahead. > > On to nominations: I nominate Zac and Brian. > > Let's do voting next week, and pick someone next Friday, unless anyone > objects to this. > I nominate you and Brian. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] Leadership election
On 10/11/2016 12:06 AM, Alexander Berntsen wrote: > Right, voting. I vote for... Brian? Yeah. Brian! > I vote for Brian too. Brian! :) -- Thanks, Zac signature.asc Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH] emerge-webrsync: use gkeys to verify gpg signatures (bug 597918)
Use gkeys to verify gpg signatures by default. Refresh the gentoo
snapshot signing key before signature verification, in order to ensure
that the latest revocation data is available. Add an --insecure option
which disables gpg signature verification. Warn about man-in-the-middle
attacks when the --insecure option is used. Deprecate the pre-existing
webrsync-gpg feature since it requires manual gpg configuration.
X-Gentoo-Bug: 597918
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=597918
---
bin/emerge-webrsync | 51 +++
man/make.conf.5 | 6 --
2 files changed, 51 insertions(+), 6 deletions(-)
diff --git a/bin/emerge-webrsync b/bin/emerge-webrsync
index 9961ad8..84609e0 100755
--- a/bin/emerge-webrsync
+++ b/bin/emerge-webrsync
@@ -66,13 +66,24 @@ fi
do_verbose=0
do_debug=0
keep=false
+insecure=false
+
+insecure_bypass_msg() {
+ wecho "The --insecure option can be used to bypass this step."
+ insecure_warning_msg
+}
+
+insecure_warning_msg() {
+ wecho "The --insecure option prevents detection of"
+ wecho "man-in-the-middle attacks!"
+}
if has webrsync-gpg ${FEATURES} ; then
- WEBSYNC_VERIFY_SIGNATURE=1
+ VERIFY_SIGNATURE_LEGACY_MODE=1
else
- WEBSYNC_VERIFY_SIGNATURE=0
+ VERIFY_SIGNATURE_LEGACY_MODE=0
fi
-if [ ${WEBSYNC_VERIFY_SIGNATURE} != 0 -a -z "${PORTAGE_GPG_DIR}" ]; then
+if [ ${VERIFY_SIGNATURE_LEGACY_MODE} != 0 -a -z "${PORTAGE_GPG_DIR}" ]; then
eecho "please set PORTAGE_GPG_DIR in make.conf"
exit 1
fi
@@ -176,7 +187,7 @@ check_file_signature() {
local file="$2"
local r=1
- if [ ${WEBSYNC_VERIFY_SIGNATURE} != 0 ]; then
+ if [ ${VERIFY_SIGNATURE_LEGACY_MODE} != 0 ]; then
__vecho "Checking signature ..."
@@ -186,6 +197,17 @@ check_file_signature() {
eecho "cannot check signature: gpg binary not found"
exit 1
fi
+ elif ! ${insecure}; then
+ __vecho "Checking signature ..."
+
+ # gkeys requires that the signature file be in the same
directory
+ # as the snapshot
+ if [[ ${signature} != ${file}.gpgsig ]]; then
+ # this should not happen
+ eecho "assertion failed: ${signature} != ${file}.gpgsig"
+ exit 1
+ fi
+ gkeys verify -C gentoo -n snapshot -F "${file}" && r=0
else
r=0
fi
@@ -445,6 +467,7 @@ usage() {
Options:
--revert=mmdd Revert to snapshot
+ --insecure Disable gpg signature verification
-k, --keep Keep snapshots in DISTDIR (don't delete)
-q, --quiet Only output errors
-v, --verbose Enable verbose output
@@ -467,6 +490,7 @@ main() {
local v=${arg#*=}
case ${arg} in
-h|--help)usage ;;
+ --insecure) insecure=true ;;
-k|--keep)keep=true ;;
-q|--quiet) PORTAGE_QUIET=1 ;;
-v|--verbose) do_verbose=1 ;;
@@ -512,6 +536,25 @@ main() {
exit 1
fi
+ if ${insecure}; then
+ insecure_warning_msg
+ elif [[ ${VERIFY_SIGNATURE_LEGACY_MODE} == 1 ]]; then
+ wecho "FEATURES=webrsync-gpg is deprecated."
+ wecho "By default, the new gkeys verification mode will be used"
+ wecho "when FEATURES=webrsync-gpg is not enabled."
+ else
+ if ! type -P gkeys >/dev/null; then
+ eecho "gkeys: command not found"
+ eecho "Please try again after installing gkeys: emerge
app-crypt/gkeys"
+ insecure_bypass_msg
+ exit 1
+ elif ! gkeys refresh-key -C gentoo -n snapshot; then
+ eecho "gkeys refresh-key failed"
+ insecure_bypass_msg
+ exit 1
+ fi
+ fi
+
[[ ${do_debug} -eq 1 ]] && set -x
if [[ -n ${revert_date} ]] ; then
diff --git a/man/make.conf.5 b/man/make.conf.5
index aea189e..5b809ed 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -1,4 +1,4 @@
-.TH "MAKE.CONF" "5" "Feb 2016" "Portage VERSION" "Portage"
+.TH "MAKE.CONF" "5" "Oct 2016" "Portage VERSION" "Portage"
.SH "NAME"
make.conf \- custom settings for Portage
.SH "SYNOPSIS"
@@ -663,7 +663,9 @@ Portage would have to waste time validating ownership for
each and every sync
operation.
.TP
.B webrsync-gpg
-Enable GPG verification when using \fIemerge\-webrsync\fR.
+Enable legacy GPG verification mode when using \fIemerge\-webrsync\fR.
+This feature is deprecated. By default, the new \fBgkeys\fR(1) verification
+mode will be used when this feature is not enabled.
.TP
.B
Re: [gentoo-portage-dev] [PATCH] emerge-webrsync: use gkeys to verify gpg signatures (bug 597918)
On 10/27/2016 11:09 AM, Alexander Berntsen wrote: > On 27/10/16 19:16, Zac Medico wrote: >> Use gkeys to verify gpg signatures by default. Refresh the gentoo >> snapshot signing key before signature verification, in order to >> ensure that the latest revocation data is available. Add an >> --insecure option which disables gpg signature verification. Warn >> about man-in-the-middle attacks when the --insecure option is used. >> Deprecate the pre-existing webrsync-gpg feature since it requires >> manual gpg configuration. > %s/ gpg/ OpenPGP/ > Thanks, fixed. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] emerge-webrsync: use gkeys to verify gpg signatures (bug 597918)
On 10/27/2016 10:38 AM, Brian Dolbec wrote: > On Thu, 27 Oct 2016 10:16:42 -0700 > Zac Medico wrote: > >> Use gkeys to verify gpg signatures by default. Refresh the gentoo >> snapshot signing key before signature verification, in order to ensure >> that the latest revocation data is available. Add an --insecure option >> which disables gpg signature verification. Warn about >> man-in-the-middle attacks when the --insecure option is used. >> Deprecate the pre-existing webrsync-gpg feature since it requires >> manual gpg configuration. >> >> X-Gentoo-Bug: 597918 >> X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=597918 >> --- >> bin/emerge-webrsync | 51 >> +++ >> man/make.conf.5 | 6 -- 2 files changed, 51 insertions(+), 6 >> deletions(-) >> > > LGTM > Thanks, merged with %s/ gpg/ OpenPGP/: https://gitweb.gentoo.org/proj/portage.git/commit/?id=98c250cceaf380d6dbeacac90482a5d1956dcb80 -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] [sync] Increase the default git sync-depth to 10
On 10/30/2016 12:08 PM, Michał Górny wrote: > Increate the default git sync-depth from 1 to 10, to ensure that > gentoo-mirror git clones retain at least a single original signed > commit. Why wouldn't the last commit always be signed? We'd also need to update man/portage.5 here, and it should probably mention the rational for using a default other than 1: .B sync\-depth Specifies clone depth to use for DVCS repositories. Defaults to 1 (only the newest commit). If set to 0, the depth is unlimited. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] [sync] Increase the default git sync-depth to 10
On Sun, Oct 30, 2016 at 1:19 PM, Zac Medico wrote: > On 10/30/2016 12:08 PM, Michał Górny wrote: >> Increate the default git sync-depth from 1 to 10, to ensure that >> gentoo-mirror git clones retain at least a single original signed >> commit. > > Why wouldn't the last commit always be signed? Oh, it could be a merge commit. Couldn't the original signed commit be much more than 10 commit behind though?
Re: [gentoo-portage-dev] [PATCH] [sync] Increase the default git sync-depth to 10
On 10/30/2016 01:22 PM, Zac Medico wrote: > On Sun, Oct 30, 2016 at 1:19 PM, Zac Medico wrote: >> On 10/30/2016 12:08 PM, Michał Górny wrote: >>> Increate the default git sync-depth from 1 to 10, to ensure that >>> gentoo-mirror git clones retain at least a single original signed >>> commit. >> >> Why wouldn't the last commit always be signed? > > Oh, it could be a merge commit. Couldn't the original signed commit be > much more than 10 commit behind though? > If most people are pushing merge commits, then it's easy to have 10 or more merge commits stacked on top of each other. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] [sync] Run `git update-index --refresh` when doing shallow pulls
On 10/30/2016 12:23 PM, Michał Górny wrote: > Run `git update-index --refresh` to force proper index recheck before > running `git reset --merge` on a shallow pull. This fixes syncing on > some filesystem configurations including overlayfs on squashfs. Maybe we should use --really-refresh in case the --assume-unchanged flag gets toggled on somehow (maybe via .git/config)? -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] [sync] Increase the default git sync-depth to 10
On 10/30/2016 01:42 PM, Michał Górny wrote: > On Sun, 30 Oct 2016 13:22:51 -0700 > Zac Medico wrote: > >> On Sun, Oct 30, 2016 at 1:19 PM, Zac Medico wrote: >>> On 10/30/2016 12:08 PM, Michał Górny wrote: >>>> Increate the default git sync-depth from 1 to 10, to ensure that >>>> gentoo-mirror git clones retain at least a single original signed >>>> commit. >>> >>> Why wouldn't the last commit always be signed? >> >> Oh, it could be a merge commit. Couldn't the original signed commit be >> much more than 10 commit behind though? > > No, gentoo-mirror is only doing a single-level merge commit to combine > signed changes with cache. I'm using 10 as round safe default that > shouldn't take up too much space. OK. I think it would be nice to mention that rationale in the man page. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] [sync] Run `git update-index --refresh` when doing shallow pulls
On 10/30/2016 01:44 PM, Michał Górny wrote: > On Sun, 30 Oct 2016 13:39:53 -0700 > Zac Medico wrote: > >> On 10/30/2016 12:23 PM, Michał Górny wrote: >>> Run `git update-index --refresh` to force proper index recheck before >>> running `git reset --merge` on a shallow pull. This fixes syncing on >>> some filesystem configurations including overlayfs on squashfs. >> >> Maybe we should use --really-refresh in case the --assume-unchanged flag >> gets toggled on somehow (maybe via .git/config)? > > I don't know. I think we can revisit it if it really becomes a problem > for anyone. OK, sounds good. The patch LGTM. -- Thanks, Zac
