Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Joakim Tjernlund]

On Fri, 2018-03-16 at 08:50 +0100, Michał Górny wrote:
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you recognize the sender and know the 
> content is safe.
> 
> 
> W dniu czw, 15.03.2018 o godzinie 21∶44 +, użytkownik Joakim
> Tjernlund napisał:
> > On Thu, 2018-03-15 at 17:02 -0400, Alec Warner wrote:
> > > 
> > > 
> > > On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny  wrote:
> > > > Allow INSTALL_MASK patterns to start with '-' to indicate that
> > > > a specific match is to be excluded from being masked. In this case,
> > > > the last matching pattern determines whether the file is actually
> > > > filtered out or kept.
> > > > ---
> > 
> > Yes, please ! I just needed this feature 2 hours ago.
> > I need the same for PKG_INSTALL_MASK
> > 
> > I hope this allows me to do:
> > INSTALL_MASK="/usr/share/i18n/locales/* -/usr/share/i18n/locales/en_GB 
> > -/usr/share/i18n/locales/sv_SE"
> > to all in /usr/share/i18n/locales/* but /usr/share/i18n/locales/en_GB and 
> > /usr/share/i18n/locales/sv_SE
> > removed in usr/share/i18n/locales/ ?
> > 
> 
> Yes, that is the intended use case.

Thanks, while on the subject I had a similar idea for stripping:

Today one have to build all files in glibc with debug syms just to get debug 
syms for
perf/valgrind in ld.so so it would be great if one could specify that just some 
files
should have debug syms.

 Jocke

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Michał Górny]

W dniu czw, 15.03.2018 o godzinie 21∶44 +, użytkownik Joakim
Tjernlund napisał:
> On Thu, 2018-03-15 at 17:02 -0400, Alec Warner wrote:
> > 
> > 
> > On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny  wrote:
> > > Allow INSTALL_MASK patterns to start with '-' to indicate that
> > > a specific match is to be excluded from being masked. In this case,
> > > the last matching pattern determines whether the file is actually
> > > filtered out or kept.
> > > ---
> 
> Yes, please ! I just needed this feature 2 hours ago.
> I need the same for PKG_INSTALL_MASK 
> 
> I hope this allows me to do:
> INSTALL_MASK="/usr/share/i18n/locales/* -/usr/share/i18n/locales/en_GB 
> -/usr/share/i18n/locales/sv_SE"
> to all in /usr/share/i18n/locales/* but /usr/share/i18n/locales/en_GB and 
> /usr/share/i18n/locales/sv_SE
> removed in usr/share/i18n/locales/ ?
> 

Yes, that is the intended use case.

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Joakim Tjernlund]

On Thu, 2018-03-15 at 17:02 -0400, Alec Warner wrote:
> 
> 
> On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny  wrote:
> > Allow INSTALL_MASK patterns to start with '-' to indicate that
> > a specific match is to be excluded from being masked. In this case,
> > the last matching pattern determines whether the file is actually
> > filtered out or kept.
> > ---

Yes, please ! I just needed this feature 2 hours ago.
I need the same for PKG_INSTALL_MASK 

I hope this allows me to do:
INSTALL_MASK="/usr/share/i18n/locales/* -/usr/share/i18n/locales/en_GB 
-/usr/share/i18n/locales/sv_SE"
to all in /usr/share/i18n/locales/* but /usr/share/i18n/locales/en_GB and 
/usr/share/i18n/locales/sv_SE
removed in usr/share/i18n/locales/ ?

 Jocke

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Michał Górny]

W dniu czw, 15.03.2018 o godzinie 17∶02 -0400, użytkownik Alec Warner
napisał:
> On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny  wrote:
> 
> > Allow INSTALL_MASK patterns to start with '-' to indicate that
> > a specific match is to be excluded from being masked. In this case,
> > the last matching pattern determines whether the file is actually
> > filtered out or kept.
> > ---
> >  pym/portage/dbapi/vartree.py | 10 ++
> >  1 file changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> > index 21904edca..16c246b11 100644
> > --- a/pym/portage/dbapi/vartree.py
> > +++ b/pym/portage/dbapi/vartree.py
> > @@ -3692,19 +3692,21 @@ class dblink(object):
> > def _is_install_masked(self, relative_path):
> > ret = False
> > for pattern in self.settings.install_mask:
> > 
> 
> +   # if pattern starts with -, possibly exclude this
> > path
> > +   pat_res = not pattern.startswith('-')
> > +   if not pat_res:
> > +   pattern = pattern[1:]
> > 
> 
> Maybe consider:
> 
> pattern = pattern[1:] if pattern.startswith('-') else pattern
> 
> I'm not super keen on this pattern in python, but it seems doable here.

I still need pat_res to know whether it's '+' or '-'.

> 
> 
> > # absolute path pattern
> > if pattern.startswith('/'):
> > # match either exact path or one of parent
> > dirs
> > # the latter is done via matching pattern/*
> > if (fnmatch.fnmatch(relative_path,
> > pattern[1:])
> > or
> > fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
> > -   ret = True
> > -   break
> > +   ret = pat_res
> > # filename
> > else:
> > if 
> > fnmatch.fnmatch(os.path.basename(relative_path),
> > pattern):
> > -   ret = True
> > -   break
> > +   ret = pat_res
> > return ret
> > 
> > def treewalk(self, srcroot, destroot, inforoot, myebuild,
> > cleanup=0,
> > --
> > 2.16.2
> > 
> > 
> > 

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Alec Warner]

On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny  wrote:

> Allow INSTALL_MASK patterns to start with '-' to indicate that
> a specific match is to be excluded from being masked. In this case,
> the last matching pattern determines whether the file is actually
> filtered out or kept.
> ---
>  pym/portage/dbapi/vartree.py | 10 ++
>  1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> index 21904edca..16c246b11 100644
> --- a/pym/portage/dbapi/vartree.py
> +++ b/pym/portage/dbapi/vartree.py
> @@ -3692,19 +3692,21 @@ class dblink(object):
> def _is_install_masked(self, relative_path):
> ret = False
> for pattern in self.settings.install_mask:
>
+   # if pattern starts with -, possibly exclude this
> path
> +   pat_res = not pattern.startswith('-')
> +   if not pat_res:
> +   pattern = pattern[1:]
>

Maybe consider:

pattern = pattern[1:] if pattern.startswith('-') else pattern

I'm not super keen on this pattern in python, but it seems doable here.


> # absolute path pattern
> if pattern.startswith('/'):
> # match either exact path or one of parent
> dirs
> # the latter is done via matching pattern/*
> if (fnmatch.fnmatch(relative_path,
> pattern[1:])
> or
> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
> -   ret = True
> -   break
> +   ret = pat_res
> # filename
> else:
> if 
> fnmatch.fnmatch(os.path.basename(relative_path),
> pattern):
> -   ret = True
> -   break
> +   ret = pat_res
> return ret
>
> def treewalk(self, srcroot, destroot, inforoot, myebuild,
> cleanup=0,
> --
> 2.16.2
>
>
>

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Zac Medico]

On 06/12/2016 03:03 AM, Michał Górny wrote:
> Dnia 12 czerwca 2016 11:43:35 CEST, Zac Medico  
> napisał(a):
>> On 06/12/2016 02:31 AM, Michał Górny wrote:
>>> Dnia 12 czerwca 2016 11:20:36 CEST, Zac Medico 
>> napisał(a):
 On 05/21/2016 11:56 PM, Michał Górny wrote:
> Allow INSTALL_MASK patterns to start with '-' to indicate that
> a specific match is to be excluded from being masked. In this case,
> the last matching pattern determines whether the file is actually
> filtered out or kept.
> ---
>  pym/portage/dbapi/vartree.py | 10 ++
>  1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/pym/portage/dbapi/vartree.py
 b/pym/portage/dbapi/vartree.py
> index 8e5ac43..d02d850 100644
> --- a/pym/portage/dbapi/vartree.py
> +++ b/pym/portage/dbapi/vartree.py
> @@ -3690,19 +3690,21 @@ class dblink(object):
>   def _is_install_masked(self, relative_path):
>   ret = False
>   for pattern in self.settings.install_mask:
> + # if pattern starts with -, possibly exclude this path
> + pat_res = not pattern.startswith('-')
> + if not pat_res:
> + pattern = pattern[1:]
>   # absolute path pattern
>   if pattern.startswith('/'):
>   # match either exact path or one of parent dirs
>   # the latter is done via matching pattern/*
>   if (fnmatch.fnmatch(relative_path, pattern[1:])
>   or 
> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
> - ret = True
> - break
> + ret = pat_res
>   # filename
>   else:
>   if 
> fnmatch.fnmatch(os.path.basename(relative_path), pattern):
> - ret = True
> - break
> + ret = pat_res
>   return ret
>  
>   def treewalk(self, srcroot, destroot, inforoot, myebuild,
 cleanup=0,
>

 In order to implement this with pre-compiled patterns, we can use a
 separate regular expression to represent all of the exclusions.
>>>
>>> That won't work since exclusive and inclusive patterns can be
>> stacked, and for this to work they are applied in order.
>>>
>>> E.g. /usr/foo -/usr/foo/bar /usr/foo/bar/baz
>>>
>>> Should remove all foo subdirectories except for bar, and also baz
>> inside bar.
>>
>> Hmm, I suppose there could be an optimized implementation to handle
>> INSTALL_MASK settings where there are no such ordering constraints.
> 
> Is this really worth the effort? I'm using the patch for a while and I/O's 
> the bottleneck, not CPU. If you really insist on optimizing this, i guess we 
> could precompile all patterns separately.

Since it's not going to make any difference when INSTALL_MASK is not
used, I guess it's fine to leave it unoptimized for now.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Michał Górny]

Dnia 12 czerwca 2016 11:43:35 CEST, Zac Medico  napisał(a):
>On 06/12/2016 02:31 AM, Michał Górny wrote:
>> Dnia 12 czerwca 2016 11:20:36 CEST, Zac Medico 
>napisał(a):
>>> On 05/21/2016 11:56 PM, Michał Górny wrote:
 Allow INSTALL_MASK patterns to start with '-' to indicate that
 a specific match is to be excluded from being masked. In this case,
 the last matching pattern determines whether the file is actually
 filtered out or kept.
 ---
  pym/portage/dbapi/vartree.py | 10 ++
  1 file changed, 6 insertions(+), 4 deletions(-)

 diff --git a/pym/portage/dbapi/vartree.py
>>> b/pym/portage/dbapi/vartree.py
 index 8e5ac43..d02d850 100644
 --- a/pym/portage/dbapi/vartree.py
 +++ b/pym/portage/dbapi/vartree.py
 @@ -3690,19 +3690,21 @@ class dblink(object):
def _is_install_masked(self, relative_path):
ret = False
for pattern in self.settings.install_mask:
 +  # if pattern starts with -, possibly exclude this path
 +  pat_res = not pattern.startswith('-')
 +  if not pat_res:
 +  pattern = pattern[1:]
# absolute path pattern
if pattern.startswith('/'):
# match either exact path or one of parent dirs
# the latter is done via matching pattern/*
if (fnmatch.fnmatch(relative_path, pattern[1:])
or 
 fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
 -  ret = True
 -  break
 +  ret = pat_res
# filename
else:
if 
 fnmatch.fnmatch(os.path.basename(relative_path), pattern):
 -  ret = True
 -  break
 +  ret = pat_res
return ret
  
def treewalk(self, srcroot, destroot, inforoot, myebuild,
>>> cleanup=0,

>>>
>>> In order to implement this with pre-compiled patterns, we can use a
>>> separate regular expression to represent all of the exclusions.
>> 
>> That won't work since exclusive and inclusive patterns can be
>stacked, and for this to work they are applied in order.
>> 
>> E.g. /usr/foo -/usr/foo/bar /usr/foo/bar/baz
>> 
>> Should remove all foo subdirectories except for bar, and also baz
>inside bar.
>
>Hmm, I suppose there could be an optimized implementation to handle
>INSTALL_MASK settings where there are no such ordering constraints.

Is this really worth the effort? I'm using the patch for a while and I/O's the 
bottleneck, not CPU. If you really insist on optimizing this, i guess we could 
precompile all patterns separately.


-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Zac Medico]

On 06/12/2016 02:31 AM, Michał Górny wrote:
> Dnia 12 czerwca 2016 11:20:36 CEST, Zac Medico  
> napisał(a):
>> On 05/21/2016 11:56 PM, Michał Górny wrote:
>>> Allow INSTALL_MASK patterns to start with '-' to indicate that
>>> a specific match is to be excluded from being masked. In this case,
>>> the last matching pattern determines whether the file is actually
>>> filtered out or kept.
>>> ---
>>>  pym/portage/dbapi/vartree.py | 10 ++
>>>  1 file changed, 6 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/pym/portage/dbapi/vartree.py
>> b/pym/portage/dbapi/vartree.py
>>> index 8e5ac43..d02d850 100644
>>> --- a/pym/portage/dbapi/vartree.py
>>> +++ b/pym/portage/dbapi/vartree.py
>>> @@ -3690,19 +3690,21 @@ class dblink(object):
>>> def _is_install_masked(self, relative_path):
>>> ret = False
>>> for pattern in self.settings.install_mask:
>>> +   # if pattern starts with -, possibly exclude this path
>>> +   pat_res = not pattern.startswith('-')
>>> +   if not pat_res:
>>> +   pattern = pattern[1:]
>>> # absolute path pattern
>>> if pattern.startswith('/'):
>>> # match either exact path or one of parent dirs
>>> # the latter is done via matching pattern/*
>>> if (fnmatch.fnmatch(relative_path, pattern[1:])
>>> or 
>>> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
>>> -   ret = True
>>> -   break
>>> +   ret = pat_res
>>> # filename
>>> else:
>>> if 
>>> fnmatch.fnmatch(os.path.basename(relative_path), pattern):
>>> -   ret = True
>>> -   break
>>> +   ret = pat_res
>>> return ret
>>>  
>>> def treewalk(self, srcroot, destroot, inforoot, myebuild,
>> cleanup=0,
>>>
>>
>> In order to implement this with pre-compiled patterns, we can use a
>> separate regular expression to represent all of the exclusions.
> 
> That won't work since exclusive and inclusive patterns can be stacked, and 
> for this to work they are applied in order.
> 
> E.g. /usr/foo -/usr/foo/bar /usr/foo/bar/baz
> 
> Should remove all foo subdirectories except for bar, and also baz inside bar.

Hmm, I suppose there could be an optimized implementation to handle
INSTALL_MASK settings where there are no such ordering constraints.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Michał Górny]

Dnia 12 czerwca 2016 11:20:36 CEST, Zac Medico  napisał(a):
>On 05/21/2016 11:56 PM, Michał Górny wrote:
>> Allow INSTALL_MASK patterns to start with '-' to indicate that
>> a specific match is to be excluded from being masked. In this case,
>> the last matching pattern determines whether the file is actually
>> filtered out or kept.
>> ---
>>  pym/portage/dbapi/vartree.py | 10 ++
>>  1 file changed, 6 insertions(+), 4 deletions(-)
>> 
>> diff --git a/pym/portage/dbapi/vartree.py
>b/pym/portage/dbapi/vartree.py
>> index 8e5ac43..d02d850 100644
>> --- a/pym/portage/dbapi/vartree.py
>> +++ b/pym/portage/dbapi/vartree.py
>> @@ -3690,19 +3690,21 @@ class dblink(object):
>>  def _is_install_masked(self, relative_path):
>>  ret = False
>>  for pattern in self.settings.install_mask:
>> +# if pattern starts with -, possibly exclude this path
>> +pat_res = not pattern.startswith('-')
>> +if not pat_res:
>> +pattern = pattern[1:]
>>  # absolute path pattern
>>  if pattern.startswith('/'):
>>  # match either exact path or one of parent dirs
>>  # the latter is done via matching pattern/*
>>  if (fnmatch.fnmatch(relative_path, pattern[1:])
>>  or 
>> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
>> -ret = True
>> -break
>> +ret = pat_res
>>  # filename
>>  else:
>>  if 
>> fnmatch.fnmatch(os.path.basename(relative_path), pattern):
>> -ret = True
>> -break
>> +ret = pat_res
>>  return ret
>>  
>>  def treewalk(self, srcroot, destroot, inforoot, myebuild,
>cleanup=0,
>> 
>
>In order to implement this with pre-compiled patterns, we can use a
>separate regular expression to represent all of the exclusions.

That won't work since exclusive and inclusive patterns can be stacked, and for 
this to work they are applied in order.

E.g. /usr/foo -/usr/foo/bar /usr/foo/bar/baz

Should remove all foo subdirectories except for bar, and also baz inside bar.


-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Zac Medico]

On 05/21/2016 11:56 PM, Michał Górny wrote:
> Allow INSTALL_MASK patterns to start with '-' to indicate that
> a specific match is to be excluded from being masked. In this case,
> the last matching pattern determines whether the file is actually
> filtered out or kept.
> ---
>  pym/portage/dbapi/vartree.py | 10 ++
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> index 8e5ac43..d02d850 100644
> --- a/pym/portage/dbapi/vartree.py
> +++ b/pym/portage/dbapi/vartree.py
> @@ -3690,19 +3690,21 @@ class dblink(object):
>   def _is_install_masked(self, relative_path):
>   ret = False
>   for pattern in self.settings.install_mask:
> + # if pattern starts with -, possibly exclude this path
> + pat_res = not pattern.startswith('-')
> + if not pat_res:
> + pattern = pattern[1:]
>   # absolute path pattern
>   if pattern.startswith('/'):
>   # match either exact path or one of parent dirs
>   # the latter is done via matching pattern/*
>   if (fnmatch.fnmatch(relative_path, pattern[1:])
>   or 
> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
> - ret = True
> - break
> + ret = pat_res
>   # filename
>   else:
>   if 
> fnmatch.fnmatch(os.path.basename(relative_path), pattern):
> - ret = True
> - break
> + ret = pat_res
>   return ret
>  
>   def treewalk(self, srcroot, destroot, inforoot, myebuild, cleanup=0,
> 

In order to implement this with pre-compiled patterns, we can use a
separate regular expression to represent all of the exclusions.
-- 
Thanks,
Zac