Re: [gentoo-dev] rfc: Remove inherit eutils from font.eclass for EAPI=6

2018-02-14 Thread Guilherme Amadio 
On Thu, Feb 15, 2018 at 03:16:22AM +0200, Mart Raudsepp wrote:
> On Wed, 2018-02-14 at 23:43 +0100, Jonas Stein wrote:
> > Did I miss something?
> > Who can help to check with (an automatic) testenvironment, if these
> > packages will survive?
>
> Don't check with test environments, read the ebuilds.

I agree with this. Font ebuilds are usually quite trivial, so it might
even be faster to just read them. The font.eclass has not been touched
since the move to git, but I suspected few fonts would actually use
eutils, so here is another grep that shows it may be easier than we
think to get rid of inherit eutils, even if for all EAPIs:

https://devmanual.gentoo.org/eclass-reference/eutils.eclass/index.html

gentoo $ export eutils_funcs=(eqawarn ecvs_clean esvn_clean \
egit_clean emktemp edos2unix strip-linguas built_with_use make_wrapper \
path_exists use_if_iuse optfeature epause ebeep usex einstalldocs \
in_iuse)
gentoo $ for func in ${eutils_funcs[@]}; do grep -rI 'inherit.*font' 
--include='*.ebuild' -l | \
xargs grep $func; done | cut -d: -f1 | sort | uniq

app-office/lyx/lyx-2.2.3-r1.ebuild
app-office/lyx/lyx-2.2.3-r2.ebuild
media-fonts/terminus-font/terminus-font-4.39-r1.ebuild
media-fonts/terminus-font/terminus-font-4.40.ebuild
media-fonts/terminus-font/terminus-font-4.46.ebuild
media-fonts/unifont/unifont-10.0.04.ebuild
media-fonts/unifont/unifont-10.0.05.ebuild
media-fonts/unifont/unifont-10.0.06.ebuild
media-fonts/unifont/unifont-9.0.06.ebuild
net-misc/suite3270/suite3270-3.4_p10.ebuild
net-misc/suite3270/suite3270-3.5_p10.ebuild
net-misc/suite3270/suite3270-3.5_p12.ebuild
net-misc/suite3270/suite3270-3.6_p4.ebuild

Most of the above only use usex, and terminus-font uses einstalldocs as well.
I think these should be pretty easy to fix to not use eutils, or simply
add 'inherit eutils' for these ebuilds, then remove from the eclass.

I can help with testing by reading ebuilds more carefully and emerging
them after the change in font.eclass.

Cheers,
-Guilherme

Re: [gentoo-dev] rfc: Remove inherit eutils from font.eclass for EAPI=6

2018-02-14 Thread Ulrich Mueller 
> On Wed, 14 Feb 2018, Jonas Stein wrote:

> I think we do not need

> inherit eutils
> https://github.com/gentoo/gentoo/blob/master/eclass/font.eclass#L9

> for the case that EAPI=6

What is eutils needed for in EAPIs 0 to 5?

Ulrich


pgpnMLnuhBQTB.pgp
Description: PGP signature

Re: [gentoo-portage-dev] [PATCH] filter-bash-environment.py: use buffered input, raw bytes (bug 647654)

2018-02-14 Thread Zac Medico 
On 02/14/2018 12:46 PM, Alec Warner wrote:
> On Wed, Feb 14, 2018 at 3:38 PM, Zac Medico  > wrote:
> 
> Use sys.stdin.buffer instead of sys.stdin.buffer.raw, for buffered
> input.
> Also use raw bytes instead of unicode strings, in order to avoid making
> assumptions about character encodings, and also to avoid overhead from
> unicode decoding/encoding.
> 
> 
> Maybe a functional test?

In PATCH v2 I've added a unit test that compares expected output to
actual output.

> take real environment, check it into source code, run filter over it to
> get output, run bash -n on output?
> 
> Better ideas?

We already have a number of tests that do the equivalent with a complete
ebuild.sh by calling emerge. The resulting environment is sourced, and
that will fail if the bash syntax is invalid. All of those tests failed
for python3.4 here because it doesn't support the % operator with bytes
operands:

https://travis-ci.org/gentoo/portage/jobs/341614451
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-portage-dev] [PATCH v2] filter-bash-environment.py: use buffered input, raw bytes (bug 647654)

2018-02-14 Thread Zac Medico 
Use sys.stdin.buffer instead of sys.stdin.buffer.raw, for buffered input.
Also use raw bytes instead of unicode strings, in order to avoid making
assumptions about character encodings, and also to avoid overhead from
unicode decoding/encoding.

Since the % operator does not support bytes operands in python3.4, use
the + operator to format strings of bytes.

Bug: https://bugs.gentoo.org/647654
---
[PATCH v2] changes:
* don't use % operator with bytes operands, for python3.4 compat
* add unit test that compares expected output to actual output

 bin/filter-bash-environment.py|  47 +--
 pym/portage/tests/bin/test_filter_bash_env.py | 113 ++
 2 files changed, 135 insertions(+), 25 deletions(-)
 create mode 100644 pym/portage/tests/bin/test_filter_bash_env.py

diff --git a/bin/filter-bash-environment.py b/bin/filter-bash-environment.py
index a4cdc5429..668aa7452 100755
--- a/bin/filter-bash-environment.py
+++ b/bin/filter-bash-environment.py
@@ -2,21 +2,19 @@
 # Copyright 1999-2014 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
-import codecs
-import io
 import os
 import re
 import sys
 
-here_doc_re = re.compile(r'.*\s<<[-]?(\w+)$')
-func_start_re = re.compile(r'^[-\w]+\s*\(\)\s*$')
-func_end_re = re.compile(r'^\}$')
+here_doc_re = re.compile(br'.*\s<<[-]?(\w+)$')
+func_start_re = re.compile(br'^[-\w]+\s*\(\)\s*$')
+func_end_re = re.compile(br'^\}$')
 
-var_assign_re = 
re.compile(r'(^|^declare\s+-\S+\s+|^declare\s+|^export\s+)([^=\s]+)=("|\')?.*$')
-close_quote_re = re.compile(r'(\\"|"|\')\s*$')
-readonly_re = re.compile(r'^declare\s+-(\S*)r(\S*)\s+')
+var_assign_re = 
re.compile(br'(^|^declare\s+-\S+\s+|^declare\s+|^export\s+)([^=\s]+)=("|\')?.*$')
+close_quote_re = re.compile(br'(\\"|"|\')\s*$')
+readonly_re = re.compile(br'^declare\s+-(\S*)r(\S*)\s+')
 # declare without assignment
-var_declare_re = re.compile(r'^declare(\s+-\S+)?\s+([^=\s]+)\s*$')
+var_declare_re = re.compile(br'^declare(\s+-\S+)?\s+([^=\s]+)\s*$')
 
 def have_end_quote(quote, line):
"""
@@ -32,16 +30,16 @@ def have_end_quote(quote, line):
 def filter_declare_readonly_opt(line):
readonly_match = readonly_re.match(line)
if readonly_match is not None:
-   declare_opts = ''
+   declare_opts = b''
for i in (1, 2):
group = readonly_match.group(i)
if group is not None:
declare_opts += group
if declare_opts:
-   line = 'declare -%s %s' % \
-   (declare_opts, line[readonly_match.end():])
+   line = b'declare -' + declare_opts + \
+   b' ' + line[readonly_match.end():]
else:
-   line = 'declare ' + line[readonly_match.end():]
+   line = b'declare ' + line[readonly_match.end():]
return line
 
 def filter_bash_environment(pattern, file_in, file_out):
@@ -57,7 +55,7 @@ def filter_bash_environment(pattern, file_in, file_out):
for line in file_in:
if multi_line_quote is not None:
if not multi_line_quote_filter:
-   file_out.write(line.replace("\1", ""))
+   file_out.write(line.replace(b"\1", b""))
if have_end_quote(multi_line_quote, line):
multi_line_quote = None
multi_line_quote_filter = None
@@ -78,7 +76,7 @@ def filter_bash_environment(pattern, file_in, file_out):
multi_line_quote_filter = filter_this
if not filter_this:
line = filter_declare_readonly_opt(line)
-   file_out.write(line.replace("\1", ""))
+   file_out.write(line.replace(b"\1", b""))
continue
else:
declare_match = var_declare_re.match(line)
@@ -98,7 +96,7 @@ def filter_bash_environment(pattern, file_in, file_out):
continue
here_doc = here_doc_re.match(line)
if here_doc is not None:
-   here_doc_delim = re.compile("^%s$" % here_doc.group(1))
+   here_doc_delim = re.compile(b'^%s' + here_doc.group(1) 
+ b'$')
file_out.write(line)
continue
# Note: here-documents are handled before functions since 
otherwise
@@ -141,18 +139,17 @@ if __name__ == "__main__":
file_in = sys.stdin
file_out = sys.stdout
if sys.hexversion >= 0x300:
-   file_in = codecs.iterdecode(sys.stdin.buffer.raw,
-   'utf_8', errors='replace')
-

Re: [gentoo-dev] stability of 17.0 hardened profile

2018-02-14 Thread Magnus Granberg 
onsdag 14 februari 2018 kl. 19:44:13 CET skrev  Paweł Hajdan, Jr.:
> I was looking into the new 17.0 profiles (nice work!), and noticed the
> hardened one is marked as dev. I'm somewhat concerned about switching to
> that on my laptop (I'm currently using hardened/linux/amd64).
> 
> Is there something I can do to help move the profile to stable?
> 
> Alternatively, is there a different profile recommended for hardened?
> 
> # eselect profile list
> ...
>   [12]  default/linux/amd64/17.0 (stable)
>   [13]  default/linux/amd64/17.0/selinux (dev)
>   [14]  default/linux/amd64/17.0/hardened (dev)
>   [15]  default/linux/amd64/17.0/hardened/selinux (dev)
>   [16]  default/linux/amd64/17.0/desktop (stable)
>   [17]  default/linux/amd64/17.0/desktop/gnome (stable)
>   [18]  default/linux/amd64/17.0/desktop/gnome/systemd (stable)
>   [19]  default/linux/amd64/17.0/desktop/plasma (stable)
>   [20]  default/linux/amd64/17.0/desktop/plasma/systemd (stable)
>   [21]  default/linux/amd64/17.0/developer (stable)
>   [22]  default/linux/amd64/17.0/no-multilib (stable)
>   [23]  default/linux/amd64/17.0/no-multilib/hardened (dev)
>   [24]  default/linux/amd64/17.0/no-multilib/hardened/selinux (dev)
>   [25]  default/linux/amd64/17.0/systemd (stable)
>   [26]  default/linux/amd64/17.0/x32 (dev)
> ...
>   [40]  hardened/linux/amd64 (stable) *
> 
> Paweł
Have mark the Hardened 17.0 as stable now
/Magnus

Re: [gentoo-dev] rfc: Remove inherit eutils from font.eclass for EAPI=6

2018-02-14 Thread Mart Raudsepp 
On Wed, 2018-02-14 at 23:43 +0100, Jonas Stein wrote:
> Did I miss something?
> Who can help to check with (an automatic) testenvironment, if these
> packages will survive?

Don't check with test environments, read the ebuilds. Test environments
will find it hard to catch changes in the installed files. For example
the first issue in that list I found was relying on the inherit for a
make_desktop_entry call. In a test environment, it'll just not install
a desktop entry while outputting a function not found call or something
I suspect. You are lucky if that message is somehow parsed out, and
then there are other similar issues possible. (In this example,
inheriting desktop.eclass should be added to the ebuild before eclass
drops the eutils inherit)

If you want to do it, really should just read over all these ebuilds
manually. A test environment has a hard time to catch some sort of
conditional calls to eutils functions as well, when they don't enter
that conditional block for any reason.
I suspect most others than this xmind are rather simple ones that don't
even have anything beyond declaring the DEPEND and other variables.

Also start with the patch to bail out on unknown EAPI's that you were
planning to do. This can be separate commit done before this change,
once checks are done. Of course ideally pushed together (if the eutils
drop pans out) to minimize cache regenerations for developers.


Mart

[gentoo-dev] rfc: Remove inherit eutils from font.eclass for EAPI=6

2018-02-14 Thread Jonas Stein 
Hi,

I think we do not need

inherit eutils
https://github.com/gentoo/gentoo/blob/master/eclass/font.eclass#L9

for the case that EAPI=6

I suggest to change the line to
 [[ ${EAPI:-0} == [012345] ]] && inherit eutils


we would have to test the following packages (the grep line does not
list matches with '\' line breaks)


$ grep -rI 'inherit.*font' --include='*.ebuild' -l |xargs grep EAPI
|grep ':.*6' |cut -d: -f1
app-misc/xmind/xmind-3.7.6_p201801311814.ebuild
media-fonts/essays1743/essays1743-2.000.ebuild
media-fonts/essays1743/essays1743-2.001.ebuild
media-fonts/fontawesome/fontawesome-5.0.4.ebuild
media-fonts/fontawesome/fontawesome-4.7.0.ebuild
media-fonts/baekmuk-fonts/baekmuk-fonts-2.2-r2.ebuild
media-fonts/terminus-font/terminus-font-4.46.ebuild
media-fonts/terminus-font/terminus-font-4.40.ebuild
media-fonts/unifont/unifont-10.0.05.ebuild
media-fonts/unifont/unifont-10.0.04.ebuild
media-fonts/unifont/unifont-9.0.06.ebuild
media-fonts/unifont/unifont-10.0.06.ebuild
media-fonts/jisx0213-fonts/jisx0213-fonts-20040425-r2.ebuild
media-fonts/paratype-astra/paratype-astra-1001.ebuild
media-fonts/ahem/ahem-1.0.ebuild
media-fonts/nunito/nunito-1.0-r1.ebuild
media-fonts/stix-fonts/stix-fonts-2.0.0.ebuild
media-fonts/dina/dina-2.93.ebuild
media-fonts/mplus-outline-fonts/mplus-outline-fonts-0_pre062.ebuild
media-fonts/mplus-outline-fonts/mplus-outline-fonts-0_pre063.ebuild
media-fonts/umeplus-fonts/umeplus-fonts-20160402.ebuild
media-fonts/opendesktop-fonts/opendesktop-fonts-1.4.2.ebuild
media-fonts/noto/noto-20170403.ebuild
media-fonts/roboto/roboto-2.138.ebuild
media-fonts/hack/hack-3.001.ebuild
media-fonts/noto-cjk/noto-cjk-20150615.ebuild
media-fonts/cm-unicode/cm-unicode-0.7.0-r1.ebuild
media-fonts/comic-neue/comic-neue-2.3.ebuild
media-fonts/dejavu/dejavu-2.37.ebuild
media-fonts/source-han-sans/source-han-sans-1.004.ebuild
media-fonts/croscorefonts/croscorefonts-1.31.0.ebuild
media-fonts/open-sans/open-sans-1-r1.ebuild
media-fonts/heuristica/heuristica-1.0.2.ebuild
media-fonts/source-pro/source-pro-20170111.ebuild
media-fonts/source-pro/source-pro-20160608.ebuild
kde-plasma/oxygen-fonts/oxygen-fonts-5.4.3.ebuild
app-office/wps-office/wps-office-10.1.0.5707_alpha21-r1.ebuild
app-office/lyx/lyx-2.2.3-r1.ebuild
app-office/lyx/lyx-2.2.3-r2.ebuild



Did I miss something?
Who can help to check with (an automatic) testenvironment, if these
packages will survive?

-- 
Best,
Jonas



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH] filter-bash-environment.py: use buffered input, raw bytes (bug 647654)

2018-02-14 Thread Alec Warner 
On Wed, Feb 14, 2018 at 3:38 PM, Zac Medico  wrote:

> Use sys.stdin.buffer instead of sys.stdin.buffer.raw, for buffered input.
> Also use raw bytes instead of unicode strings, in order to avoid making
> assumptions about character encodings, and also to avoid overhead from
> unicode decoding/encoding.
>

Maybe a functional test?

take real environment, check it into source code, run filter over it to get
output, run bash -n on output?

Better ideas?

-A


>
> Bug: https://bugs.gentoo.org/647654
> ---
>  bin/filter-bash-environment.py | 45 --
> 
>  1 file changed, 21 insertions(+), 24 deletions(-)
>
> diff --git a/bin/filter-bash-environment.py b/bin/filter-bash-environment.
> py
> index a4cdc5429..91c194b95 100755
> --- a/bin/filter-bash-environment.py
> +++ b/bin/filter-bash-environment.py
> @@ -2,21 +2,19 @@
>  # Copyright 1999-2014 Gentoo Foundation
>  # Distributed under the terms of the GNU General Public License v2
>
> -import codecs
> -import io
>  import os
>  import re
>  import sys
>
> -here_doc_re = re.compile(r'.*\s<<[-]?(\w+)$')
> -func_start_re = re.compile(r'^[-\w]+\s*\(\)\s*$')
> -func_end_re = re.compile(r'^\}$')
> +here_doc_re = re.compile(br'.*\s<<[-]?(\w+)$')
> +func_start_re = re.compile(br'^[-\w]+\s*\(\)\s*$')
> +func_end_re = re.compile(br'^\}$')
>
> -var_assign_re = re.compile(r'(^|^declare\s+-\
> S+\s+|^declare\s+|^export\s+)([^=\s]+)=("|\')?.*$')
> -close_quote_re = re.compile(r'(\\"|"|\')\s*$')
> -readonly_re = re.compile(r'^declare\s+-(\S*)r(\S*)\s+')
> +var_assign_re = re.compile(br'(^|^declare\s+-\
> S+\s+|^declare\s+|^export\s+)([^=\s]+)=("|\')?.*$')
> +close_quote_re = re.compile(br'(\\"|"|\')\s*$')
> +readonly_re = re.compile(br'^declare\s+-(\S*)r(\S*)\s+')
>  # declare without assignment
> -var_declare_re = re.compile(r'^declare(\s+-\S+)?\s+([^=\s]+)\s*$')
> +var_declare_re = re.compile(br'^declare(\s+-\S+)?\s+([^=\s]+)\s*$')
>
>  def have_end_quote(quote, line):
> """
> @@ -32,16 +30,16 @@ def have_end_quote(quote, line):
>  def filter_declare_readonly_opt(line):
> readonly_match = readonly_re.match(line)
> if readonly_match is not None:
> -   declare_opts = ''
> +   declare_opts = b''
> for i in (1, 2):
> group = readonly_match.group(i)
> if group is not None:
> declare_opts += group
> if declare_opts:
> -   line = 'declare -%s %s' % \
> +   line = b'declare -%s %s' % \
> (declare_opts, line[readonly_match.end():])
> else:
> -   line = 'declare ' + line[readonly_match.end():]
> +   line = b'declare ' + line[readonly_match.end():]
> return line
>
>  def filter_bash_environment(pattern, file_in, file_out):
> @@ -57,7 +55,7 @@ def filter_bash_environment(pattern, file_in, file_out):
> for line in file_in:
> if multi_line_quote is not None:
> if not multi_line_quote_filter:
> -   file_out.write(line.replace("\1", ""))
> +   file_out.write(line.replace(b"\1", b""))
> if have_end_quote(multi_line_quote, line):
> multi_line_quote = None
> multi_line_quote_filter = None
> @@ -78,7 +76,7 @@ def filter_bash_environment(pattern, file_in, file_out):
> multi_line_quote_filter =
> filter_this
> if not filter_this:
> line = filter_declare_readonly_opt(
> line)
> -   file_out.write(line.replace("\1",
> ""))
> +   file_out.write(line.replace(b"\1",
> b""))
> continue
> else:
> declare_match = var_declare_re.match(line)
> @@ -98,7 +96,7 @@ def filter_bash_environment(pattern, file_in, file_out):
> continue
> here_doc = here_doc_re.match(line)
> if here_doc is not None:
> -   here_doc_delim = re.compile("^%s$" %
> here_doc.group(1))
> +   here_doc_delim = re.compile(b"^%s$" %
> here_doc.group(1))
> file_out.write(line)
> continue
> # Note: here-documents are handled before functions since
> otherwise
> @@ -141,18 +139,17 @@ if __name__ == "__main__":
> file_in = sys.stdin
> file_out = sys.stdout
> if sys.hexversion >= 0x300:
> -   file_in = codecs.iterdecode(sys.stdin.buffer.raw,
> -   'utf_8', errors='replace')
> -   file_out =

[gentoo-portage-dev] [PATCH] filter-bash-environment.py: use buffered input, raw bytes (bug 647654)

2018-02-14 Thread Zac Medico 
Use sys.stdin.buffer instead of sys.stdin.buffer.raw, for buffered input.
Also use raw bytes instead of unicode strings, in order to avoid making
assumptions about character encodings, and also to avoid overhead from
unicode decoding/encoding.

Bug: https://bugs.gentoo.org/647654
---
 bin/filter-bash-environment.py | 45 --
 1 file changed, 21 insertions(+), 24 deletions(-)

diff --git a/bin/filter-bash-environment.py b/bin/filter-bash-environment.py
index a4cdc5429..91c194b95 100755
--- a/bin/filter-bash-environment.py
+++ b/bin/filter-bash-environment.py
@@ -2,21 +2,19 @@
 # Copyright 1999-2014 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
-import codecs
-import io
 import os
 import re
 import sys
 
-here_doc_re = re.compile(r'.*\s<<[-]?(\w+)$')
-func_start_re = re.compile(r'^[-\w]+\s*\(\)\s*$')
-func_end_re = re.compile(r'^\}$')
+here_doc_re = re.compile(br'.*\s<<[-]?(\w+)$')
+func_start_re = re.compile(br'^[-\w]+\s*\(\)\s*$')
+func_end_re = re.compile(br'^\}$')
 
-var_assign_re = 
re.compile(r'(^|^declare\s+-\S+\s+|^declare\s+|^export\s+)([^=\s]+)=("|\')?.*$')
-close_quote_re = re.compile(r'(\\"|"|\')\s*$')
-readonly_re = re.compile(r'^declare\s+-(\S*)r(\S*)\s+')
+var_assign_re = 
re.compile(br'(^|^declare\s+-\S+\s+|^declare\s+|^export\s+)([^=\s]+)=("|\')?.*$')
+close_quote_re = re.compile(br'(\\"|"|\')\s*$')
+readonly_re = re.compile(br'^declare\s+-(\S*)r(\S*)\s+')
 # declare without assignment
-var_declare_re = re.compile(r'^declare(\s+-\S+)?\s+([^=\s]+)\s*$')
+var_declare_re = re.compile(br'^declare(\s+-\S+)?\s+([^=\s]+)\s*$')
 
 def have_end_quote(quote, line):
"""
@@ -32,16 +30,16 @@ def have_end_quote(quote, line):
 def filter_declare_readonly_opt(line):
readonly_match = readonly_re.match(line)
if readonly_match is not None:
-   declare_opts = ''
+   declare_opts = b''
for i in (1, 2):
group = readonly_match.group(i)
if group is not None:
declare_opts += group
if declare_opts:
-   line = 'declare -%s %s' % \
+   line = b'declare -%s %s' % \
(declare_opts, line[readonly_match.end():])
else:
-   line = 'declare ' + line[readonly_match.end():]
+   line = b'declare ' + line[readonly_match.end():]
return line
 
 def filter_bash_environment(pattern, file_in, file_out):
@@ -57,7 +55,7 @@ def filter_bash_environment(pattern, file_in, file_out):
for line in file_in:
if multi_line_quote is not None:
if not multi_line_quote_filter:
-   file_out.write(line.replace("\1", ""))
+   file_out.write(line.replace(b"\1", b""))
if have_end_quote(multi_line_quote, line):
multi_line_quote = None
multi_line_quote_filter = None
@@ -78,7 +76,7 @@ def filter_bash_environment(pattern, file_in, file_out):
multi_line_quote_filter = filter_this
if not filter_this:
line = filter_declare_readonly_opt(line)
-   file_out.write(line.replace("\1", ""))
+   file_out.write(line.replace(b"\1", b""))
continue
else:
declare_match = var_declare_re.match(line)
@@ -98,7 +96,7 @@ def filter_bash_environment(pattern, file_in, file_out):
continue
here_doc = here_doc_re.match(line)
if here_doc is not None:
-   here_doc_delim = re.compile("^%s$" % here_doc.group(1))
+   here_doc_delim = re.compile(b"^%s$" % here_doc.group(1))
file_out.write(line)
continue
# Note: here-documents are handled before functions since 
otherwise
@@ -141,18 +139,17 @@ if __name__ == "__main__":
file_in = sys.stdin
file_out = sys.stdout
if sys.hexversion >= 0x300:
-   file_in = codecs.iterdecode(sys.stdin.buffer.raw,
-   'utf_8', errors='replace')
-   file_out = io.TextIOWrapper(sys.stdout.buffer,
-   'utf_8', errors='backslashreplace')
-
-   var_pattern = args[0].split()
+   file_in = sys.stdin.buffer
+   file_out = sys.stdout.buffer
+   var_pattern = os.fsencode(args[0]).split()
+   else:
+   var_pattern = args[0].split()
 
# Filter invalid variable names that are not supported by bash.
-   var_pattern.append(r'\d.*')
-

Re: [gentoo-portage-dev] emerging binary pkgs very slow on ppc32

2018-02-14 Thread Zac Medico 
On 02/14/2018 07:49 AM, Joakim Tjernlund wrote:
> We got an embedded target which we upded with binary pks(157 in this case) 
> using:
>  PKGDIR=/opt/fs/osappl04a-r30b-1/usr/portage/packages emerge 
> --rebuilt-binaries --verbose --usepkgonly -NDu @world @cusfpv3
> and this takes forever, about 1 hour
> 
> not sure where to start looking for a cause, is this known for ppc32 ?

No, I haven't seen any reports like this.

> I got btrfs root FS on 4 core, 1.3 GHz CPU on eMMC media. The rw performance 
> is decent I think.
> load avg. about 1.1 so no CPU is not limiting.
> sys-apps/portage:
>  Installed versions:  2.3.19-r1(17:41:24 24/01/18)(ipc native-extensions 
> xattr -build -doc -epydoc -selinux LINGUAS="-ru" PYTHON_TARGETS="python2_7 
> python3_4 -pypy -python3_5 -python3_6")
> 
> Is there some I can speed up binary pkg emerge?
> when stracing I see way too many system calls 
> but this one stands out, 1 byte reads:
> 
> 17750 16:39:40 brk(0x100d7000)  = 0x100d7000
> 17750 16:39:40 brk(0x100d6000)  = 0x100d6000
> 17750 16:39:40 read(0, "d", 1)  = 1
> 17750 16:39:40 read(0, "e", 1)  = 1
> 17750 16:39:40 read(0, "c", 1)  = 1
> 17750 16:39:40 read(0, "l", 1)  = 1
> 17750 16:39:40 read(0, "a", 1)  = 1
> 17750 16:39:40 read(0, "r", 1)  = 1
> 17750 16:39:40 read(0, "e", 1)  = 1
> 17750 16:39:40 read(0, " ", 1)  = 1
> 17750 16:39:40 read(0, "-", 1)  = 1
> 17750 16:39:40 read(0, "x", 1)  = 1
> 17750 16:39:40 read(0, " ", 1)  = 1
> 17750 16:39:40 read(0, "A", 1)  = 1
> 17750 16:39:40 read(0, "=", 1)  = 1
> 17750 16:39:40 read(0, "\"", 1) = 1
> 
> 17750 16:39:43 read(0, "t", 1)  = 1
> 17750 16:39:43 read(0, "\n", 1) = 1
> 17750 16:39:43 read(0, "}", 1)  = 1
> 17750 16:39:43 read(0, "\n", 1) = 1
> 17750 16:39:43 read(0, "", 1)   = 0
> 17750 16:39:43 write(1, "function _eapply_patch () \n "..., 4345) = 4345


I see what's causing those 1 byte reads, I've filed this bug:

https://bugs.gentoo.org/647654

-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation

2018-02-14 Thread Robin H. Johnson 
On Thu, Feb 08, 2018 at 06:09:22PM +0100, Michał Górny wrote:
> Remove the limitation that all files covered by the Manifest must reside
> on a single filesystem. This breaks valid uses of overlayfs without
> providing any real advantage.
Replying for the record:

I approve this change, because I didn't realize how overlayfs could
cause files & directories to show up in a single place with different
device IDs. Keeping overlayfs working is more important than the slight
security concerns of traversing device boundaries.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[gentoo-dev] stability of 17.0 hardened profile

2018-02-14 Thread Paweł Hajdan , Jr . 
I was looking into the new 17.0 profiles (nice work!), and noticed the
hardened one is marked as dev. I'm somewhat concerned about switching to
that on my laptop (I'm currently using hardened/linux/amd64).

Is there something I can do to help move the profile to stable?

Alternatively, is there a different profile recommended for hardened?

# eselect profile list
...
  [12]  default/linux/amd64/17.0 (stable)
  [13]  default/linux/amd64/17.0/selinux (dev)
  [14]  default/linux/amd64/17.0/hardened (dev)
  [15]  default/linux/amd64/17.0/hardened/selinux (dev)
  [16]  default/linux/amd64/17.0/desktop (stable)
  [17]  default/linux/amd64/17.0/desktop/gnome (stable)
  [18]  default/linux/amd64/17.0/desktop/gnome/systemd (stable)
  [19]  default/linux/amd64/17.0/desktop/plasma (stable)
  [20]  default/linux/amd64/17.0/desktop/plasma/systemd (stable)
  [21]  default/linux/amd64/17.0/developer (stable)
  [22]  default/linux/amd64/17.0/no-multilib (stable)
  [23]  default/linux/amd64/17.0/no-multilib/hardened (dev)
  [24]  default/linux/amd64/17.0/no-multilib/hardened/selinux (dev)
  [25]  default/linux/amd64/17.0/systemd (stable)
  [26]  default/linux/amd64/17.0/x32 (dev)
...
  [40]  hardened/linux/amd64 (stable) *

Paweł



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] The future of virtual/mysql and virtual/libmysqlclient

2018-02-14 Thread Brian Evans 
On 2/14/2018 9:35 AM, Francesco Riosa wrote:
> 
> On 14/02/2018 05:37, Robin H. Johnson wrote:
>> On Tue, Feb 13, 2018 at 09:32:32PM -0500, Brian Evans wrote:
>>> I have a plan I would like some eyes on...
>>>
>>> I want to gradually *BAN* the use of virtual/mysql and
>>> virtual/libmysqlclient as dependencies.
>> Overall I agree, but there's some slight concerns I have.
>>
>>> To accomplish this, force dev-db/mysql-connector-c to be the only souce
>>> of libmysqlclient.so.
>>>
>>> Packages that choose to support  libmariadb.so instead can include a
>>> libmariadb USE to hook up to dev-db/mariadb-connector-c that will be
>>> introduced (and they can live side-by-side).  The motivation for this
>>> could be licensing with libmariadb being LGPL instead of GPL.  This is
>>> similar to ffmpeg/libav, except the libraries can co-exist.
>> Have all the concerns about using slightly different libmysqlclient.so
>> builds been resolved? Esp for pre-built binaries (I don't know if there
>> are any left in the tree).
>>
>>> The current providers of virtual/mysql would get a new USE flag that is
>>> MASKED for all users for the transition period and pull in the lib
>>> package(s) when that USE is disabled.
>>>
>>> virtual/mysql would become a server reference for USERS only.  It would
>>> be a QA warning violation to depend directly on virtual/mysql as it can
>>> live anywhere.
>> This part worries me slightly. I do understand that mysql-embedded is
>> retired entirely, but apps that spun up their own local mysqld instance
>> would still be affected this this change.
> 
> Checked a random desktop install and found these packages depending on
> virtual/mysql:
> 
> vivo@Monfi ~ $ equery d virtual/mysql
>  * These packages depend on virtual/mysql:
> dev-db/mariadb-10.2.12 (server ? ~virtual/mysql-5.6[embedded=,static=])
> dev-libs/apr-util-1.6.1 (mysql ? =virtual/mysql-5*)
> dev-libs/cyrus-sasl-2.1.26-r11 (mysql ? virtual/mysql)
> dev-libs/redland-1.0.17-r1 (mysql ? virtual/mysql)
> kde-apps/akonadi-17.12.2 (mysql ? virtual/mysql)
> net-analyzer/net-snmp-5.7.3_p3 (mysql ? virtual/mysql)
> net-mail/mailutils-3.4 (mysql ? virtual/mysql)
> sci-mathematics/glpk-4.63 (mysql ? virtual/mysql)
dev-libs/apr-util-1.6.1 (mysql ? dev-db/mysql-connector-c:=)
dev-libs/cyrus-sasl-2.1.26-r11 (mysql ? dev-db/mysql-connector-c:=)
dev-libs/redland-1.0.17-r1 (mysql ? dev-db/mysql-connector-c:=)
kde-apps/akonadi-17.12.2 (mysql ? dev-db/mysql-connector-c:=)
net-analyzer/net-snmp-5.7.3_p3 (mysql ? dev-db/mysql-connector-c:=)
net-mail/mailutils-3.4 (mysql ? dev-db/mysql-connector-c:=)
sci-mathematics/glpk-4.63 (mysql ? dev-db/mysql-connector-c:=)


> 
> It would be interesting to know how their ${*DEPEND} should look after
> the change.
> 
> Honestly I'd also like to understand the rationale better, why this
> change is needed?
> Are the implementations of Oracle mysql and Mariadb so different that
> it's not possible anymore to consider those equivalent (for what
> packages see)?

This is needed for a few reasons:

1) Virtuals cannot rebuild dependencies when there are subslot changes
2) dev-db/mysql-5.7 and dev-db/percona-server-5.7 are prevented from
entering the tree simply due to the client library SOVERSION change.  It
is difficult for the user to swap between server versions.
3) libmariadb.so is a pthread_once library which many other packages are
not expecting.  So they may have to write things two different ways



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] The future of virtual/mysql and virtual/libmysqlclient

2018-02-14 Thread Francesco Riosa 

On 14/02/2018 05:37, Robin H. Johnson wrote:
> On Tue, Feb 13, 2018 at 09:32:32PM -0500, Brian Evans wrote:
>> I have a plan I would like some eyes on...
>>
>> I want to gradually *BAN* the use of virtual/mysql and
>> virtual/libmysqlclient as dependencies.
> Overall I agree, but there's some slight concerns I have.
>
>> To accomplish this, force dev-db/mysql-connector-c to be the only souce
>> of libmysqlclient.so.
>>
>> Packages that choose to support  libmariadb.so instead can include a
>> libmariadb USE to hook up to dev-db/mariadb-connector-c that will be
>> introduced (and they can live side-by-side).  The motivation for this
>> could be licensing with libmariadb being LGPL instead of GPL.  This is
>> similar to ffmpeg/libav, except the libraries can co-exist.
> Have all the concerns about using slightly different libmysqlclient.so
> builds been resolved? Esp for pre-built binaries (I don't know if there
> are any left in the tree).
>
>> The current providers of virtual/mysql would get a new USE flag that is
>> MASKED for all users for the transition period and pull in the lib
>> package(s) when that USE is disabled.
>>
>> virtual/mysql would become a server reference for USERS only.  It would
>> be a QA warning violation to depend directly on virtual/mysql as it can
>> live anywhere.
> This part worries me slightly. I do understand that mysql-embedded is
> retired entirely, but apps that spun up their own local mysqld instance
> would still be affected this this change.

Checked a random desktop install and found these packages depending on
virtual/mysql:

vivo@Monfi ~ $ equery d virtual/mysql
 * These packages depend on virtual/mysql:
dev-db/mariadb-10.2.12 (server ? ~virtual/mysql-5.6[embedded=,static=])
dev-libs/apr-util-1.6.1 (mysql ? =virtual/mysql-5*)
dev-libs/cyrus-sasl-2.1.26-r11 (mysql ? virtual/mysql)
dev-libs/redland-1.0.17-r1 (mysql ? virtual/mysql)
kde-apps/akonadi-17.12.2 (mysql ? virtual/mysql)
net-analyzer/net-snmp-5.7.3_p3 (mysql ? virtual/mysql)
net-mail/mailutils-3.4 (mysql ? virtual/mysql)
sci-mathematics/glpk-4.63 (mysql ? virtual/mysql)

It would be interesting to know how their ${*DEPEND} should look after
the change.

Honestly I'd also like to understand the rationale better, why this
change is needed?
Are the implementations of Oracle mysql and Mariadb so different that
it's not possible anymore to consider those equivalent (for what
packages see)?

Thanks in advance,
Francesco



signature.asc
Description: OpenPGP digital signature

[gentoo-portage-dev] emerging binary pkgs very slow on ppc32

2018-02-14 Thread Joakim Tjernlund 
We got an embedded target which we upded with binary pks(157 in this case) 
using:
 PKGDIR=/opt/fs/osappl04a-r30b-1/usr/portage/packages emerge --rebuilt-binaries 
--verbose --usepkgonly -NDu @world @cusfpv3
and this takes forever, about 1 hour

not sure where to start looking for a cause, is this known for ppc32 ?

I got btrfs root FS on 4 core, 1.3 GHz CPU on eMMC media. The rw performance is 
decent I think.
load avg. about 1.1 so no CPU is not limiting.
sys-apps/portage:
 Installed versions:  2.3.19-r1(17:41:24 24/01/18)(ipc native-extensions 
xattr -build -doc -epydoc -selinux LINGUAS="-ru" PYTHON_TARGETS="python2_7 
python3_4 -pypy -python3_5 -python3_6")

Is there some I can speed up binary pkg emerge?
when stracing I see way too many system calls 
but this one stands out, 1 byte reads:

17750 16:39:40 brk(0x100d7000)  = 0x100d7000
17750 16:39:40 brk(0x100d6000)  = 0x100d6000
17750 16:39:40 read(0, "d", 1)  = 1
17750 16:39:40 read(0, "e", 1)  = 1
17750 16:39:40 read(0, "c", 1)  = 1
17750 16:39:40 read(0, "l", 1)  = 1
17750 16:39:40 read(0, "a", 1)  = 1
17750 16:39:40 read(0, "r", 1)  = 1
17750 16:39:40 read(0, "e", 1)  = 1
17750 16:39:40 read(0, " ", 1)  = 1
17750 16:39:40 read(0, "-", 1)  = 1
17750 16:39:40 read(0, "x", 1)  = 1
17750 16:39:40 read(0, " ", 1)  = 1
17750 16:39:40 read(0, "A", 1)  = 1
17750 16:39:40 read(0, "=", 1)  = 1
17750 16:39:40 read(0, "\"", 1) = 1

17750 16:39:43 read(0, "t", 1)  = 1
17750 16:39:43 read(0, "\n", 1) = 1
17750 16:39:43 read(0, "}", 1)  = 1
17750 16:39:43 read(0, "\n", 1) = 1
17750 16:39:43 read(0, "", 1)   = 0
17750 16:39:43 write(1, "function _eapply_patch () \n "..., 4345) = 4345

 Jocke

Re: [gentoo-dev] Re: [gentoo-project] rfc: council members and appeals

2018-02-14 Thread Rich Freeman 
On Tue, Feb 13, 2018 at 5:44 PM, Ulrich Mueller  wrote:
>
> At least for QA this is quite an oversimplified description of the
> team's role. Quoting GLEP 48, first bullet point of the specification:
> "The QA team's purpose is to provide cross-team assistance in keeping
> the tree in a good state. This is done primarily by finding and
> pointing out issues to maintainers and, where necessary, taking direct
> action."
>

I would suggest that if for whatever reason we do want to impose
further restrictions on Comrel membership, that we consider that these
may not be as necessary for QA membership.

A key difference between the two groups is that as far as I'm aware
all QA actions are completely public.  If there is an issue everybody
can see that it exists, and take whatever action needed to correct it.

With Comrel there is more of a need to trust the individuals involved,
since the proceedings are not as transparent.

One thing I would suggest doing in general is to apply the same rules
to the Comrel lead as to the QA lead:

* The QA team is directed by a lead, chosen yearly by private or
public election among the members of the team, and confirmed by the
council. The QA team lead can choose one member as a deputy. The
deputy has all of his powers directly delegated from the QA team lead
and thus his actions and decisions should be considered equal to those
of the QA team lead. The deputy is directly responsible only to the QA
team lead.

* The QA lead's term expires one year after confirmation, and during
any period that the position is vacant the council may appoint an
interim lead.


Applying the same rules to Comrel would give it a bit more of a
mandate, though if the goal is some kind of independence from the
Council this policy would in fact reduce it.  I personally don't like
having multiple leadership teams in Gentoo that do not have any kind
of hierarchy, because it can lead to sustained conflict.  So, given a
choice of a directly-elected Comrel or a Comrel appointed by Council
I'd prefer the latter.

Part of me wonders if a lot of this debate is really a proxy for a
different debate: whether Comrel ought to be more or less active.  If
you're a fan of Comrel being less active then you'd want a Comrel and
Council that largely disagreed, because it meant that any action taken
by the one would probably be undone by the other, and to the degree
that members of one can't serve on the other causes manpower issues,
so much the better.  If you're a fan of Comrel being more active then
you'd want to ensure that only seriously flawed actions get undone,
and you would want to ensure that Comrel is well-manned.

Finally, I'd just like to note that as far as I'm aware there have
only been two appeals of Comrel decisions in the time I served on the
council (a number of years) and in both cases the decisions were
upheld despite all Comrel members recusing themselves from votes.
Comrel actions historically have been rare, and recusal vs non-recusal
wouldn't have made any difference (to do so the Comrel members would
have had to have voted against the previous Comrel decisions).  That
isn't necessarily a reason to not have this discussion, but IMO in
practice this hasn't been much of a historical problem.

-- 
Rich