Re: [gentoo-portage-dev] [PATCH 2/2] pym/portage/util/locale.py: add a C module to help check locale

[Brian Dolbec]

On Fri, 27 May 2016 10:40:46 -0400
"Anthony G. Basile"  wrote:

> On 5/23/16 10:25 AM, Michał Górny wrote:
> > On Mon, 23 May 2016 08:08:18 -0400
> > "Anthony G. Basile"  wrote:
> >   
> >> On 5/23/16 2:44 AM, Michał Górny wrote:  
> >>> On Sun, 22 May 2016 13:04:40 -0400
> >>> "Anthony G. Basile"  wrote:
> >>> 

> > 
> > 1. I think old versions of Python did not support named properties
> > in sys.version_info, back when Portage was written.  
> 
> I didn't need this because I found _unicode_decode() which does what I
> want.  Thanks for the clue.  BTW, why are those functions/classes in
> pym/portage/__init__.py?  All that code in there is just cluttering
> __init__.py.  Shouldn't that stuff be pulled into a separate file and
> imported cleanly?
> 

Yes, there is generally far too much code in many of the __init__.py's.
There are many with over 1K LOC

-- 
Brian Dolbec 

Re: [gentoo-portage-dev] [PATCH 2/2] pym/portage/util/locale.py: add a C module to help check locale

[Anthony G. Basile]

On 5/23/16 10:25 AM, Michał Górny wrote:
> On Mon, 23 May 2016 08:08:18 -0400
> "Anthony G. Basile"  wrote:
> 
>> On 5/23/16 2:44 AM, Michał Górny wrote:
>>> On Sun, 22 May 2016 13:04:40 -0400
>>> "Anthony G. Basile"  wrote:
>>>   
 From: "Anthony G. Basile" 

 The current method to check for a sane system locale is to use python's
 ctypes.util.find_library() to construct a full library path to the
 system libc.so and pass that path to ctypes.CDLL() so we can call
 toupper() and tolower() directly.  However, this gets bogged down in
 implementation details and fails with musl.

 We work around this design flaw in ctypes with a small python module
 written in C which provides thin wrappers to toupper() and tolower(),
 and only fall back on the current ctypes-based check when this module
 is not available.

 This has been tested on glibc, uClibc and musl systems.

 X-Gentoo-bug: 571444
 X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=571444

 Signed-off-by: Anthony G. Basile 
 ---
  pym/portage/util/locale.py   | 32 ++-
  setup.py |  6 ++-
  src/portage_c_convert_case.c | 94 
 
  3 files changed, 121 insertions(+), 11 deletions(-)
  create mode 100644 src/portage_c_convert_case.c

 diff --git a/pym/portage/util/locale.py b/pym/portage/util/locale.py
 index 2a15ea1..85ddd2b 100644
 --- a/pym/portage/util/locale.py
 +++ b/pym/portage/util/locale.py
 @@ -11,6 +11,7 @@ from __future__ import absolute_import, unicode_literals
  import locale
  import logging
  import os
 +import sys
  import textwrap
  import traceback
  
 @@ -34,18 +35,26 @@ def _check_locale(silent):
"""
The inner locale check function.
"""
 -
 -  libc_fn = find_library("c")
 -  if libc_fn is None:
 -  return None
 -  libc = LoadLibrary(libc_fn)
 -  if libc is None:
 -  return None
 +  try:
 +  from portage_c_convert_case import _c_toupper, _c_tolower
 +  libc_tolower = _c_tolower
 +  libc_toupper = _c_toupper  
>>>
>>> Now I'm being picky... but if you named the functions toupper()
>>> and tolower(), you could actually import the whole module as 'libc'
>>> and have less code!  
>>
>> I see what you're saying, and its tempting because its elegant, but I'm
>> afraid of a clash of names.  I've got a bad feeling this will get us
>> into trouble later.
>>
>> Let me play with this and see what happens.
> 
> I don't think this will be problematic since things like this happen
> in Python all the time ;-). And after all, C function names can be
> different than Python function names.

It works fine so my last set of patches adopts this approach.

> 
>>> Also it would be nice to actually make the module more generic. There
>>> are more places where we use CDLL, and all of them could eventually be
>>> supported by the module (unshare() would be much better done in C, for
>>> example).  
>>
>> Yeah I get your point here.  Let me convince myself first.
> 
> I've got a killer argument: right now we hardcode constants from Linux
> headers in the Python code!
> 
> Not that I'm asking you to actually add code for that as well. Just
> rename the module to something more generic like portage.util.libc ;-).

Well you might as well point me in this direction since I'm working on
this now.

> 
 +  except ImportError:
 +  writemsg_level("!!! Unable to import 
 portage_c_convert_case\n!!!\n",
 +  level=logging.WARNING, noiselevel=-1)  
>>>
>>> Do we really want to warn verbosely about this? I think it'd be
>>> a pretty common case for people running the git checkout.  
>>
>> This should stay.  Its good to know that the module is not being
>> imported and silently falling back on the ctypes stuff.
>>
>> 1) its only going to happen in the rare occasion that you're using
>> something like a turkish locale and can't import the module.
> 
> Wrong. This happens before the check is done, so it will be output
> every time Portage is started, also with good locale.

Right I dropped it.

> 
>> 2) people who do a git checkout should add
>> PYTHONPATH=build/lib.linux-x86_64-3.4 to their env to test the module.
>> I can add something to testpath.  Users will have to be instructed to
>> run `./setup build` and then the script shoudl read something like this
>>
>> unamem=$(uname -m)
>>
>> pythonversion=$(python --version 2>&1 | cut -c8-)
>> pythonversion=${pythonversion%\.*}
>>
>> portagedir=$(dirname ${BASH_SOURCE[0]})
>>
>> export PATH="${portagedir}/bin:${PATH}"
>>
>> export
>> PYTHONPATH="${portagedir}/build/lib.linux-${unamem}-${pythonversion}:${portagedir}/pym:${PYTHONPATH:+:}${PYTHONPATH}"
>>
>> export PYTHONWARNINGS=d,i::ImportWarning
>>
>>
>> BTW, the original code must have a bug in i

[gentoo-portage-dev] [PATCH 1/3] pym/portage/util/locale.py: fix decoding for python2 with some locales

[Anthony G. Basile]

From: "Anthony G. Basile" 

When using python2 with some locales, like turkish, chr() is passed values not 
in
range(128) which cannot be decoded as ASCII, thus throwing a UnicodeDecodeError
exception.  We use _unicode_decode() from portage.util to address this.

Signed-off-by: Anthony G. Basile 
---
 pym/portage/util/locale.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pym/portage/util/locale.py b/pym/portage/util/locale.py
index 2a15ea1..093eb86 100644
--- a/pym/portage/util/locale.py
+++ b/pym/portage/util/locale.py
@@ -15,7 +15,7 @@ import textwrap
 import traceback
 
 import portage
-from portage.util import writemsg_level
+from portage.util import _unicode_decode, writemsg_level
 from portage.util._ctypes import find_library, LoadLibrary
 
 
@@ -62,7 +62,7 @@ def _check_locale(silent):
"as LC_CTYPE in make.conf.")
msg = [l for l in textwrap.wrap(msg, 70)]
msg.append("")
-   chars = lambda l: ''.join(chr(x) for x in l)
+   chars = lambda l: ''.join(_unicode_decode(chr(x)) for x in l)
if uc != ruc:
msg.extend([
"  %s -> %s" % (chars(lc), chars(ruc)),
-- 
2.7.3

[gentoo-portage-dev] [PATCH 3/3] pym/portage/util/locale.py: add a C module to help check locale

[Anthony G. Basile]

From: "Anthony G. Basile" 

The current method to check for a sane system locale is to use python's
ctypes.util.find_library() to construct a full library path to the
system libc.so and pass that path to ctypes.CDLL() so we can call
toupper() and tolower() directly.  However, this gets bogged down in
implementation details and fails with musl.

We work around this design flaw in ctypes with a small python module
written in C which provides thin wrappers to toupper() and tolower(),
and only fall back on the current ctypes-based check when this module
is not available.

This has been tested on glibc, uClibc and musl systems.

X-Gentoo-bug: 571444
X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=571444

Signed-off-by: Anthony G. Basile 
---
 pym/portage/util/locale.py | 16 ++-
 setup.py   |  6 +++-
 src/portage_util_libc.c| 70 ++
 3 files changed, 84 insertions(+), 8 deletions(-)
 create mode 100644 src/portage_util_libc.c

diff --git a/pym/portage/util/locale.py b/pym/portage/util/locale.py
index 093eb86..5b09945 100644
--- a/pym/portage/util/locale.py
+++ b/pym/portage/util/locale.py
@@ -34,13 +34,15 @@ def _check_locale(silent):
"""
The inner locale check function.
"""
-
-   libc_fn = find_library("c")
-   if libc_fn is None:
-   return None
-   libc = LoadLibrary(libc_fn)
-   if libc is None:
-   return None
+   try:
+   from portage.util import libc
+   except ImportError:
+   libc_fn = find_library("c")
+   if libc_fn is None:
+   return None
+   libc = LoadLibrary(libc_fn)
+   if libc is None:
+   return None
 
lc = list(range(ord('a'), ord('z')+1))
uc = list(range(ord('A'), ord('Z')+1))
diff --git a/setup.py b/setup.py
index 25429bc..5ca8156 100755
--- a/setup.py
+++ b/setup.py
@@ -47,7 +47,11 @@ x_scripts = {
 # Dictionary custom modules written in C/C++ here.  The structure is
 #   key   = module name
 #   value = list of C/C++ source code, path relative to top source directory
-x_c_helpers = {}
+x_c_helpers = {
+   'portage.util.libc' : [
+   'src/portage_util_libc.c',
+   ],
+}
 
 class x_build(build):
""" Build command with extra build_man call. """
diff --git a/src/portage_util_libc.c b/src/portage_util_libc.c
new file mode 100644
index 000..00b09c2
--- /dev/null
+++ b/src/portage_util_libc.c
@@ -0,0 +1,70 @@
+/* Copyright 2005-2016 Gentoo Foundation
+ * Distributed under the terms of the GNU General Public License v2
+ */
+
+#include 
+#include 
+#include 
+
+static PyObject * _libc_tolower(PyObject *, PyObject *);
+static PyObject * _libc_toupper(PyObject *, PyObject *);
+
+static PyMethodDef LibcMethods[] = {
+   {"tolower", _libc_tolower, METH_VARARGS, "Convert to lower case using 
system locale."},
+   {"toupper", _libc_toupper, METH_VARARGS, "Convert to upper case using 
system locale."},
+   {NULL, NULL, 0, NULL}
+};
+
+#if PY_MAJOR_VERSION >= 3
+static struct PyModuleDef moduledef = {
+   PyModuleDef_HEAD_INIT,
+   "libc", /* 
m_name */
+   "Module for converting case using the system locale",   /* 
m_doc */
+   -1, /* 
m_size */
+   LibcMethods,/* 
m_methods */
+   NULL,   /* 
m_reload */
+   NULL,   /* 
m_traverse */
+   NULL,   /* 
m_clear */
+   NULL,   /* 
m_free */
+};
+#endif
+
+PyMODINIT_FUNC
+
+#if PY_MAJOR_VERSION >= 3
+PyInit_libc(void)
+{
+   PyObject *m;
+   m = PyModule_Create(&moduledef);
+   return m;
+}
+#else
+initlibc(void)
+{
+   Py_InitModule("libc", LibcMethods);
+}
+#endif
+
+
+static PyObject *
+_libc_tolower(PyObject *self, PyObject *args)
+{
+   int c;
+
+   if (!PyArg_ParseTuple(args, "i", &c))
+   return NULL;
+
+   return Py_BuildValue("i", tolower(c));
+}
+
+
+static PyObject *
+_libc_toupper(PyObject *self, PyObject *args)
+{
+   int c;
+
+   if (!PyArg_ParseTuple(args, "i", &c))
+   return NULL;
+
+   return Py_BuildValue("i", toupper(c));
+}
-- 
2.7.3

[gentoo-portage-dev] [PATCH 2/3] setup.py: add stub for building custom modules in C/C++

[Anthony G. Basile]

From: "Anthony G. Basile" 

Currently portage doesn't include any custom modules written in C/C++.
This commit introduces stub code for building such modules in setup.py.

Signed-off-by: Anthony G. Basile 
---
 setup.py | 11 ++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/setup.py b/setup.py
index 75c4bcb..25429bc 100755
--- a/setup.py
+++ b/setup.py
@@ -4,7 +4,7 @@
 
 from __future__ import print_function
 
-from distutils.core import setup, Command
+from distutils.core import setup, Command, Extension
 from distutils.command.build import build
 from distutils.command.build_scripts import build_scripts
 from distutils.command.clean import clean
@@ -30,6 +30,9 @@ import sys
 # TODO:
 # - smarter rebuilds of docs w/ 'install_docbook' and 'install_epydoc'.
 
+# Dictionary of scripts.  The structure is
+#   key   = location in filesystem to install the scripts
+#   value = list of scripts, path relative to top source directory
 x_scripts = {
'bin': [
'bin/ebuild', 'bin/egencache', 'bin/emerge', 
'bin/emerge-webrsync',
@@ -41,6 +44,10 @@ x_scripts = {
],
 }
 
+# Dictionary custom modules written in C/C++ here.  The structure is
+#   key   = module name
+#   value = list of C/C++ source code, path relative to top source directory
+x_c_helpers = {}
 
 class x_build(build):
""" Build command with extra build_man call. """
@@ -636,6 +643,8 @@ setup(
['$sysconfdir/portage/repo.postsync.d', 
['cnf/repo.postsync.d/example']],
],
 
+   ext_modules = [Extension(name=n, sources=m) for n, m in 
x_c_helpers.items()],
+
cmdclass = {
'build': x_build,
'build_man': build_man,
-- 
2.7.3

[gentoo-portage-dev] New meeting

[Alexander Berntsen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Brian approached me a while back saying we need a new meeting. So
let's do that. Sorry for taking so long.

So set your availability[0], and add your items to the agenda[1].


[0]  
[1]  
- -- 
Alexander
berna...@gentoo.org
https://secure.plaimi.net/~alexander
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCgAGBQJXSAlcAAoJENQqWdRUGk8BXxIQAPLvut/67z3uzKqHhP0aE3+R
YIPymDjKWwU6NYbRNDIhhhGNCJnDuHAjX6oEay17Li0yxlqajKqcZwoxDLjlmHyL
+oZh/oIHEesbjFY2F3awiyWlT2NYfkqvTqaliejB8iy21t1qHRFayvSUhTvh8VZm
6i6Z4zYbbFz7hU2c1utdMlfvBx0sPUS5bsKluJYexsTyLp6K8KYfx8WDOJ2efTdv
aoSrhoyEB6hMFmEUxQJdbdnSKhvNoMCnlE+fBN6/xpx3p/FnbBzQ9tNBZqhWWIwL
iQz1WyHhn+uhXyoad+4795exBb+YQen32djIyISAcCH4+kPdiWF7+XQSMZvyexFf
smZnZWwQYIr5prxqIit63yGyICFaLjqMIeas5MA0v6M0SKsAZnsRxj66eE5e6Jut
oaj9Od3RWyjuHuhsnqu2QbSyRe5DsWfKVwC8YGe64oRzFHgc2wlSZlTn0QIFtITB
XeOvKMopvHocUSF/UMgdcvmbbXROVzlOpJFB168YeMHINMffzxaGGHkY64Zg6+3o
AgWlL3vWIsBv9v0AlOn+b7ydopP6CqMeEn/ezRpmvAzRyaHpY925MvJg/b9eIh4V
asasfhkm++5ktwN5jR8dmi8d0sE6J8VTi6anLkJwNOudrW28IHIv/PO2LPfCcc5P
a+2LVaqVB0goSFCJ5PoE
=aM7+
-END PGP SIGNATURE-

Re: [gentoo-portage-dev] [PATCH v2 2/2] Colorize packages in world_sets (bug 583164)

[Alexander Berntsen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 26/05/16 17:29, Zac Medico wrote:
> Looks good to me. ACK.
Thank you. Pushed as 40cdc1c3f467ac94d3a966777eb6a0907c269550.

- -- 
Alexander
berna...@gentoo.org
https://secure.plaimi.net/~alexander
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCgAGBQJXSAZlAAoJENQqWdRUGk8BEOoQAKKrF/Bl/DpEE/7Svd8NgQVv
XJFFW6Da8zeKoJYsms3kAELaRl4UCmxrIPvfrZWpPcX7+CWtwxTf/i35agdCzo+l
u7NlVDjWuHYxydAjN9E5IWi+ds4xUU7xy4i8Vlpcawdv3R3lDmpkv59mfuDa
2ZhwC6oSi8IvqeokblafJNvn7+nvjGxl3qr+6Q2GeJEutrelf7q0/k9qfXldRYMG
PDJdYfUdNCfs9HgnCOud+LRlcAq6Qp17qFnKwQm2ODjGPigvM+87cq7wqiv1iEg0
FwMVvz1LQZGRmRQI7liRGqw5vZ3f5yZEjG0wX1hFoJU5CxsUgAQItHmtKLsjM7Vn
1+Jmi+8C7QV86lIdUJgEdFcvWBZrl1cyYH2vxzs0GZcA5lZ2+ivnMKwyUtrv7714
bvR4W5Rc/Y6laz9eNtvJ3AtIneXm73hGSwGJQbVw66J5GdRMKdXEG/Krr9HPpGdg
E8IR/6AH8r4MHZAGw49TtUzhYmexdazs4auyF05wtLMUdCccVuIY4p/tGCiOpBUL
+jR9RpfZ9LgNcegArzsygAsr8qIQxAm3BTfVajtYKz3whg48xvXmErW7+wZHXvgl
XyQsGTzVnjIPuUswUcR3l4XDikMZJ54+/M+2q/p0LaCZlOiDIYag2d91GCc+jXwm
B7pFObxaargOfPJ9I/nf
=X651
-END PGP SIGNATURE-