Re: change in the extention importation with 3.11
On 06/12/2022 16:13, PICCA Frederic-Emmanuel wrote: There is a fix from the upstream around enum. https://github.com/boostorg/python/commit/a218babc8daee904a83f550fb66e5cb3f1cb3013 Fix enum_type_object type on Python 3.11 The enum_type_object type inherits from PyLong_Type which is not tracked by the GC. Instances doesn't have to be tracked by the GC: remove the Py_TPFLAGS_HAVE_GC flag. The Python C API documentation says: "To create a container type, the tp_flags field of the type object must include the Py_TPFLAGS_HAVE_GC and provide an implementation of the tp_traverse handler." https://docs.python.org/dev/c-api/gcsupport.html The new exception was introduced in Python 3.11 by: python/cpython#88429 an opinion ? I'd favour that being added to boost1.74. It would fix my ecflow bug (#1024911). A minor (?) fix also needed for Swig wrapping too (#1024555) -- Alastair McKinstry, GPG: 82383CE9165B347C787081A2CBE6BB4E5D9AD3A5 ph: +353 87 6847928 e: alast...@sceal.ie, im: @sceal.ie:mckinstry
Fwd: change in the extention importation with 3.11
On 06/12/2022 13:47, picca wrote:
Hello, I am trying to fix this bug
I'm debugging something similar,
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024911
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024859
I find the error message not very informative...
:~/$ python3.11
Python 3.11.0+ (main, Nov 4 2022, 09:23:33) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
import scitbx_linalg_ext
Traceback (most recent call last):
File "", line 1, in
SystemError: initialization of scitbx_linalg_ext raised unreported
exception
The problem is that python is catching a C/C++ exception.
You can debug with:
gdb /usr/bin/python3.11
...
(gdb) catch throw
(gdb) run
import scitbx_linalg_ext
...
I would like your opinion and some help in order to fix this or at
least understand what is going on.
ther extension was build with boost_python, maybe something is wrong
also with boost_python and Python3.11.
I do not know if other packages using boost_python are affected like this.
thanks for considering
Frederic
Both cctbx and ecflow are breaking on generating new enums.
A simple test case is:
```
#include
using namespace boost::python;
// enum has flags 0x42000, no traversefunction set
class CheckPt {
public:
/// NEVER - the check pt file is never saved
/// ON_TIME - the check pt file is saved periodically. specified by
checkPtInterval.
/// ALWAYS - the check pt file is saved after any state change
/// UNDEFINED - Internal use only
enum Mode { NEVER, ON_TIME, ALWAYS, UNDEFINED};
};
void inc_enum () {
enum_("CheckPt",
"CheckPt is enum that is used to control check pointing in
the `ecflow_server`_\n\n"
"- NEVER : Switches of check pointing\n"
"- ON_TIME: `check point`_ file is saved periodically,
specified by checkPtInterval. This is the default.\n"
"- ALWAYS : `check point`_ file is saved after any state
change, *not* recommended for large definitions\n"
"- UNDEFINED : None of the the above, used to provide
default argument\n"
)
.value("NEVER", CheckPt::NEVER)
.value("ON_TIME",CheckPt::ON_TIME)
.value("ALWAYS", CheckPt::ALWAYS)
.value("UNDEFINED", CheckPt::UNDEFINED)
;
}
char const *greet()
{
return "Hello world\n";
}
BOOST_PYTHON_MODULE(wrapper)
{
def ("greet", greet);
inc_enum();
}
```
What appears to be happening is a change in python3.11:
https://docs.python.org/ja/3.11/whatsnew/3.11.html
The PyType_Ready() function now raises an error if a type is
defined with the Py_TPFLAGS_HAVE_GC flag set but has no traverse
function (PyTypeObject.tp_traverse). (Contributed by Victor
Stinner in bpo-44263.)
and indeed, the enum types are created without a traverse function but
with the HAVE_GC flag set.
It doesn't at first glance look like the code for boost1.80 is different.
A Python/boost expert is now needed.
Regards
Alastair McKinstry
mckins...@debian.org
--
Alastair McKinstry,
GPG: 82383CE9165B347C787081A2CBE6BB4E5D9AD3A5
ph: +353 87 6847928 e: alast...@sceal.ie, im: @sceal.ie:mckinstry
Re: change in the extention importation with 3.11
There is a fix from the upstream around enum. https://github.com/boostorg/python/commit/a218babc8daee904a83f550fb66e5cb3f1cb3013 Fix enum_type_object type on Python 3.11 The enum_type_object type inherits from PyLong_Type which is not tracked by the GC. Instances doesn't have to be tracked by the GC: remove the Py_TPFLAGS_HAVE_GC flag. The Python C API documentation says: "To create a container type, the tp_flags field of the type object must include the Py_TPFLAGS_HAVE_GC and provide an implementation of the tp_traverse handler." https://docs.python.org/dev/c-api/gcsupport.html The new exception was introduced in Python 3.11 by: python/cpython#88429 an opinion ?
Re: change in the extention importation with 3.11
My pytango package has the same probleme... https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024078 I: pybuild base:240: cd /<>/.pybuild/cpython3_3.11_tango/build; python3.11 -m pytest tests ImportError while loading conftest '/<>/.pybuild/cpython3_3.11_tango/build/tests/conftest.py'. tests/conftest.py:3: in from tango.test_utils import state, typed_values, server_green_mode tango/__init__.py:83: in from ._tango import ( E SystemError: initialization of _tango raised unreported exception It use also the boost_python library... something is smelly around boost_python and python3.11... Cheers.
Re: change in the extention importation with 3.11
in order to debug this, I started gdb set a breakpoint in init_module_scitbx_linalg_ext then a catch throw and I end up with this backtrace Catchpoint 2 (exception thrown), 0x770a90a1 in __cxxabiv1::__cxa_throw (obj=0xb542e0, tinfo=0x772d8200 , dest=0x772c1290 ) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:81 81 ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc: Le dossier n'est pas vide. (gdb) bt #0 0x770a90a1 in __cxxabiv1::__cxa_throw (obj=0xb542e0, tinfo=0x772d8200 , dest=0x772c1290 ) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:81 #1 0x772ad089 in boost::python::throw_error_already_set () at libs/python/src/errors.cpp:61 #2 0x772b6f05 in boost::python::objects::(anonymous namespace)::new_enum_type (doc=0x0, name=0x7743ddf9 "bidiagonal_matrix_kind") at libs/python/src/object/enum.cpp:169 #3 boost::python::objects::enum_base::enum_base (this=this@entry=0x7fffcee0, name=name@entry=0x7743ddf9 "bidiagonal_matrix_kind", to_python=to_python@entry=0x7741f720 ::to_python(void const*)>, convertible=convertible@entry=0x77422e50 ::convertible_from_python(_object*)>, construct=construct@entry=0x7741fb60 ::construct(_object*, boost::python::converter::rvalue_from_python_stage1_data*)>, id=..., doc=0x0) at libs/python/src/object/enum.cpp:204 #4 0x774203cb in boost::python::enum_::enum_ (this=0x7fffcee0, name=0x7743ddf9 "bidiagonal_matrix_kind", doc=0x0) at /usr/include/boost/python/enum.hpp:45 #5 0x77428330 in scitbx::matrix::boost_python::bidiagonal_matrix_svd_decomposition_wrapper::wrap (name=name@entry=0x7743dbd0 "svd_decomposition_of_bidiagonal_matrix") at ./scitbx/linalg/boost_python/svd.cpp:19 #6 0x7741f6b0 in scitbx::matrix::boost_python::wrap_svd () at ./scitbx/linalg/boost_python/svd.cpp:66 #7 0x773f8aa3 in scitbx::matrix::boost_python::(anonymous namespace)::init_module () at ./scitbx/linalg/boost_python/linalg_ext.cpp:19 #8 0x772c13e3 in boost::function0::operator() (this=0x7fffd2b0) at ./boost/function/function_template.hpp:763 #9 boost::python::handle_exception_impl (f=...) at libs/python/src/errors.cpp:25 #10 0x772c1b69 in boost::python::handle_exception (f=) at ./boost/function/function_template.hpp:635 #11 boost::python::detail::(anonymous namespace)::init_module_in_scope (init_function=0x773f8ac0 , m=) at libs/python/src/module.cpp:24 #12 boost::python::detail::init_module (moduledef=..., init_function=0x773f8ac0 ) at libs/python/src/module.cpp:43 not crystal clear to me :)
Re: change in the extention importation with 3.11
Hi! On 07/12/2022 00:47, picca wrote: > Hello, I am trying to fix this bug > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024859 > > I find the error message not very informative... > > :~/$ python3.11 > Python 3.11.0+ (main, Nov 4 2022, 09:23:33) [GCC 12.2.0] on linux > Type "help", "copyright", "credits" or "license" for more information. import scitbx_linalg_ext > Traceback (most recent call last): >File "", line 1, in > SystemError: initialization of scitbx_linalg_ext raised unreported > exception > > I would like your opinion and some help in order to fix this or at least > understand what is going on. > > ther extension was build with boost_python, maybe something is wrong > also with boost_python and Python3.11. > > I do not know if other packages using boost_python are affected like > this. I briefly looked at #1023909 (python-pgmagick FTBFS with Python 3.11 as supported version) and found the same error. I spent some time trying to step through loading the extension with gdb and found that the failure is somewhere in amongst some boost macros associated with initialising the extension. I didn't actually manage to isolate the issue, however. Is there a wider problem with boost python that needs addressing perhaps? regards Stuart -- Stuart Prescott http://www.nanonanonano.net/ stu...@nanonanonano.net Debian Developer http://www.debian.org/ stu...@debian.org GPG fingerprint 90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
change in the extention importation with 3.11
Hello, I am trying to fix this bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024859 I find the error message not very informative... :~/$ python3.11 Python 3.11.0+ (main, Nov 4 2022, 09:23:33) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. import scitbx_linalg_ext Traceback (most recent call last): File "", line 1, in SystemError: initialization of scitbx_linalg_ext raised unreported exception I would like your opinion and some help in order to fix this or at least understand what is going on. ther extension was build with boost_python, maybe something is wrong also with boost_python and Python3.11. I do not know if other packages using boost_python are affected like this. thanks for considering Frederic
