Bug#1024271: python-cffi: FTBFS on hppa - c/test_c.py test faults and drops core

[John David Anglin]

Hi Stefano,

On 2022-11-17 3:37 p.m., Stefano Rivera wrote:

Generally speaking python-cffi test failures point to libffi bugs. So,
I'd start by trying to verify the behaviour of the underlying behaviour
in libffi, from C.

The attached patch to python-cffi fixes the test failures on hppa. Please add 
to patches
for python-cffi and push upstream if okay.

A couple of years ago the libffi port for hppa-linux was changed from using a 
trampoline
executed on the stack to the function descriptor technique used by ia64. This 
doesn't
require an executable stack and is simpler. However, function pointers need to 
have the
PLABEL bit set in the pointer. As a result, a simple cast can't be used to 
convert closure
pointers to function pointers.

python-cffi uses its own closure allocation so the problem can't be fixed in 
libffi.

I added a macro CFFI_FN to do the conversion. It shouldn't affect other 
architectures.
There is a similar define in libffi.

Regards,
Dave Anglin

--
John David Anglin  dave.ang...@bell.net
--- ./c/_cffi_backend.c.save2022-12-18 13:56:40.706730405 +
+++ ./c/_cffi_backend.c 2022-12-18 13:57:03.306944998 +
@@ -60,6 +60,13 @@
 # endif
 #endif
 
+/* Convert from closure pointer to function pointer. */
+#if defined(__hppa__) && !defined(__LP64__)
+#define CFFI_FN(f) ((void (*)(void))((unsigned int)(f) | 2))
+#else
+#define CFFI_FN(f) ((void (*)(void))f)
+#endif
+
 
 /* Define the following macro ONLY if you trust libffi's version of
  * ffi_closure_alloc() more than the code in malloc_closure.h.
@@ -3191,7 +3198,7 @@
 
 Py_BEGIN_ALLOW_THREADS
 restore_errno();
-ffi_call(_descr->cif, (void (*)(void))(cd->c_data),
+ffi_call(_descr->cif, (void (*)(void)) CFFI_FN(cd->c_data),
  resultdata, buffer_array);
 save_errno();
 Py_END_ALLOW_THREADS
@@ -6394,7 +6401,7 @@
 goto error;
 Py_INCREF(ct);
 cd->head.c_type = ct;
-cd->head.c_data = (char *)closure_exec;
+cd->head.c_data = (char *)CFFI_FN(closure_exec);
 cd->head.c_weakreflist = NULL;
 closure->user_data = NULL;
 cd->closure = closure;

Bug#1024271: python-cffi: FTBFS on hppa - c/test_c.py test faults and drops core

[John David Anglin]

On 2022-11-18 4:00 a.m., stefa...@debian.org wrote:

Hi John (2022.11.17_23:22:15_+)

If the python-cffi test checks the passing of structs larger than 8
bytes, then maybe there is a problem:

 From a quick look at the test suite, there are a lot of tests that do
that.

I submitted a patch upstream to fix the libffi test failures on hppa and 
rebuilt libffi for hppa:
https://buildd.debian.org/status/fetch.php?pkg=libffi=hppa=3.4.4-1%2Bb1=1668799491=0

The testsuite is now clean.  Unfortunately, the python-cffi test still fails in 
the same way:

dave@mx3210:~/debian/python-cffi/python-cffi-1.15.1$ gdb -c core
GNU gdb (Debian 12.1-4) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "hppa-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
    .

For help, type "help".
Type "apropos word" to search for commands related to "word".
[New LWP 25666]
Core was generated by `python3.10 -m pytest c/ testing/'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0xf8d03f94 in ?? ()
(gdb) bt
#0  0xf8d03f94 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) disass 0xf8d03f94-8,0xf8d03f94+8
Dump of assembler code from 0xf8d03f8c to 0xf8d03f9c:
   0xf8d03f8c:  break 0,0
   0xf8d03f90:  depd,z,* r18,39,44,r22
=> 0xf8d03f94:  #f8d03f90
   0xf8d03f98:  depd,z,*= r18,35,52,r22
End of assembler dump.

Dave

--
John David Anglin  dave.ang...@bell.net

Bug#1024271: python-cffi: FTBFS on hppa - c/test_c.py test faults and drops core

[John David Anglin]

On 2022-11-18 4:00 a.m., stefa...@debian.org wrote:

Also, FWIW: before this test suite crash, we had a different test
failure on hppa:https://foss.heptapod.net/pypy/cffi/-/issues/186

Probably, not relevant.  At the time of this commit,

commit 8eb2d2b05626b1cbbed100725bc440709499e8a6
Author: Moxie Bot 
Date:   Mon Feb 24 10:29:20 2020 -0500

    Revamp PA_LINUX and PA_HPUX target closures to use function descriptors.

    2020-02-23  John David Anglin 

The libffi testsuite was not clean.

Dave

--
John David Anglin  dave.ang...@bell.net

Bug#1024271: python-cffi: FTBFS on hppa - c/test_c.py test faults and drops core

[stefanor]

Hi John (2022.11.17_23:22:15_+)
> If the python-cffi test checks the passing of structs larger than 8
> bytes, then maybe there is a problem:

From a quick look at the test suite, there are a lot of tests that do
that.

Also, FWIW: before this test suite crash, we had a different test
failure on hppa: https://foss.heptapod.net/pypy/cffi/-/issues/186

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

Bug#1024271: python-cffi: FTBFS on hppa - c/test_c.py test faults and drops core

[John David Anglin]

Hi Stefano,

On 2022-11-17 3:37 p.m., Stefano Rivera wrote:

Hi John (2022.11.16_21:17:59_+)

Generally speaking python-cffi test failures point to libffi bugs. So,
I'd start by trying to verify the behaviour of the underlying behaviour
in libffi, from C.

If the python-cffi test checks the passing of structs larger than 8 bytes, then 
maybe there is a problem:
https://github.com/libffi/libffi/issues/749

The hppa libffi code passes these by value.

This seems to have changed in this commit:
commit e4c5d46b0abaa8a12e97f0e0673e3efdda59fab8
Author: Anthony Green 
Date:   Mon Sep 5 12:19:57 2022 -0400

    Pass large structs by value

Dave

--
John David Anglin  dave.ang...@bell.net

Bug#1024271: python-cffi: FTBFS on hppa - c/test_c.py test faults and drops core

[Stefano Rivera]

Hi John (2022.11.16_21:17:59_+)

Generally speaking python-cffi test failures point to libffi bugs. So,
I'd start by trying to verify the behaviour of the underlying behaviour
in libffi, from C.

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

Bug#1024271: python-cffi: FTBFS on hppa - c/test_c.py test faults and drops core

[John David Anglin]

Source: python-cffi
Version: 1.15.1-3
Severity: normal
Tags: ftbfs

Dear Maintainer,

For example, see the following build log:
https://buildd.debian.org/status/fetch.php?pkg=python-cffi=hppa=1.15.1-3=1668250642=0

Running the test under gdb, I get the following backtrace:
(gdb) set args -m pytest c/ testing/
(gdb) r
Starting program: /usr/bin/python3.10 -m pytest c/ testing/
warning: Unable to find libthread_db matching inferior's thread library, thread 
debugging will not be available.
= test session starts ==
platform linux -- Python 3.10.8, pytest-7.1.2, pluggy-1.0.0+repack
rootdir: /home/dave/debian/python-cffi/python-cffi-1.15.1
plugins: openfiles-0.5.0, mock-3.8.2, remotedata-0.3.3, arraydiff-0.5.0, 
hypothesis-6.36.0, astropy-0.10.0, astropy-header-0.2.2, doctestplus-0.12.1, 
filter-subpackage-0.1.1, cov-4.0.0
collected 2043 items

c/test_c.py [Detaching after fork from child process 23024]
[Detaching after fork from child process 23025]
[Detaching after fork from child process 23026]
 [  2%]
...[Detaching after fork from child process 23027]
.[Detaching after fork from child process 23028]
.[Detaching after fork from child process 23029]
.[Detaching after fork from child process 23030]
.
Program received signal SIGILL, Illegal instruction.
0xfaa0ef90 in ?? ()
(gdb) bt
#0  0xfaa0ef90 in ?? ()
#1  0xf55120dc in ffi_call_pa32 () at ../src/pa/linux.S:102
#2  0xf5511b08 in ffi_call (cif=, fn=,
rvalue=, avalue=) at ../src/pa/ffi.c:404
#3  0xf19fa240 in cdata_call (cd=0xf8e07cc8, args=,
kwds=) at c/_cffi_backend.c:3192
#4  0x00045a0c in _PyObject_MakeTpCall (tstate=0xf1a07614, callable=0x0,
args=, nargs=-114179252,
keywords=) at ../Objects/call.c:215
#5  0x00034af8 in _PyObject_VectorcallTstate (kwnames=0x0,
nargsf=, args=, callable=,
tstate=) at ../Include/cpython/abstract.h:112
#6  _PyObject_VectorcallTstate (kwnames=0x0, nargsf=,
args=, callable=, tstate=)
at ../Include/cpython/abstract.h:99
#7  PyObject_Vectorcall (kwnames=0x0, nargsf=,
args=, callable=)
at ../Include/cpython/abstract.h:123
#8  call_function (kwnames=0x0, oparg=,
pp_stack=, trace_info=0xf8f08088, tstate=0xf931c340)
at ../Python/ceval.c:5891
#9  _PyEval_EvalFrameDefault (tstate=0xf931c340, f=0x0,
throwflag=) at ../Python/ceval.c:4213
--Type  for more, q to quit, c to continue without paging--q
Quit

(gdb) directory /home/dave/debian/python3.10/python3.10-3.10.8/Objects
Source directories searched: 
/home/dave/debian/python3.10/python3.10-3.10.8/Objects:$cdir:$cwd
(gdb) directory /home/dave/debian/libffi/libffi-3.4.4/src/pa
Source directories searched: 
/home/dave/debian/libffi/libffi-3.4.4/src/pa:/home/dave/debian/python3.10/python3.10-3.10.8/Objects:$cdir:$cwd
(gdb) bt
#0  0xfaa0ef90 in ?? ()
#1  0xf55120dc in ffi_call_pa32 () at ../src/pa/linux.S:102
#2  0xf5511b08 in ffi_call (cif=, fn=,
rvalue=, avalue=) at ../src/pa/ffi.c:404
#3  0xf19fa240 in cdata_call (cd=0xf8e07cc8, args=,
kwds=) at c/_cffi_backend.c:3192
#4  0x00045a0c in _PyObject_MakeTpCall (tstate=0xf1a07614, callable=0x0,
args=, nargs=-114179252,
keywords=) at ../Objects/call.c:215
#5  0x00034af8 in _PyObject_VectorcallTstate (kwnames=0x0,
nargsf=, args=, callable=,
tstate=) at ../Include/cpython/abstract.h:112
#6  _PyObject_VectorcallTstate (kwnames=0x0, nargsf=,
args=, callable=, tstate=)
at ../Include/cpython/abstract.h:99
#7  PyObject_Vectorcall (kwnames=0x0, nargsf=,
args=, callable=)
at ../Include/cpython/abstract.h:123
#8  call_function (kwnames=0x0, oparg=,
pp_stack=, trace_info=0xf8f08088, tstate=0xf931c340)
at ../Python/ceval.c:5891
#9  _PyEval_EvalFrameDefault (tstate=0xf931c340, f=0x0,
throwflag=) at ../Python/ceval.c:4213
--Type  for more, q to quit, c to continue without paging--
#10 0x000e7854 in _PyEval_EvalFrame (throwflag=0, f=(-142,), tstate=0xf1a07614)
at ../Include/internal/pycore_ceval.h:46
#11 _PyEval_Vector (tstate=0xf1a07614, con=0x0, locals=,
args=, argcount=, kwnames=)
at ../Python/ceval.c:5065
#12 0x00045880 in _PyFunction_Vectorcall (func=,
stack=, nargsf=, kwnames=)
at ../Objects/call.c:342
#13 0x000455cc in PyVectorcall_Call (callable=(-142,), tuple=,
kwargs=) at ../Objects/call.c:255
#14 0x000e43d8 in do_call_core (tstate=0x0, trace_info=0xf8df9a88,
func=(-142,), callargs=,
kwdict=) at ../Python/ceval.c:5915
#15 0x0002f97c in _PyEval_EvalFrameDefault (tstate=0xf931c340, f=0x0,
throwflag=) at ../Python/ceval.c:4277
#16 0x000e7854 in _PyEval_EvalFrame (throwflag=0, f=(-142,), tstate=0xf1a07614)
at ../Include/internal/pycore_ceval.h:46
#17 _PyEval_Vector (tstate=0xf1a07614, con=0x0, locals=,
args=, argcount=, kwnames=)
at ../Python/ceval.c:5065
#18 0x00045880 in _PyFunction_Vectorcall (func=,
stack=, nargsf=, kwnames=)
at ../Objects/call.c:342
--Type  for