On 21.10.2019 01:15, Yasuhito FUTATSUKI wrote: > On 2019/10/20 21:58, Branko Čibej wrote: >> On 20.10.2019 06:35, Yasuhito FUTATSUKI wrote: >>> On 2019/10/20 8:37, Branko Čibej wrote: >>>> On 20.10.2019 01:10, Yasuhito FUTATSUKI wrote: >>>>> On 2019/10/20 6:59, Branko Čibej wrote: >>>>>> On 19.10.2019 23:06, Branko Čibej wrote: >>>>>>> On 19.10.2019 19:55, Branko Čibej wrote: >>>>>>>> On 19.10.2019 11:45, Yasuhito FUTATSUKI wrote: >>>>>>>>> On 2019/10/18 8:39, Branko Čibej wrote: >>>>>>>>>> On 17.10.2019 23:46, Branko Čibej wrote: >>>>>>>>>>> On 17.10.2019 23:01, Daniel Shahaf wrote: >>>>>>>>>>>> Branko Čibej wrote on Thu, Oct 17, 2019 at 22:56:29 +0200: >>>>>>>>>>>>> On 17.10.2019 22:37, Daniel Shahaf wrote: >>>>>>>>>>>>>> Branko Čibej wrote on Thu, 17 Oct 2019 06:49 +00:00: >>>>>>>>>>>>>>> On 16.10.2019 10:01, Daniel Shahaf wrote: >>>>>>>>>>>>>>>> Which reminds me: it'd be nice to set up a buildbot to >>>>>>>>>>>>>>>> run the >>>>>>>>>>>>>>>> build and >>>>>>>>>>>>>>>> test process under Python 3. Any committer can edit the >>>>>>>>>>>>>>>> buildbot >>>>>>>>>>>>>>>> scripts[1], but the question is which of the buildbot >>>>>>>>>>>>>>>> slaves >>>>>>>>>>>>>>>> has Python 3 >>>>>>>>>>>>>>>> installed? >>>>>>>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Been there, done that, bought the DVD. >>>>>>>>>>>>>> Why are svnadmin_tests 69 and tree_conflict_tests 26 passing >>>>>>>>>>>>>> on that >>>>>>>>>>>>>> bot? They (currently) fail under python3. >>>>>>>>>>>>> I don't know. It's possible, I suppose, that the >>>>>>>>>>>>> activation of >>>>>>>>>>>>> the >>>>>>>>>>>>> python3 virtual environment has no effect on the test >>>>>>>>>>>>> driver. It >>>>>>>>>>>>> might >>>>>>>>>>>>> not be a bad idea to have the tests print the Python version >>>>>>>>>>>>> in the >>>>>>>>>>>>> summary and possibly in the log of every test case. >>>>>>>>>>>> What's the value of $(PYTHON) in Makefile? That's what the >>>>>>>>>>>> «check» >>>>>>>>>>>> target uses. >>>>>>>>>>> Yep, apparently that's the bug ... I'm testing script changes >>>>>>>>>>> now, >>>>>>>>>>> along >>>>>>>>>>> with r1868566 for good measure. >>>>>>>>>> I've set up this: >>>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig >>>>>>>>>> >>>>>>>>>> It will build and test the core libraries and swig-py bindings >>>>>>>>>> with >>>>>>>>>> Python 3 on the swig-py3 branch. >>>>>>>>>> >>>>>>>>>> I hope ... build #0 running as we speak. >>>>>>>>> Unfortunately build #2, which ran after upgrading SWIG and >>>>>>>>> Python, >>>>>>>>> failed >>>>>>>>> to build SWIG Python bindings because of SWIG 4.0, as reported on >>>>>>>>> SVN-4818. >>>>>>>>> >>>>>>>>> This also affects on trunk with SWIG 4.0. >>>>>>>>> (e.g. >>>>>>>>> https://ci.apache.org/builders/svn-x64-macosx-full/builds/2418) >>>>>>>>> >>>>>>>>> With attached patch on trunk >>>>>>>>> (trunk_build_with_swig4_patch.txt) and >>>>>>>>> on swig-py3 (swig_py3_build_with_swig4_patch.txt), *.so can be >>>>>>>>> produced, >>>>>>>>> but the modules don't work correctly. >>>>>>>>> >>>>>>>>> It seems they were caused by incompatibility of Python code for >>>>>>>>> proxy >>>>>>>>> object generated by SWIG, and it can not be resolved so >>>>>>>>> simple.... >>>>>>>>> (importlib module vs simply use 'import', absense of >>>>>>>>> _swig_setattr() >>>>>>>>> and _swig_getattr(), etc.) >>>>>>>> My bad. I downgraded to Swig 3.0.12 and restarted the failed >>>>>>>> builds. >>>>>>> >>>>>>> Hm, that did not really help. On the swig-py3 branch: >>>>>>> >>>>>>> /bin/sh: line 1: 62481 Segmentation fault: 11 >>>>>>> /srv/virtualenv-3/bin/python >>>>>>> /srv/buildbot/slave/svn-x64-macosx-local-python3-swig/build/subversion/bindings/swig/python/tests/run_all.py >>>>>>> >>>>>>> >>>>>>> make: *** [check-swig-py] Error 139 >>>>>>> >>>>>>> >>>>>>> So, the Python bindings seem to have crashed the Python 3 >>>>>>> interpreter on >>>>>>> macOS? >>>>>> >>>>>> Yes, they did: >>>>>> >>>>>> Process 28278 stopped >>>>>> * thread #2, queue = 'com.apple.main-thread', stop reason = >>>>>> EXC_BAD_ACCESS (code=1, address=0x48) >>>>>> frame #0: 0x00007fff373ab746 Python`PyErr_Restore + 61 >>>>>> Python`PyErr_Restore: >>>>>> -> 0x7fff373ab746 <+61>: movq 0x48(%rbx), %rdi >>>>>> 0x7fff373ab74a <+65>: movq 0x50(%rbx), %r13 >>>>>> 0x7fff373ab74e <+69>: movq 0x58(%rbx), %r12 >>>>>> 0x7fff373ab752 <+73>: movq %r15, 0x48(%rbx) >>>>>> Target 0: (Python) stopped. >>>>>> >>>>>> frame #3: 0x0000000103800fea _core.so`PyInit__core at >>>>>> core.c:40158:12 >>>>>> 40155 m = Py_InitModule((char *) SWIG_name, SwigMethods); >>>>>> 40156 #endif >>>>>> 40157 >>>>>> -> 40158 md = d = PyModule_GetDict(m); >>>>>> 40159 (void)md; >>>>>> 40160 >>>>>> 40161 SWIG_InitializeModule(0); >>>>>> (lldb) print m >>>>>> (PyObject *) $0 = 0x000000010279ad70 >>>>>> (lldb) print *m >>>>>> (PyObject) $1 = { >>>>>> ob_refcnt = 785 >>>>>> ob_type = 0x000000010026ea30 >>>>>> } >>>>> >>>>> I guess the _core module linked incompatible Python library to >>>>> execute. >>>>> >>>>> on >>>>> https://ci.apache.org/builders/svn-x64-macosx-local-python3-swig/builds/6/steps/Build/logs/stdio >>>>> >>>>> >>>>> : >>>>> [[[ >>>>> checking for compiling Python extensions... clang >>>>> checking for linking Python extensions... clang -bundle -undefined >>>>> dynamic_lookup -isysroot >>>>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk >>>>> >>>>> >>>>> -F/usr/local/opt/python/Frameworks -framework Python >>>>> checking for linking Python libraries... -bundle -undefined >>>>> dynamic_lookup -F/usr/local/opt/python/Frameworks -framework Python >>>>> ]]] >>>> >>>> >>>> Good catch. This certainly looks correct, it's where Homebrew installs >>>> python. However ... >>>> >>>> $ otool -L /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so >>>> /opt/mine/svn-swig-py3/lib/svn-python/libsvn/_core.so: >>>> /System/Library/Frameworks/Python.framework/Versions/2.7/Python >>>> (compatibility version 2.7.0, current version 2.7.10) >>>> ... >>>> >>>> >>>> That is wrong. Looks like we'll have to set DYLD_LIBRARY_PATH so >>>> that it >>>> finds the Python 3 installation. >> >> (Actually that would be DYLD_FRAMEWORK_PATH.) >> >>> ... or have to use Run-Path Dependent Libraries feature on link time, >>> if DYLD_LIBRARY_PATH cannot work well due to SIP. >>> >>> https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html >>> >>> >>> >>> The link option in log above comes from result of >>> 'build/get-py-info.py --link', so should we tweak build/get-py-info.py, >>> or add option to configure to pass Python dynamic library path? >> >> >> Using RPATH for the Python framework on macOS would be the right >> solution, but it doesn't solve the problem. It looks like we need some >> magic incantation on macOS to explicitly pick the Python 3 framework; >> using just the -F option doesn't work as expected, the path to the >> system default Python 2.7 framework is hard-coded in to the extension >> libraries. > > Umm.... it seems extension libraries actually don't need to link > "Python" dynamic library. > > e.g. > $ otool -L `python -c 'import _socket;print(_socket.__file__)'` > /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_socket.so: > > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, > current version 1252.250.1)
You're right. Fixed in r1868674, this makes the Python bindings tests work on my laptop, with both the system Python 2 and the Homebrewed Python 3. I've forced the tests to run on the buildslave. -- Brane
