I'ts time for me to come back to the discussion after a longer break. some personal history: I was looking for a 64bit mingw more than a year ago (unrelated to python) for Fortran development and tried out quite some mingw toolchain variants based on the mingw-w64 project. In a nutshell: the most appropriate and best documentated solution are the toolchains provided by the mingw builds project IMHO.
The next step was the idea to use this toolchain for compiling python extensions (C and Fortran) and then to try out compiling numpy and scipy with OpenBLAS. Despite the fact, that a mingw-w64 based toolchain is rock solid today the following possible issues should be considered for Python development: (1) deploy problem: mingw runtime DLLs can not be found at runtime Solution: use flags for static linking or use a dedicated 'static' GCC toolchain for compiling and linking. Both solutions should work. (2) Win32 default stack alignment incompatibility: GCC uses 16 bytes since GCC4.6, MSVC uses 4 bytes Solution: use the -mincoming-stack-boundary=2 flag for compiling. Win64 X86_64 is not affected. This issue is the major cause for segment faults on 32bit systems. (3) Import library problem: numpy distutils does not play well with mingw-w64 Solution: create a Python import library with the mingw-w64 tools. Use a patched numpy distutils. A detailed description can be found here: http://article.gmane.org/gmane.comp.python.numeric.general/56749 . (4) linking against the correct msvcrXXX Version. Solution: create a 'specs' file (howto see http://www.mingw.org/wiki/HOWTO_Use_the_GCC_specs_file ) that includes necessary informations. (5) manifest resources Solution: extend the GCC toolchain with the Manifest resource files and ensure linkage with the help of the 'specs' file. (6) Blas Lapack for numpy scipy There is no silver bullet! A trade-off between licence acceptance, performance and stability remains to be found. OpenBLAS on Win32 seems to be quite stable. Some OpenBLAS issues on Win64 can be adressed with a single threaded version of that library. On my google drive: https://drive.google.com/folderview?id=0B4DmELLTwYmldUVpSjdpZlpNM1k&usp=sharingI provide the necessary parts to try the procedures described at http://article.gmane.org/gmane.comp.python.numeric.general/56749 and http://article.gmane.org/gmane.comp.python.numeric.general/56767 Up to now I didn't find time to put a comprehensive description on the Web and to update all that stuff (MSVCR100 support for the toolchain still missing), so I add my incomplete, not yet published mingw-w64 FAQ at the end of my longish E-Mail for further discussions. Carl --- my personal mingw-w64 FAQ ========================= what is mingw-w64 ----------------- mingw-w64 is a fork of the mingw32 project - http://sourceforge.net/apps/trac/mingw-w64/wiki/History why choose mingw-w64 over mingw ------------------------------- - 32 AND 64bit support - large file support - winpthread pthreads implementation, MIT licenced. - cross compiler toolchains availabe for Linux official mingw-w64 releases --------------------------- source releases of the mingw-64 repository - http://sourceforge.net/projects/mingw-w64/files/mingw-w64/mingw-w64-release/ official mingw-w64 GCC toolchains --------------------------------- 'recommened' builds are available from the mingw-builds project http://mingw-w64.sourceforge.net/download.php#mingw-builds for example - http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.8.2/threads-posix/seh/ - http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/4.8.2/threads-posix/dwarf/ These are common combinations of exception and thread models. You can find other combinations as well. Exception handling affects C++ development. Don't ever link object code with different types of exception and/or thread handling! threads concerning the question 'where to find mingw-w64 builds' - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/7700 - http://article.gmane.org/gmane.comp.gnu.mingw.w64.general/8484 how to build a mingw-w64 based GCC toolchain on Windows ------------------------------------------------------- "mingw-builds" is a set of scripts and patches for compiling the GCC toolchain under Windows with the help of msys2 POSIX enviroment - https://github.com/niXman/mingw-builds/tree/develop recent 'mingw-builds' GCC toolchains can be downloaded from the mingw-w64 sf.net (4) what is msys2 ------------- msys2 is the successor of msys. Msys2 is necessary as enviroment for the mingw build process on Windows. - http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/ where to get precompiled mingw-w64 compiled libraries ----------------------------------------------------- recent mingw-w64 based tools and library packages together with sources and patches are available from archlinux as well as from the msys2 maintainers. - http://sourceforge.net/projects/mingw-w64-archlinux/files/ (i686: Sjlj | x86_64: SEH) - http://sourceforge.net/projects/msys2/files/REPOS/MINGW/ (i686: Dwarf | x86_64: SEH) what is a static GCC toolchain ------------------------------ GCC as well as the all neccessary libraries for a toolchain can be compiled with "-disabled-shared". This is supported by the mingw-builds scripts as an option. All the necessary object code from the GCC runtimes will be statically linked into the binaries. As a consequence the binary size will be increased in comparison to the standard toolchains. The advantage is, that there will be no dependancy to external GCC runtime libraries, so the deployment of python extensions is greatly improved. Using such a toolchain is more reliable than using -static-XXX flags. However, exception heavy C++ programms (i.e. QT) should be compiled with shared runtimes to avoid problems with exceptions handling over DLL boundaries. For building typically Python extensions a customized static GCC toolchain is the best compromise IMHO. customizations over standard mingw-builds releases -------------------------------------------------- - two dedicated GCC toolchains for both 32bit (posix threads, Dwarf exceptions) and 64 bit (posix threads, SEH exceptions) - statically build toolchain based on gcc-4.8.2 and mingw-w64 v 3.1.0 - languages: C, C++, gfortran, LTO - customized 'specs' file for MSVCR90 linkage and manifest support (MSVCR100 linkage coming soon) - additional ftime64 patch to allow winpthreads and OpenMP to work with MSVCR90 linkage - openblas-2.9rc1 with windows thread support (OpenMP disabled) included 2014-04-02 1:46 GMT+02:00 David Cournapeau <[email protected]>: > > > > On Wed, Apr 2, 2014 at 12:36 AM, Nathaniel Smith <[email protected]> wrote: > >> On Tue, Apr 1, 2014 at 11:58 PM, David Cournapeau <[email protected]> >> wrote: >> > On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith <[email protected]> wrote: >> >> >> >> On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett <[email protected] >> > >> >> wrote: >> >> > I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL >> >> > loaded via: >> >> > >> >> > hDLL = LoadLibraryEx(pathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); >> >> > >> >> > will in turn (by default) search for its dependent DLLs in their own >> >> > directory. Or maybe in the directory of the first DLL to be loaded >> >> > with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the >> >> > documentation. Looking forward to doing my tax return after this. >> >> > >> >> > But - anyway - that means that any extensions in the DLLs directory >> >> > will get their dependencies from the DLLs directory, but that is only >> >> > true for extensions in that directory. >> >> >> >> So in conclusion, if we just drop our compiled dependencies next to >> >> the compiled module files then we're good, even on older Windows >> >> versions? That sounds much simpler than previous discussions, but good >> >> news if it's true... >> > >> > >> > That does not work very well in my experience: >> > >> > - numpy has extension modules in multiple directories, so we would >> need to >> > copy the dlls in multiple subdirectories >> > - copying dlls means that windows will load that dll multiple times, >> with >> > all the ensuing problems (I don't know for MKL/OpenBlas, but we've seen >> > serious issues when doing something similar for hdf5 dll and >> pytables/h5py). >> >> We could just ship all numpy's extension modules in the same directory >> if we wanted. It would be pretty easy to stick some code at the top of >> numpy/__init__.py to load them from numpy/all_dlls/ and then slot them >> into the appropriate places in the package namespace. >> >> Of course scipy and numpy will still both have to ship BLAS etc., and >> so I guess it will get loaded at least twice in *any* binary install >> system. I'm not sure why this would be a problem (Windows, unlike >> Unix, carefully separates DLL namespaces, right?) > > > It does not really matter here. For pure blas/lapack, that may be ok > because the functions are "stateless", but I would not count on it either. > > The cleanest solution I can think of is to have 'privately shared DLL', > but that would AFAIK require patching python, so not really an option. > > > , but if it is a >> problem then it's a very fundamental one for any binaries we ship. >> >> Do the binaries we ship now have this problem? Or are we currently >> managing to statically link everything? >> > > We currently statically link everything. The main challenge is that 'new' > (>= 4) versions of mingw don't easily allow statically linking all the > mingw-related dependencies. While the options are there, everytime I tried > to do it with an official build of mingw, I had some weird, very hard to > track crashes. The other alternative that has been suggested is to build > one own's toolchain where everything is static by default. I am not sure > why that works, and that brings the risk of depending on a toolchain that > we can't really maintain. > > David > >> >> -n >> >> -- >> Nathaniel J. Smith >> Postdoctoral researcher - Informatics - University of Edinburgh >> http://vorpus.org >> _______________________________________________ >> NumPy-Discussion mailing list >> [email protected] >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > _______________________________________________ > NumPy-Discussion mailing list > [email protected] > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >
_______________________________________________ NumPy-Discussion mailing list [email protected] http://mail.scipy.org/mailman/listinfo/numpy-discussion
