On 23/06/14 22:54, Greg Dhuse wrote: > I see the same error using iPython on Linux with custom use of > multiprocessing Pool/Queue (not scikit-learn).
I have no idea about this. The problem could be in iPython, multiprocessing, libgomp (the GNU OpenMP runtime), your LAPACK or BLAS libraries, or a number of other places. I don't know. In general, only POSIX APIs are safe to use on both sides of a fork without an exec. On Python 2.7 and 3.3, multiprocessing does exactly that (fork without exec). On Python 3.4 we can tell multiprocessing to use spawn (fork and exec) instead of fork to start new processes. If we use non-POSIX APIs, multiprocessing can fail in mysterious ways on both Linux and Mac. Common offenders on Linux are libgomp, ATLAS built with OpenMP instead of pthreads, GotoBLAS2, and older versions of OpenBLAS. Common offenders on Mac are libgomp, Grand Central Dispatch, Accelerate Framework, and multimedia APIs. Accelerate Framework is only available on Mac OS X and iOS, so that is clearly not your problem. But Rich Lewis reported his problem occured on a Mac, which makes me suspicious of the well known Accelerate problem: BLAS and LAPACK are non-POSIX API. It is therefore not an error in Accelerate Framework to deadlock on calls to Accelerate subsequent to forking without also doing an exec. This behavior should be expected whenever we use BLAS and LAPACK, on any system that supports a POSIX fork. Apple has stated very clearly that they will not patch Accelerate to correct a programming errors that some inexperienced users of POSIX will do. If we use POSIX incorrectly, we cannot blame Accelerate (or any other framework or library) for subsequent failures. Sturla ------------------------------------------------------------------------------ Open source business process management suite built on Java and Eclipse Turn processes into business applications with Bonita BPM Community Edition Quickly connect people, data, and systems into organized workflows Winner of BOSSIE, CODIE, OW2 and Gartner awards http://p.sf.net/sfu/Bonitasoft _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
