Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 12, 2007 4:20 AM, Jarrod Millman [EMAIL PROTECTED] wrote: I will put new binaries on the sourceforge site this weekend for both NumPy 1.0.4 and SciPy 0.6.0. I should be able to find an old PIII WinXP machine around somewhere, which I will devote to building the official non-SSE2 releases from here on out. Hey, I just wanted to update everyone that the new binaries are still coming. They are taking longer to build than expected. I will make an announcement as soon as the new binaries are available. Sorry, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 11, 2007 5:58 PM, Fernando Perez [EMAIL PROTECTED] wrote: On Dec 11, 2007 6:45 PM, Ryan Krauss [EMAIL PROTECTED] wrote: Near as I can tell, this is still unresolved for people with non-sse2 machines. Is that right? Yup. Your more detailed testing seems to confirm the hunch I had at the weekend workshop that SSE2 is the culprit. Thanks for the info. It would be really great if we could somehow resolve this quickly. I have a WinXP install under Linux, but it seems to see my CPU as an Athlon X2, so that won't work. But I also have an old laptop with a dual-boot XP that's a PIII (no SSE2, and probably the oldest reasonable hardware we can expect to support). I will put new binaries on the sourceforge site this weekend for both NumPy 1.0.4 and SciPy 0.6.0. I should be able to find an old PIII WinXP machine around somewhere, which I will devote to building the official non-SSE2 releases from here on out. Cheers, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
I will put new binaries on the sourceforge site this weekend for both NumPy 1.0.4 and SciPy 0.6.0. I should be able to find an old PIII WinXP machine around somewhere, which I will devote to building the official non-SSE2 releases from here on out. Great, many thanks! Keep in mind that as Ryan's tests showed, Athlon XP chips also seem to lack the SSE2 set, and those are a bit newer than the PIII, so they might be a bit easier to find. The Athlon 64 and Athlon X2 do have SSE2, but apparently the XP doesn't. I'm sure there's a wikipedia page somewhere with a list of all the chips where SSE2 first appeared... Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
Fernando Perez wrote: I will put new binaries on the sourceforge site this weekend for both NumPy 1.0.4 and SciPy 0.6.0. I should be able to find an old PIII WinXP machine around somewhere, which I will devote to building the official non-SSE2 releases from here on out. Great, many thanks! Keep in mind that as Ryan's tests showed, Athlon XP chips also seem to lack the SSE2 set, and those are a bit newer than the PIII, so they might be a bit easier to find. The Athlon 64 and Athlon X2 do have SSE2, but apparently the XP doesn't. I'm sure there's a wikipedia page somewhere with a list of all the chips where SSE2 first appeared... I suspect the problem is that I was using an Athlon chip without SSE2 to build binaries in the past. That machine is now being used by my kids and my environment on it has been messed-up There may also be a problem with using the Microsoft compiler to build NumPy (and then calling out to ATLAS which was built with a different compiler). Issues like this have happened in the past. And, the most recent binaries of NumPy were built with the Microsoft compiler. -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 11, 2007 3:16 PM, Fernando Perez [EMAIL PROTECTED] wrote: On Dec 10, 2007 11:04 PM, David Cournapeau [EMAIL PROTECTED] wrote: On Dec 11, 2007 12:46 PM, Andrew Straw [EMAIL PROTECTED] wrote: According to the QEMU website, QEMU does not (yet) emulate SSE on x86 target, so a Windows installation on a QEMU virtual machine may be a good way to build binaries free of these issues. http://fabrice.bellard.free.fr/qemu/qemu-tech.html I tried this, this does not work (it actually emulates SSE). I went further, and managed to disable SSE support in qemu... But again, what's the point: it takes ages to compile (qemu without the hardware accelerator is slow, like ten times slower), and you will end up with a really bad atlas, since atlas optimizaton is entirely based on runtime timers, which do not make sense anymore. I mean, really, what's the point of doing all this compared to using blas/lapack from netlib ? In practice, is it really slower ? For what ? I know I don't care so much, and I am a heavy user of numpy. For certain cases the difference can be pretty dramatic, Is it if you use it on a CPU which is really different than the one used for the compilation ? For example, the default atlas on debian (built on pentium 2, no sse) is not really much faster than netlib, IMHO. ship TWO binaries of Numpy/Scipy each time: I think that in the short term, that's the only solution. The netlib one being the default, the atlas one being the optional (with CoreDuoSSE2 something in the name). It should work by default everywhere. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
Hello I think this idea is the way to go (maybe along with an ACML build, but my limited testing seemed to indicate that MKL works on AMD CPUs). In fact, I apparently proposed it about a year ago: https://svn.enthought.com/enthought/ticket/899 No takers so far... Cheers, Albert P.S. NumPy on Windows and Linux built with MKL works like a charm for me. - Original Message - From: Christopher Barker [EMAIL PROTECTED] To: Discussion of Numerical Python numpy-discussion@scipy.org Sent: Tuesday, December 11, 2007 7:28 AM Subject: Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ? Andrew Straw wrote: A function could be called at numpy import time that specifically checks for the instruction set on the CPU running Even better would be a run-time selection of the best version. I've often fantasized about an ATLAS that could do this. I think the Intel MKL has this feature (though maybe only for Intel processors). The MKL runtime is re-distributable, but somehow I doubt that we could have one person buy one copy and distribute binaries to the entire numpy-using world --- but does anyone know? http://www.intel.com/cd/software/products/asmo-na/eng/346084.htm and http://www.intel.com/cd/software/products/asmo-na/eng/266854.htm#copies -Chris ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 11, 2007 8:47 PM, Albert Strasheim [EMAIL PROTECTED] wrote: Hello I think this idea is the way to go (maybe along with an ACML build, but my limited testing seemed to indicate that MKL works on AMD CPUs). I am personally totally against it. It is one thing to support proprietary software, that's quite another to build our official binaries against it. I consider myself far from any kind of open source zealot, but that would be crossing a line I would much prefer avoiding to cross. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
At 02:32 AM 12/11/2007, you wrote: If so I'd be happy to contribute part of the purchase price, and I assume others would too. What's more, I *have* an old PIII at home. The main company I consult for is set to buy the Intel compiler and FFT lib for Windows, for the express purpose of compiling Python, numpy, and the fastest FFT for each CPU, for new products. I develop on both an old dual PIII and a new Core Duo, and there is also every other flavor in the shop. As I read it, the binaries should be distributable. The catch is, I have never compiled Python or numpy - I had seen a poster who offered help. When the company actually purchases the product I'd be glad to do it on 2-3 targets if someone can assist with the parameters. We have one consultant here who has done it on Linux. Ray Schumacher Congitive Vision 8580 Production Ave., Suite B San Diego, CA 92121 858.578.2778 http://cognitivevision.com/ -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.503 / Virus Database: 269.17.0/1180 - Release Date: 12/10/2007 2:51 PM ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 12, 2007 2:58 AM, Christopher Barker [EMAIL PROTECTED] wrote: David Cournapeau wrote: I think this idea is the way to go (maybe along with an ACML build, but my limited testing seemed to indicate that MKL works on AMD CPUs). I am personally totally against it. It is one thing to support proprietary software, that's quite another to build our official binaries against it. I consider myself far from any kind of open source zealot, but that would be crossing a line I would much prefer avoiding to cross. Interesting -- I DO consider myself a kind of Open Source Zealot -- and this doesn't bother me a bit. It would bother me a LOT if numpy could only be built against this lib, and not an Open Source one -- but I don't see this as any different than providing a binary built with the Microsoft compiler. For me it is: when using a MS compiler, you are not forcing people to use a non open source product (except maybe the C runtime). What will happen if we offer binaries using MKL ? The ATLAS will not be tested anymore on windows, it forces every developer to use the MKL to support it At least, now, with atlas problems, I can reproduce the problems. With the MKL, not so much. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 12, 2007 3:04 AM, Christopher Barker [EMAIL PROTECTED] wrote:
Fernando Perez wrote:
a simple, reasonable solution that is likely to work: ship TWO
binaries of Numpy/Scipy each time:
1. {numpy,scipy}-reference: built with the reference blas from netlib,
no atlas, period.
2. {}-atlas: built with whatever the developers have at the time,
which will likely mean these days a core 2 duo with SSE2 support.
What hardware it was built on should be indicated, so people can at
least know this fact.
I disagree -- having an atlas version that only works on recent hardware
is just asking for complaints -- I think the ONLY way to go is for the
standard binary to be universal. Instructions should be provided for
building other versions, and if third parties want to distribute
processor-dependent versions, then great, but that's an extra.
But that's the problem: it is next to impossible to build ATLAS which
works on any processor ! At least, it is not supported by ATLAS
developers, and the way it suggested did not work for me.
By the way, I've always been confused by static linking of lapack/atlas
-- it seems to me that this kind of thing is on of the best uses of
dynamic linking -- the main binary is processor dependent, and it is
linked, at runtime, with the host's processor specific lib. -- could we
do it that way:
I believe that lapack/blas are dynamically linked by default. But for
a portable solution, I don't see any other solution than dynamic
loading. The BLAS/LAPACK library would be like a plug-in, loaded at
runtime. But this requires some work, and in perticular, doing it in a
cross platform way is not trivial
The standard distro includes a universal dynamic lib.
Folks build processor-specific libs that can be dropped in to replace
the universal one if someone so desires.
Asking the users to do is just asking for new a set of problems,
IMHO. I used this approach for the rpms on ashigabou repository (they
are compiled against netlib blas/lapack, but you can change the used
libraries using runtime library path), but that's not portable. What
is needed is a true runtime system in numpy which can detect the
processor (not too difficult, although doing it for all compilers is
not trivial either), load the right library (difficult to do in a
cross platform way), and use it accordingly (not too difficult, but
requires some work).
David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
Near as I can tell, this is still unresolved for people with non-sse2 machines. Is that right? I have a student trying to get started with such a machine. Numpy is causing Python to crash. What is the easiest solution? Does he need to build numpy from source on that machine (I actually still have access to one and could do it)? Is it just Numpy or also Scipy? Here is his responses to me: Laptop - Ok Windows XP Professional, Service Pack 2 AMD Athlon 64 3400+ (ClawHammer) 1.67 GHz, 768 MB of RAM Chipset: SiS 755/755FX Southbridge: SiS LPC Bridge Instructions: MMX (+), 3DNow! (+), SSE, SSE2, x86-64 Machine 1 - Crashes Windows XP Professional, Service Pack 2 AMD Athlon XP 2000+ (Thoroughbred) 1.67 GHz, 768 MB of RAM ASUS A7V8X-X motherboard Chipset: VIA KT400 (VT8377) Southbridge: VIA VT8235 Instructions: MMX (+), 3DNow! (+), SSE Machine 2 - Crashes Windows XP Professional, Service Pack 2 AMD Athlon XP 2600+ (Barton) 1.92 GHz, 2.0 GB of RAM ASUS A7V880 motherboard Chipset: VIA KT880 Southbridge: VIA VT8237 Instructions: MMX (+), 3DNow! (+), SSE I ran the following statements on both machines which caused it to crash: import numpy numpy.test() Here is the output: Numpy is installed in C:\Python25\lib\site-packages\numpy Numpy version 1.0.4 Python version 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Int el)] Found 10/10 tests for numpy.core.defmatrix Found 36/36 tests for numpy.core.ma Found 223/223 tests for numpy.core.multiarray Found 65/65 tests for numpy.core.numeric Found 31/31 tests for numpy.core.numerictypes Found 12/12 tests for numpy.core.records Found 6/6 tests for numpy.core.scalarmath Found 14/14 tests for numpy.core.umath Found 4/4 tests for numpy.ctypeslib Found 5/5 tests for numpy.distutils.misc_util Found 1/1 tests for numpy.fft.fftpack Found 3/3 tests for numpy.fft.helper Found 9/9 tests for numpy.lib.arraysetops Found 46/46 tests for numpy.lib.function_base Found 5/5 tests for numpy.lib.getlimits Found 4/4 tests for numpy.lib.index_tricks Found 3/3 tests for numpy.lib.polynomial Found 49/49 tests for numpy.lib.shape_base Found 15/15 tests for numpy.lib.twodim_base Found 43/43 tests for numpy.lib.type_check Found 1/1 tests for numpy.lib.ufunclike Found 40/40 tests for numpy.linalg Found 2/2 tests for numpy.random Found 0/0 tests for __main__ . Sounds like the problem is the fact that my desktop computers do not support SSE2 instructions which are in the latest numpy binaries. This also explains why it works fine on the laptop which does support SSE2. On Dec 11, 2007 6:48 PM, Robert Kern [EMAIL PROTECTED] wrote: David Cournapeau wrote: On Dec 12, 2007 2:58 AM, Christopher Barker [EMAIL PROTECTED] wrote: David Cournapeau wrote: I think this idea is the way to go (maybe along with an ACML build, but my limited testing seemed to indicate that MKL works on AMD CPUs). I am personally totally against it. It is one thing to support proprietary software, that's quite another to build our official binaries against it. I consider myself far from any kind of open source zealot, but that would be crossing a line I would much prefer avoiding to cross. Interesting -- I DO consider myself a kind of Open Source Zealot -- and this doesn't bother me a bit. It would bother me a LOT if numpy could only be built against this lib, and not an Open Source one -- but I don't see this as any different than providing a binary built with the Microsoft compiler. For me it is: when using a MS compiler, you are not forcing people to use a non open source product (except maybe the C runtime). What will happen if we offer binaries using MKL ? The ATLAS will not be tested anymore on windows, it forces every developer to use the MKL to support it At least, now, with atlas problems, I can reproduce the problems. With the MKL, not so much. I agree. The official-official Win32 binaries (numpy-version-pypyversion.msi and numpy-version-pypyversion-win32.egg on the SourceForge donwload page) should be unencumbered. Other versions can be on the download page, too, but they should be named differently, like numpy-mkl-version-... . -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
Hello all, I'm not sure the licensing really makes it possible though. Numpy isn't exactly an application, but rather a development tool, so I'm not sure how Intel would feel about it being distributed. Also, it looks like they require each developer to have license, rather than only the person building the final binary -- so having the one person building the final distro may not be kosher. IANAL. It comes down to who is allowed to have the link libraries and who isn't. I doubt whether Intel's license agreement distinguishes between normal programs and development tools. If you're a developer, you need the link libraries (.lib files) to link your program against Intel MKL. According to Intel's redist.txt, you are not allowed to redistribute these files. Without these files, you can't link a new program against the Intel MKL DLLs (generally speaking). You are allowed to redistribute the DLLs (as listed in redist.txt), without having to pay any further royalties. This means that you give any user the files they need to run a program you have linked against Intel MKL. So as I see it, one NumPy developer would need to pay for Intel MKL. Cheers, Albert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
Hi, Several people reported problems with numpy 1.0.4 (See #627 and #628, but also other problems mentionned on the ML, which I cannot find). They were all solved, as far as I know, by a binary I produced (simply using mingw + netlib BLAS/LAPACK, no ATLAS). Maybe it would be good to use those instead ? (I can recompile them if there is a special thing to do to build them) cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 10, 2007 6:48 AM, David Cournapeau [EMAIL PROTECTED] wrote: Hi, Several people reported problems with numpy 1.0.4 (See #627 and #628, but also other problems mentionned on the ML, which I cannot find). They were all solved, as far as I know, by a binary I produced (simply using mingw + netlib BLAS/LAPACK, no ATLAS). Maybe it would be good to use those instead ? (I can recompile them if there is a special thing to do to build them) Do I understand correctly that you are suggesting removing ATLAS from the Windows distribution? Wouldn't this make numpy very slow? I know on RHEL5 I see a very large improvement between the basic BLAS/LAPACK and ATLAS. Perhaps we should make an alternative Windows binary available without ATLAS just for those having problems with ATLAS? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
2007/12/10, Alexander Michael [EMAIL PROTECTED]: On Dec 10, 2007 6:48 AM, David Cournapeau [EMAIL PROTECTED] wrote: Hi, Several people reported problems with numpy 1.0.4 (See #627 and #628, but also other problems mentionned on the ML, which I cannot find). They were all solved, as far as I know, by a binary I produced (simply using mingw + netlib BLAS/LAPACK, no ATLAS). Maybe it would be good to use those instead ? (I can recompile them if there is a special thing to do to build them) Do I understand correctly that you are suggesting removing ATLAS from the Windows distribution? Wouldn't this make numpy very slow? I know on RHEL5 I see a very large improvement between the basic BLAS/LAPACK and ATLAS. Perhaps we should make an alternative Windows binary available without ATLAS just for those having problems with ATLAS? That's why David proposed the netlib version of BLAS/LAPACK and not the default implementation in numpy. I would agree with David ;) Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 10, 2007, at 10:30 , Matthieu Brucher wrote: 2007/12/10, Alexander Michael [EMAIL PROTECTED]: On Dec 10, 2007 6:48 AM, David Cournapeau [EMAIL PROTECTED] wrote: Hi, Several people reported problems with numpy 1.0.4 (See #627 and #628, but also other problems mentionned on the ML, which I cannot find). They were all solved, as far as I know, by a binary I produced (simply using mingw + netlib BLAS/LAPACK, no ATLAS). Maybe it would be good to use those instead ? (I can recompile them if there is a special thing to do to build them) Do I understand correctly that you are suggesting removing ATLAS from the Windows distribution? Wouldn't this make numpy very slow? I know on RHEL5 I see a very large improvement between the basic BLAS/LAPACK and ATLAS. Perhaps we should make an alternative Windows binary available without ATLAS just for those having problems with ATLAS? That's why David proposed the netlib version of BLAS/LAPACK and not the default implementation in numpy. I would agree with David ;) Our versions of BLAS/LAPACK are f2c'd versions of the netlib 3.0 BLAS/ LAPACK (actually, of Debian's version of these -- they include several fixes that weren't upstream). So netlib's versions aren't going to be any faster, really. And netlib's BLAS is slow. Now, if there is a BLAS that's easier to compile than ATLAS on windows, that'd be improvement. -- ||\/| /--\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |[EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 10, 2007 10:59 PM, Alexander Michael [EMAIL PROTECTED] wrote: On Dec 10, 2007 6:48 AM, David Cournapeau [EMAIL PROTECTED] wrote: Hi, Several people reported problems with numpy 1.0.4 (See #627 and #628, but also other problems mentionned on the ML, which I cannot find). They were all solved, as far as I know, by a binary I produced (simply using mingw + netlib BLAS/LAPACK, no ATLAS). Maybe it would be good to use those instead ? (I can recompile them if there is a special thing to do to build them) Do I understand correctly that you are suggesting removing ATLAS from the Windows distribution? Wouldn't this make numpy very slow? I know on RHEL5 I see a very large improvement between the basic BLAS/LAPACK and ATLAS. Perhaps we should make an alternative Windows binary available without ATLAS just for those having problems with ATLAS? If you care about speed, then you should compile your own atlas anyway. I don't quite understand the discussion about speed: BLAS/LAPACK is really slower for relatively big sizes, which many people do not use. And more important, a non working, crashing, fast BLAS is much slower than a working one :) We could propose two versions, but that just makes things more complicated: on average, people using windows are less inclined to try to understand why things do not work, and when your numpy crashes, it is not obvious that ATLAS is involved (we still do not know the exact problem). David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
David M. Cooke wrote: On Dec 10, 2007, at 10:30 , Matthieu Brucher wrote: 2007/12/10, Alexander Michael [EMAIL PROTECTED]: On Dec 10, 2007 6:48 AM, David Cournapeau [EMAIL PROTECTED] wrote: Hi, Several people reported problems with numpy 1.0.4 (See #627 and #628, but also other problems mentionned on the ML, which I cannot find). They were all solved, as far as I know, by a binary I produced (simply using mingw + netlib BLAS/LAPACK, no ATLAS). Maybe it would be good to use those instead ? (I can recompile them if there is a special thing to do to build them) Do I understand correctly that you are suggesting removing ATLAS from the Windows distribution? Wouldn't this make numpy very slow? I know on RHEL5 I see a very large improvement between the basic BLAS/LAPACK and ATLAS. Perhaps we should make an alternative Windows binary available without ATLAS just for those having problems with ATLAS? That's why David proposed the netlib version of BLAS/LAPACK and not the default implementation in numpy. I would agree with David ;) Our versions of BLAS/LAPACK are f2c'd versions of the netlib 3.0 BLAS/ LAPACK (actually, of Debian's version of these -- they include several fixes that weren't upstream). So netlib's versions aren't going to be any faster, really. And netlib's BLAS is slow. Now, if there is a BLAS that's easier to compile than ATLAS on windows, that'd be improvement. The current situation is untenable. I will gladly accept a slow BLAS for an official binary that won't segfault anywhere. We can look for a faster BLAS later. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 10, 2007 4:41 PM, Robert Kern [EMAIL PROTECTED] wrote: The current situation is untenable. I will gladly accept a slow BLAS for an official binary that won't segfault anywhere. We can look for a faster BLAS later. Just to add a note to this: John Hunter and I just finished teaching a python workshop here in Boulder, and one attendee had a recurring all-out crash on WinXP. Eventually John was able to track it to a bad BLAS call, but the death was an 'illegal instruction'. We then noticed that this was on an older Pentium III laptop, and I'd be willing to bet that the problem is an ATLAS compiled with SSE2 support. The PIII chip only has plain SSE, not SSE2, and that's the kind of crash I've seen when accidentally running code compiled in my office machine (a P4) on my laptop (a similarly old PIII). It may very well be that it's OK to ship binaries with ATLAS, but just to build them without any fancy instruction support (no SSE, SSE2 or anything else of that kind, just plain x87 code). Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
An idea that occurred to me after reading Fernando's email. A function could be called at numpy import time that specifically checks for the instruction set on the CPU running and makes sure that is completely covers the instruction set available through all the various calls, including to BLAS. If this kind of thing were added, numpy could fail with a loud warning rather than dying with mysterious errors later on. The trouble would seem that you can switch your BLAS shared library without re-compiling numpy, so numpy would have to do a run-time query of ATLAS, etc. for compilation issues. Which is likely library-dependent, and furthermore, not having looked into BLAS implementations, I'm not sure that (m)any of them provide such information. Do they? Is this idea technically possible? -Andrew Fernando Perez wrote: On Dec 10, 2007 4:41 PM, Robert Kern [EMAIL PROTECTED] wrote: The current situation is untenable. I will gladly accept a slow BLAS for an official binary that won't segfault anywhere. We can look for a faster BLAS later. Just to add a note to this: John Hunter and I just finished teaching a python workshop here in Boulder, and one attendee had a recurring all-out crash on WinXP. Eventually John was able to track it to a bad BLAS call, but the death was an 'illegal instruction'. We then noticed that this was on an older Pentium III laptop, and I'd be willing to bet that the problem is an ATLAS compiled with SSE2 support. The PIII chip only has plain SSE, not SSE2, and that's the kind of crash I've seen when accidentally running code compiled in my office machine (a P4) on my laptop (a similarly old PIII). It may very well be that it's OK to ship binaries with ATLAS, but just to build them without any fancy instruction support (no SSE, SSE2 or anything else of that kind, just plain x87 code). Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
Fernando Perez wrote: On Dec 10, 2007 4:41 PM, Robert Kern [EMAIL PROTECTED] wrote: The current situation is untenable. I will gladly accept a slow BLAS for an official binary that won't segfault anywhere. We can look for a faster BLAS later. Just to add a note to this: John Hunter and I just finished teaching a python workshop here in Boulder, and one attendee had a recurring all-out crash on WinXP. Eventually John was able to track it to a bad BLAS call, but the death was an 'illegal instruction'. We then noticed that this was on an older Pentium III laptop, and I'd be willing to bet that the problem is an ATLAS compiled with SSE2 support. The PIII chip only has plain SSE, not SSE2, and that's the kind of crash I've seen when accidentally running code compiled in my office machine (a P4) on my laptop (a similarly old PIII). It may very well be that it's OK to ship binaries with ATLAS, but just to build them without any fancy instruction support (no SSE, SSE2 or anything else of that kind, just plain x87 code). I think this is what the default should be (but plain SSE allowed). However, since I have moved, the machine I was using to build official binaries has switched and that is probably at the core of the problem. Also, I've tried to build ATLAS 3.8.0 without SSE without success (when I'm on a machine that has it). It would be useful to track which binaries are giving people problems as I built the most recent ones on a VM against an old version of ATLAS (3.6.0) that has been compiled on windows for a long time. I'm happy to upload a better binary of NumPy (if I can figure out which one is giving people grief and how to create a decent one). -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
According to the QEMU website, QEMU does not (yet) emulate SSE on x86 target, so a Windows installation on a QEMU virtual machine may be a good way to build binaries free of these issues. http://fabrice.bellard.free.fr/qemu/qemu-tech.html -Andrew Travis E. Oliphant wrote: Fernando Perez wrote: On Dec 10, 2007 4:41 PM, Robert Kern [EMAIL PROTECTED] wrote: The current situation is untenable. I will gladly accept a slow BLAS for an official binary that won't segfault anywhere. We can look for a faster BLAS later. Just to add a note to this: John Hunter and I just finished teaching a python workshop here in Boulder, and one attendee had a recurring all-out crash on WinXP. Eventually John was able to track it to a bad BLAS call, but the death was an 'illegal instruction'. We then noticed that this was on an older Pentium III laptop, and I'd be willing to bet that the problem is an ATLAS compiled with SSE2 support. The PIII chip only has plain SSE, not SSE2, and that's the kind of crash I've seen when accidentally running code compiled in my office machine (a P4) on my laptop (a similarly old PIII). It may very well be that it's OK to ship binaries with ATLAS, but just to build them without any fancy instruction support (no SSE, SSE2 or anything else of that kind, just plain x87 code). I think this is what the default should be (but plain SSE allowed). However, since I have moved, the machine I was using to build official binaries has switched and that is probably at the core of the problem. Also, I've tried to build ATLAS 3.8.0 without SSE without success (when I'm on a machine that has it). It would be useful to track which binaries are giving people problems as I built the most recent ones on a VM against an old version of ATLAS (3.6.0) that has been compiled on windows for a long time. I'm happy to upload a better binary of NumPy (if I can figure out which one is giving people grief and how to create a decent one). -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
This may be a naive question, but just to be sure... If troubles building without SSE2 support on an SSE2 processor are the problem, withould the problem be addressed by purchasing an old PIII like URL:http://cgi.ebay.com/Dell-OptiPlex-GX110-Pentium-III-1GHz-40GB-256MB-DVD-XP_W0QQitemZ130180707038QQihZ003QQcategoryZ140070QQcmdZViewItem or URL:http://cgi.ebay.com/Dell-Precision-210-Pentium-III-Dual-500MHz-512MB-30GB_W0QQitemZ130181576949QQihZ003QQcategoryZ51225QQcmdZViewItem If so I'd be happy to contribute part of the purchase price, and I assume others would too. What's more, I *have* an old PIII at home. (Doesn't everybody?) Unfortunately, I have almost no experience with compiled languages. However if it would be useful, I'd be happy to try to build on my home machine (after this Friday). I would have to ask a lot of questions... Cheers, Alan Isaac ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
Andrew Straw wrote: A function could be called at numpy import time that specifically checks for the instruction set on the CPU running Even better would be a run-time selection of the best version. I've often fantasized about an ATLAS that could do this. I think the Intel MKL has this feature (though maybe only for Intel processors). The MKL runtime is re-distributable, but somehow I doubt that we could have one person buy one copy and distribute binaries to the entire numpy-using world --- but does anyone know? http://www.intel.com/cd/software/products/asmo-na/eng/346084.htm and http://www.intel.com/cd/software/products/asmo-na/eng/266854.htm#copies -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 11, 2007 11:03 AM, Fernando Perez [EMAIL PROTECTED] wrote: On Dec 10, 2007 4:41 PM, Robert Kern [EMAIL PROTECTED] wrote: The current situation is untenable. I will gladly accept a slow BLAS for an official binary that won't segfault anywhere. We can look for a faster BLAS later. Just to add a note to this: John Hunter and I just finished teaching a python workshop here in Boulder, and one attendee had a recurring all-out crash on WinXP. Eventually John was able to track it to a bad BLAS call, but the death was an 'illegal instruction'. We then noticed that this was on an older Pentium III laptop, and I'd be willing to bet that the problem is an ATLAS compiled with SSE2 support. The PIII chip only has plain SSE, not SSE2, and that's the kind of crash I've seen when accidentally running code compiled in my office machine (a P4) on my laptop (a similarly old PIII). It may very well be that it's OK to ship binaries with ATLAS, but just to build them without any fancy instruction support (no SSE, SSE2 or anything else of that kind, just plain x87 code). The problem is that this is non trivial, and unsupported. I tried to do it, asked the main author of ATLAS, but building an ATLAS which does not use any instruction set present on the CPU used for build is too difficult, and not a priority for the main author of ATLAS. The build system of ATLAS being extremely complex (code is generated by C programs themselves compiled on the fly by other C softwares), I gave up on the idea of doing it by myself. But anyway, honestly, this is kind of stupid: ATLAS needs to be compiled with the same CPU it is run for for good performances. For example, if the L1 cache is of different size, ATLAS performances are already not so good. So if you are not event using SSE/SSE2... David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 11, 2007 11:59 AM, Andrew Straw [EMAIL PROTECTED] wrote: An idea that occurred to me after reading Fernando's email. A function could be called at numpy import time that specifically checks for the instruction set on the CPU running and makes sure that is completely covers the instruction set available through all the various calls, including to BLAS. If this kind of thing were added, numpy could fail with a loud warning rather than dying with mysterious errors later on. The trouble would seem that you can switch your BLAS shared library without re-compiling numpy, so numpy would have to do a run-time query of ATLAS, etc. for compilation issues. Which is likely library-dependent, and furthermore, not having looked into BLAS implementations, I'm not sure that (m)any of them provide such information. Do they? Is this idea technically possible? It is possible, and has been done: that's how matlab did it when it used ATLAS. Now, it is not easy, and would require some changes Basically, the solution I would see would be to have a wrapper library, to which every call would be done, and the wrapper library could reroute the calls to the right, dynamically loaded library. This requires several things which are not theoretically difficult, but a pain to do right (cross platform library loader, etc...) David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 11, 2007 12:46 PM, Andrew Straw [EMAIL PROTECTED] wrote: According to the QEMU website, QEMU does not (yet) emulate SSE on x86 target, so a Windows installation on a QEMU virtual machine may be a good way to build binaries free of these issues. http://fabrice.bellard.free.fr/qemu/qemu-tech.html I tried this, this does not work (it actually emulates SSE). I went further, and managed to disable SSE support in qemu... But again, what's the point: it takes ages to compile (qemu without the hardware accelerator is slow, like ten times slower), and you will end up with a really bad atlas, since atlas optimizaton is entirely based on runtime timers, which do not make sense anymore. I mean, really, what's the point of doing all this compared to using blas/lapack from netlib ? In practice, is it really slower ? For what ? I know I don't care so much, and I am a heavy user of numpy. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 10, 2007 11:04 PM, David Cournapeau [EMAIL PROTECTED] wrote:
On Dec 11, 2007 12:46 PM, Andrew Straw [EMAIL PROTECTED] wrote:
According to the QEMU website, QEMU does not (yet) emulate SSE on x86
target, so a Windows installation on a QEMU virtual machine may be a
good way to build binaries free of these issues.
http://fabrice.bellard.free.fr/qemu/qemu-tech.html
I tried this, this does not work (it actually emulates SSE). I went
further, and managed to disable SSE support in qemu...
But again, what's the point: it takes ages to compile (qemu without
the hardware accelerator is slow, like ten times slower), and you will
end up with a really bad atlas, since atlas optimizaton is entirely
based on runtime timers, which do not make sense anymore.
I mean, really, what's the point of doing all this compared to using
blas/lapack from netlib ? In practice, is it really slower ? For what
? I know I don't care so much, and I am a heavy user of numpy.
For certain cases the difference can be pretty dramatic, but I think
there's a simple, reasonable solution that is likely to work: ship TWO
binaries of Numpy/Scipy each time:
1. {numpy,scipy}-reference: built with the reference blas from netlib,
no atlas, period.
2. {}-atlas: built with whatever the developers have at the time,
which will likely mean these days a core 2 duo with SSE2 support.
What hardware it was built on should be indicated, so people can at
least know this fact.
Just indicate that:
- The atlas version is likely faster, but fully unsupported and likely
to crash older platforms, no refunds.
- If you *really* care about performance, you should build Atlas
yourself or be 100% sure that you're using an Atlas built on the same
chip you're using, so the build-time timing and blocking choices are
actually meaningful.
That sounds like a reasonable bit of extra work (a lot easier than
building a run-time dynamic atlas) with a true payoff in terms of
stability. No?
Cheers,
f
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
