Re: [Numpy-discussion] [Numpy] quadruple precision
Charles R Harris charlesr.harris at gmail.com writes: The quad precision library has been there for a while, and quad precision is also supported by the Intel compiler. I don't know about MSVC. Intel has been working on adding quad precision to their hardware for several years and there is an IEEE spec for it, so some day it will be here, but it isn't here yet. It's a bit sad, I could use quad precision in FORTRAN on a VAX 25 years ago. Mind, I only needed it once ;) I suppose lack of pressing need accounts for the delay.Chuck ___ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Waiting for hardware support can last forever, and __float128 is already here. Despite being software supported, it is still reasonably fast for people who need it. The slow-down depends on a case and optimization and can be roughly from x2 (using sse) to x10 (without optimization), but you gain x2 significant digits when compared to double, see for example http://locklessinc.com/articles/classifying_floats/. This is still faster than mpfr for example. And gcc-4.6 already supports __float128 on a number of machines: i386, x86_64, ia64 and HP-UX. Also fftw now supports binary128: http://www.fftw.org/release-notes.html (although this might not be the most representative numerical software, it confirms that it is unlikely that __float128 will be ignored by the others unless hardware supported). The portability is broken for numpy.float128 anyway (as I understand, it behaves in different ways on different architectures), so adding a new type (call it, say, quad128) that properly supports binary128 shouldn't be a drawback. Later on, when the hardware support for binary128 shows up, the quad128 will be already there. Paweł. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Numpy] quadruple precision
On Mar 2, 2012 10:48 AM, Paweł Biernat pw...@wp.pl wrote: The portability is broken for numpy.float128 anyway (as I understand, it behaves in different ways on different architectures), so adding a new type (call it, say, quad128) that properly supports binary128 shouldn't be a drawback. Later on, when the hardware support for binary128 shows up, the quad128 will be already there. There's already been movement to deprecate using float128 as the name for machine-specific long doubles. This just gives even more reason. If/when someone adds __float128 support to numpy we should really just call it float128, not quad128. (This would even be backwards compatible, since float128 currently gives no guarantees on precision or representation.) - n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Numpy] quadruple precision
Le 02/03/2012 14:39, Nathaniel Smith a écrit : If/when someone adds __float128 support to numpy we should really just call it float128 I agree! Other types could become float80_128 and float80_96, as mentioned about a week ago by Matthew. -- Pierre signature.asc Description: OpenPGP digital signature ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
Hi, mmap can give a speed up in some case, but slow down in other. So care must be taken when using it. For example, the speed difference between read and mmap are not the same when the file is local and when it is on NFS. On NFS, you need to read bigger chunk to make it worthwhile. Another example is on an SMP computer. If for example you have a 8 cores computer but have only enought ram for 1 or 2 copy of your dataset, using mmap is a bad idea. If you read the file by chunk normally the OS will keep the file in its cache in ram. So if you launch 8 jobs, they will all use the system cache to shared the data. If you use mmap, I think this bypass the OS cache. So you will always read the file. On NFS with a cluster of computer, this can bring a high load on the file server. So having a way to specify to use or not to use mmap would be great as you can't always guess the right thing to do. (Except if I'm wrong and this don't by pass the OS cache) Anyway, it is great to see people work in this problem, this was just a few comments I had in mind when I read this thread. Frédéric On Sun, Feb 26, 2012 at 4:22 PM, Warren Weckesser warren.weckes...@enthought.com wrote: On Sun, Feb 26, 2012 at 3:00 PM, Nathaniel Smith n...@pobox.com wrote: On Sun, Feb 26, 2012 at 7:58 PM, Warren Weckesser warren.weckes...@enthought.com wrote: Right, I got that. Sorry if the placement of the notes about how to clear the cache seemed to imply otherwise. OK, cool, np. Clearing the disk cache is very important for getting meaningful, repeatable benchmarks in code where you know that the cache will usually be cold and where hitting the disk will have unpredictable effects (i.e., pretty much anything doing random access, like databases, which have complicated locality patterns, you may or may not trigger readahead, etc.). But here we're talking about pure sequential reads, where the disk just goes however fast it goes, and your code can either keep up or not. One minor point where the OS interface could matter: it's good to set up your code so it can use mmap() instead of read(), since this can reduce overhead. read() has to copy the data from the disk into OS memory, and then from OS memory into your process's memory; mmap() skips the second step. Thanks for the tip. Do you happen to have any sample code that demonstrates this? I'd like to explore this more. No, I've never actually run into a situation where I needed it myself, but I learned the trick from Tridge so I tend to believe it :-). mmap() is actually a pretty simple interface -- the only thing I'd watch out for is that you want to mmap() the file in pieces (so as to avoid VM exhaustion on 32-bit systems), but you want to use pretty big pieces (because each call to mmap()/munmap() has overhead). So you might want to use chunks in the 32-128 MiB range. Or since I guess you're probably developing on a 64-bit system you can just be lazy and mmap the whole file for initial testing. git uses mmap, but I'm not sure it's very useful example code. Also it's not going to do magic. Your code has to be fairly quick before avoiding a single memcpy() will be noticeable. HTH, Yes, thanks! I'm working on a mmap version now. I'm very curious to see just how much of an improvement it can give. Warren ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
Frédéric Bastien writes: Hi, mmap can give a speed up in some case, but slow down in other. So care must be taken when using it. For example, the speed difference between read and mmap are not the same when the file is local and when it is on NFS. On NFS, you need to read bigger chunk to make it worthwhile. Another example is on an SMP computer. If for example you have a 8 cores computer but have only enought ram for 1 or 2 copy of your dataset, using mmap is a bad idea. If you read the file by chunk normally the OS will keep the file in its cache in ram. So if you launch 8 jobs, they will all use the system cache to shared the data. If you use mmap, I think this bypass the OS cache. So you will always read the file. Not according to mmap(2): MAP_SHARED Share this mapping. Updates to the mapping are visible to other processes that map this file, and are carried through to the underlying file. The file may not actually be updated until msync(2) or munmap() is called. My understanding is that all processes will use exactly the same physical memory, and swapping that memory will use the file itself. On NFS with a cluster of computer, this can bring a high load on the file server. So having a way to specify to use or not to use mmap would be great as you can't always guess the right thing to do. (Except if I'm wrong and this don't by pass the OS cache) Anyway, it is great to see people work in this problem, this was just a few comments I had in mind when I read this thread. Lluis -- And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer. -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Bus error for Debian / SPARC on current trunk
Hi, Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp: nosetests ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values test_user_missing_values (test_io.TestFromTxt) ... Bus error This on current master : 1.7.0.dev-b9872b4 Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bus error for Debian / SPARC on current trunk
On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp: nosetests ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values test_user_missing_values (test_io.TestFromTxt) ... Bus error This on current master : 1.7.0.dev-b9872b4 Hmm, some tests might have been recently enabled. Any chance of doing a bisection? Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bus error for Debian / SPARC on current trunk
Hi, On Fri, Mar 2, 2012 at 9:05 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Mar 2, 2012 at 4:36 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, Sorry that this report is not complete, I don't have full access to this box but, on a Debian squeeze machine running linux 2.6.32-5-sparc64-smp: nosetests ~/usr/local/lib/python2.6/site-packages/numpy/lib/tests/test_io.py:TestFromTxt.test_user_missing_values test_user_missing_values (test_io.TestFromTxt) ... Bus error This on current master : 1.7.0.dev-b9872b4 Hmm, some tests might have been recently enabled. Any chance of doing a bisection? I'm on it - will get back to you tomorrow. See you, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
