Re: [Numpy-discussion] Loading a GB file into array

2007-12-04 Thread Gael Varoquaux
On Tue, Dec 04, 2007 at 02:13:53PM +0900, David Cournapeau wrote: With recent kernels, you can get really good latency if you do it right (around 1-2 ms worst case under high load, including high IO pressure). As you can see on my page, I indeed measured less than 1ms latency on Linux under

Re: [Numpy-discussion] Loading a GB file into array

2007-12-04 Thread David Cournapeau
Gael Varoquaux wrote: On Tue, Dec 04, 2007 at 02:13:53PM +0900, David Cournapeau wrote: With recent kernels, you can get really good latency if you do it right (around 1-2 ms worst case under high load, including high IO pressure). As you can see on my page, I indeed measured less

Re: [Numpy-discussion] Loading a GB file into array

2007-12-04 Thread Andrew Straw
Hi all, I haven't done any serious testing in the past couple years, but for this particular task -- drawing frames using OpenGL without ever skipping a video update -- it is my impression that as of a few Ubuntu releases ago (Edgy?) Windows still beat linux. Just now, I have investigated on

Re: [Numpy-discussion] Loading a GB file into array

2007-12-04 Thread David Cournapeau
Andrew Straw wrote: Hi all, I haven't done any serious testing in the past couple years, but for this particular task -- drawing frames using OpenGL without ever skipping a video update -- it is my impression that as of a few Ubuntu releases ago (Edgy?) Windows still beat linux. The

[Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Cournapeau
Hi, I've just reached a first usable scipy.scons branch, so that scipy can be built entirely with scons (assuming you build numpy with scons too). You can get it from http://svn.scipy.org/svn/scipy/branches/scipy.scons. To build it, you just need to use numpy.scons branch instead of the

[Numpy-discussion] Missing file in SVN repository (test_ufunc)

2007-12-04 Thread David Cournapeau
Hi, Since revision 4528, it is not possible to run the tests for numpy.core, because of a missing file, test_ufunc.py. Since the change in the test was done in r4528 whose log refers increase of code coverage, I guess the file was not added to the repository, cheers, David

Re: [Numpy-discussion] Missing file in SVN repository (test_ufunc)

2007-12-04 Thread Stefan van der Walt
On Tue, Dec 04, 2007 at 08:13:56PM +0900, David Cournapeau wrote: Since revision 4528, it is not possible to run the tests for numpy.core, because of a missing file, test_ufunc.py. Since the change in the test was done in r4528 whose log refers increase of code coverage, I guess the

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Huard
David, I tried building the scons numpy and scipy. Numpy apparently built fine since all tests pass. For scipy, I am having problem during the build. Something seems to be wrong with libdfftpack.a I'm attaching the terminal output. Ubuntu 7.10, Xeon 64. HTH, David This is great stuff by the

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Matthieu Brucher
This is great stuff by the way, consider me a scons convert. I'm already convinced for some time. Now, we just need an egg Builder :| Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn :

Re: [Numpy-discussion] Loading a GB file into array

2007-12-04 Thread Timothy Hochberg
On Dec 4, 2007 3:05 AM, David Cournapeau [EMAIL PROTECTED] wrote: Gael Varoquaux wrote: On Tue, Dec 04, 2007 at 02:13:53PM +0900, David Cournapeau wrote: With recent kernels, you can get really good latency if you do it right (around 1-2 ms worst case under high load, including high IO

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Cournapeau
On Dec 4, 2007 11:20 PM, David Huard [EMAIL PROTECTED] wrote: David, I tried building the scons numpy and scipy. Numpy apparently built fine since all tests pass. For scipy, I am having problem during the build. Something seems to be wrong with libdfftpack.a G, this is because of a scons

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Cournapeau
On Dec 4, 2007 11:23 PM, Matthieu Brucher [EMAIL PROTECTED] wrote: This is great stuff by the way, consider me a scons convert. I'm already convinced for some time. Now, we just need an egg Builder :| No we don't, since we are still using distutils: scons really just handles compiled

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Robert Kern
David Cournapeau wrote: Some of the most interesting things I can think of which work with scons: - you can control fortran and C flags from the command line: CFLAGS and FFLAGS won't override necessary flags, only optimization flags, so you can easily play with warning, optimization

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Fernando Perez
On Dec 4, 2007 12:27 PM, Robert Kern [EMAIL PROTECTED] wrote: user-friendly. Another option is to have our Fortran compiler knowledge-base separable from the rest of the package. scons could try to import them from, say, numpy_fcompilers first and then look inside numpy.distutils if

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Robert Kern
Fernando Perez wrote: On Dec 4, 2007 12:27 PM, Robert Kern [EMAIL PROTECTED] wrote: user-friendly. Another option is to have our Fortran compiler knowledge-base separable from the rest of the package. scons could try to import them from, say, numpy_fcompilers first and then look inside

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Fernando Perez
On Dec 4, 2007 1:24 PM, Robert Kern [EMAIL PROTECTED] wrote: Fernando Perez wrote: Is this something that really needs to be a code package? Why can't this knowledge (or at least the easily overridable part of it) be packaged in one or more .conf/.ini plaintext files? In that way,

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Robert Kern
Fernando Perez wrote: On Dec 4, 2007 1:24 PM, Robert Kern [EMAIL PROTECTED] wrote: Fernando Perez wrote: Is this something that really needs to be a code package? Why can't this knowledge (or at least the easily overridable part of it) be packaged in one or more .conf/.ini plaintext files?

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Robert Kern
David Cournapeau wrote: - I have not yet tweaked fortran compiler configurations for optimizations except for gnu compilers Can you give us a brief overview about how to do this? For example, the Intel Fortran compiler's SHLINKFLAGS in scons-local/.../SCons/Tool/ifort.py are incorrect for

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Cournapeau
Robert Kern wrote: David Cournapeau wrote: Some of the most interesting things I can think of which work with scons: - you can control fortran and C flags from the command line: CFLAGS and FFLAGS won't override necessary flags, only optimization flags, so you can easily play with

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Cournapeau
Robert Kern wrote: Tweaking should be the province of the developer and the desperate. We have to be able to tweak, but we should spend more time on preventing the need for tweaking. Yes, I can only agree 100 % with you. This is maybe *the* reason why I started this whole thing.

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Cournapeau
Robert Kern wrote: David Cournapeau wrote: - I have not yet tweaked fortran compiler configurations for optimizations except for gnu compilers Can you give us a brief overview about how to do this? For example, the Intel Fortran compiler's SHLINKFLAGS in

Re: [Numpy-discussion] Missing file in SVN repository (test_ufunc)

2007-12-04 Thread David Cournapeau
Stefan van der Walt wrote: On Tue, Dec 04, 2007 at 08:13:56PM +0900, David Cournapeau wrote: Since revision 4528, it is not possible to run the tests for numpy.core, because of a missing file, test_ufunc.py. Since the change in the test was done in r4528 whose log refers increase of

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Robert Kern
David Cournapeau wrote: So to go back to your problem: if I understand correctly, what is needed is to update the scons tools. Since those are kept at one place, I think it would be safe to update them independently. But I don't understand exactly how this could be installed in the source

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Robert Kern
David Cournapeau wrote: Robert Kern wrote: David Cournapeau wrote: - I have not yet tweaked fortran compiler configurations for optimizations except for gnu compilers Can you give us a brief overview about how to do this? For example, the Intel Fortran compiler's SHLINKFLAGS in

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Cournapeau
On Dec 5, 2007 1:19 PM, Robert Kern [EMAIL PROTECTED] wrote: David Cournapeau wrote: Robert Kern wrote: David Cournapeau wrote: - I have not yet tweaked fortran compiler configurations for optimizations except for gnu compilers Can you give us a brief overview about how to do

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Robert Kern
David Cournapeau wrote: On Dec 5, 2007 1:14 PM, Robert Kern [EMAIL PROTECTED] wrote: This, too, is a workaround for the less-than-desirable situation of having numpy's sizable build infrastructure bundled with the numpy package itself. If this stuff were an entirely separate package focused

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Cournapeau
On Dec 5, 2007 1:14 PM, Robert Kern [EMAIL PROTECTED] wrote: David Cournapeau wrote: So to go back to your problem: if I understand correctly, what is needed is to update the scons tools. Since those are kept at one place, I think it would be safe to update them independently. But I don't

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Robert Kern
David Cournapeau wrote: On Dec 5, 2007 1:19 PM, Robert Kern [EMAIL PROTECTED] wrote: David Cournapeau wrote: Robert Kern wrote: David Cournapeau wrote: - I have not yet tweaked fortran compiler configurations for optimizations except for gnu compilers Can you give us a brief overview

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Cournapeau
On Dec 5, 2007 3:07 PM, Robert Kern [EMAIL PROTECTED] wrote: I don't see how that's relevant to the problem I raised. Supporting Fortran in the standard library would make the problem even worse. distutils is certainly not broken because it doesn't support Fortran. Fortran support is indeed

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread Robert Kern
David Cournapeau wrote: On Dec 5, 2007 3:07 PM, Robert Kern [EMAIL PROTECTED] wrote: I don't see how that's relevant to the problem I raised. Supporting Fortran in the standard library would make the problem even worse. distutils is certainly not broken because it doesn't support Fortran.

Re: [Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

2007-12-04 Thread David Cournapeau
On Dec 5, 2007 3:11 PM, Robert Kern [EMAIL PROTECTED] wrote: I was only reading code; I haven't tested building Fortran extensions, yet. However, using a generic link tool would be the wrong thing to do for most Fortran extensions, I think. Where does it get the correct Fortran runtime