Re: [Numpy-discussion] vectorizing loops
On Thu, Oct 25, 2007 at 04:16:06PM -0700, Mathew Yeates wrote: Anybody know of any tricks for handling something like z[0]=1.0 for i in range(100): out[i]=func1(z[i]) z[i+1]=func2(out[i]) Something like: z[0] = 1. out = func1(z) z[1:] = func2(out[:-1]) HTH, Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] vectorizing loops
Gael Varoquaux wrote: On Thu, Oct 25, 2007 at 04:16:06PM -0700, Mathew Yeates wrote: Anybody know of any tricks for handling something like z[0]=1.0 for i in range(100): out[i]=func1(z[i]) z[i+1]=func2(out[i]) Something like: z[0] = 1. out = func1(z) z[1:] = func2(out[:-1]) This only works if func1 has no side effect on its argument. The problem boils down to whether the above algorithm is recursive or not (does z[i+1] needs z[i]). If func1 has no side effect, then your solution is fine (but then writing the thing as out[i] = func1(z[i]); z[i+1] = func2(z[i]) is not really intuitive). If func1 has side effect, then you cannot vectorize easily. cheers, David P.S: IMHO, this is one of the main limitation of numpy (or any language using arrays for speed; and this is really difficult to optimize: you need compilation, JIT or similar to solve those efficiently). ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] vectorizing loops
Gael Varoquaux wrote: On Thu, Oct 25, 2007 at 04:16:06PM -0700, Mathew Yeates wrote: Anybody know of any tricks for handling something like z[0]=1.0 for i in range(100): out[i]=func1(z[i]) z[i+1]=func2(out[i]) Something like: z[0] = 1. out = func1(z) z[1:] = func2(out[:-1]) No, that doesn't work. The way you have it, for each i0, z[i] = func2(func1(0)) What Matthew wants is this z[0] = 1.0 z[1] = func2(func1(1.0)) z[2] = func2(func1(func2(func1(1.0 ... I'm afraid that there is no fast, elegant way to do this. If you could turn func2(func1(x)) into a binary ufunc f(x, y) with an ignored y, then you could call f.accumulate(z). However, if that ufunc is not implemented in C, there's not much point. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] vectorizing loops
On Fri, Oct 26, 2007 at 01:56:26AM -0500, Robert Kern wrote: Gael Varoquaux wrote: On Thu, Oct 25, 2007 at 04:16:06PM -0700, Mathew Yeates wrote: Anybody know of any tricks for handling something like z[0]=1.0 for i in range(100): out[i]=func1(z[i]) z[i+1]=func2(out[i]) Something like: z[0] = 1. out = func1(z) z[1:] = func2(out[:-1]) No, that doesn't work. The way you have it, for each i0, z[i] = func2(func1(0)) What Matthew wants is this z[0] = 1.0 z[1] = func2(func1(1.0)) z[2] = func2(func1(func2(func1(1.0 Yes, obviously. Sorry for being dense. I can't see a fast way of doing this appart in Python. Gaël ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Best way to handle wrapped array accesses?
I've seen a few references on this, but hadn't found a proper solution... I'm doing Lattice-Boltzmann simulations with periodic boundary conditions, which always necessarily involve either padding the edges and doing additional steps, or making a wrapped array (for example, if I have an array that's N rows by M columns, and cells are moving right, I then make a wrapped array a la D = hstack( A[:,0:-1], A[:,-1:] ) (creating a temporary) and adding them. I'd like to avoid the temporary and still make the operation as fast as possible. A really elegant solution would be a wrapped array subclass that would automatically handle the indexing (C = A[:,1:M+1] + B), but I'm not certain how friendly the array class is with subclassing. What's the best way to handle this? Am I pretty much stuck with temporaries or hacked functions (add_wrapped(A, offset, B, C))? (It's probably a moot point in the long term, as once things get big I will have to use the cluster, which is probably not SciPy-friendly and it'll likely be a bit before I can convince the sys admin to let me play outside the installed software boundaries. But now I just want to know). Thanks, --VPutz ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Memory leak in ndarray
Hi all I seem to have tracked down a memory leak in the string conversion mechanism of numpy. It is demonstrated using the following code: import numpy as np a = np.array([1.0, 2.0, 3.0]) while True: b = str(a) What happens above is that is repeatedly converted to a string. The process size grow quite rapidly. Has anyone else come across this? Where do I look to try to correct it? Thanks Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in ndarray
Robert Crida wrote: Hi all I seem to have tracked down a memory leak in the string conversion mechanism of numpy. It is demonstrated using the following code: import numpy as np a = np.array([1.0, 2.0, 3.0]) while True: b = str(a) What happens above is that is repeatedly converted to a string. The process size grow quite rapidly. Has anyone else come across this? Where do I look to try to correct it? Hi Robert, I cannot reproduce this on my machine. Could you give more details (which numpy version, etc...) ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Memory leak in ndarray, more info
Hi all I recently posted about a memory leak in numpy and failed to mention the version. The leak manifests itself in numpy-1.0.3.1 but is not present in numpy-1.0.2 The following code reproduces the bug: import numpy as np a = np.array([1.0, 2.0, 3.0]) while True: b = str(a) What happens above is that is repeatedly converted to a string. The process size grow quite rapidly. Has anyone else come across this? Where do I look to try to correct it? Thanks Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in ndarray, more info
Which version Python are you using ? Matthieu 2007/10/26, Robert Crida [EMAIL PROTECTED]: Hi all I recently posted about a memory leak in numpy and failed to mention the version. The leak manifests itself in numpy-1.0.3.1 but is not present in numpy-1.0.2 The following code reproduces the bug: import numpy as np a = np.array([1.0, 2.0, 3.0]) while True: b = str(a) What happens above is that is repeatedly converted to a string. The process size grow quite rapidly. Has anyone else come across this? Where do I look to try to correct it? Thanks Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in ndarray
Hi I don't think it is a python issue because if you change the line b = str(a) to just read str(a) then the problem still occurs. Also, if you change a to be a list instead of ndarray then the problem does not occur. Cheers Robert On 10/26/07, Matthew Brett [EMAIL PROTECTED] wrote: Hi, I seem to have tracked down a memory leak in the string conversion mechanism of numpy. It is demonstrated using the following code: import numpy as np a = np.array([1.0, 2.0, 3.0]) while True: b = str(a) Would you not expect python rather than numpy to be dealing with the memory here though? Matthew ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fortran array storage question
Georg Holzmann wrote: Hallo! I found now a way to get the data: Therefore I do the following (2D example): obj = PyArray_FromDimsAndData(2, dim0, PyArray_DOUBLE, (char*)data); PyArrayObject *tmp = (PyArrayObject*)obj; tmp-flags = NPY_FARRAY; if in that example I also change the strides: int s = tmp-strides[1]; tmp-strides[0] = s; tmp-strides[1] = s * dim0[0]; Then I get in python the fortran-style array in right order. However, I don't know if this is the usual way or if it has a performance overhead ... This depends on what you are trying to do, but generally, I find that if you can afford it memory-wise, it is much faster to just get a C contiguous array if you treat your C array element per element. If you don't access element per element, then it can become much more difficult, of course (specially if you have to use several times the same parts of the memory). cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fortran array storage question
Hallo! I found now a way to get the data: Therefore I do the following (2D example): obj = PyArray_FromDimsAndData(2, dim0, PyArray_DOUBLE, (char*)data); PyArrayObject *tmp = (PyArrayObject*)obj; tmp-flags = NPY_FARRAY; if in that example I also change the strides: int s = tmp-strides[1]; tmp-strides[0] = s; tmp-strides[1] = s * dim0[0]; Then I get in python the fortran-style array in right order. However, I don't know if this is the usual way or if it has a performance overhead ... Thanks, LG Georg ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] vectorizing loops
On 10/26/07, David Cournapeau [EMAIL PROTECTED] wrote: P.S: IMHO, this is one of the main limitation of numpy (or any language using arrays for speed; and this is really difficult to optimize: you need compilation, JIT or similar to solve those efficiently). This is where the scipy - sandbox numexpr project comes in - if I'm not misaken http://www.scipy.org/SciPyPackages/NumExpr Description The scipy.sandbox.numexpr package supplies routines for the fast evaluation of array expressions elementwise by using a vector-based virtual machine. It's comparable to scipy.weave.blitz (in Weave), but doesn't require a separate compile step of C or C++ code. I hope that more noise around this will result in more interest and subsequentially result in more support. I think numexpr might be one of the most powerful ideas in numpy / scipy recently. Did you know about numexpr - David ? Cheers, Sebastian Haase ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in ndarray
Robert Crida wrote: Hi I don't think it is a python issue because if you change the line b = str(a) to just read str(a) then the problem still occurs. Also, if you change a to be a list instead of ndarray then the problem does not occur. How do you know there is a memory leak ? cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] [Announce] numpy.scons , ALPHA version
Hi there,
I've finally managed to implement most of the things I wanted for
numpy.scons, hence a first alpha. Compared to the 2d milestone from a
few days ago, a few optimized libraries are supported (ATLAS, Sun
sunperf, Apple Accelerate and vecLib, Intel MKL).
Who
===
Outside people interested in numpy.scons, people who have trouble
building numpy may want to try this. In particular, using MKL or sunperf
should be easier (for sunperf, it should work out of the box if sun
compilers are used). Also, compiling with intel compilers on linux
should be easier.
DO NOT USE THIS FOR PRODUCTION USE ! I am interested in success (that is
build and all the test pass) as well as failure.
Numpy developers: I would be interested in comments about the code:
- the package are built with the setupscons.py/SConstruct
- most of the new code is in numpy/distutils/scons
How to test
===
Grab the sources:
svn co http://svn.scipy.org/svn/numpy/branches/numpy.scons
And then just do: python setup.py scons; python setup.py install. If you
have multiple CPU/Cores, you may want to use the parallel build options:
python setup.py scons --jobs=N with N an integer (number of CPU/CORES +
1 is a rule of thumb).
For non default compilers, use the --compiler and --fcompiler options
(distutils names).
For the MKL, you should add a mkl option in the site.cfg. For ATLAS or
sunperf, if it is not in standard places, it won't work (yet).
What works:
===
The tested platforms have been tested:
- linux (ubuntu x86) with gcc, g77 and ATLAS.
- linux (x86 and x64) with gcc, without BLAS or LAPACK.
- linux (ubuntu x86) with gcc, g77, and mkl (9.1.023).
- linux (ubuntu x86) with intel compilers.
- windows (32 bits) with mingw.
- windows (32 bits) with VS 2003.
- solaris (solaris studio express 12) with sunperf on x86.
- Mac Os X Intel with Accelerate framework.
- People reported basic success on IRIX, too.
What does NOT work
==
One important target missing is windows 64, but this should not be too
difficult to solve.
There are still many corner cases not yet solved (in particular some
windows things, most libraries path cannot yet be overriden in
site.cfg); also, I do not attempt to be 100 % backward compatible with
the current build system (that for a given environment, you won't
necessarily have the same build options). Some things are not yet
supported either (overriding libraries with env variables, for example,
is not supported).
There is no real doc yet (I will do this once the basic public API is
stabilized).
Details
===
Library checkers
The main difference compared to the second milestone is the support for
libraries checkers (think BLAS, Atlas, etc...). There is now support to
check for different libraries, with different compilers. This
effectively replaces numpy.distutils.system_info. The way I've
implemented those checks are fundamentally different compared to the
distutils way: instead of looking for files, I look for capabilities at
runtime.
For example, for mkl, instead of looking for libmkl.so, etc... I try to
compile, link and run a small mkl program: if any of the step failed,
then the mkl is not used. This means that I can detect config errors
much sooner; also, since I use rpath, there is no need to set things
like LD_LIBRARY_PATH and so on. If the path is set right in site.cfg,
then everything is taken care of.
Also, as before, a general library checker for new libraries is
available: look at basic examples in numpy/scons_fake/. In particular,
there is a NumpyCheckLib which can be used to check a library with some
symbols, whose paths can be overriden with a site.cfg.
Fortran support
---
The other big work has been on fortran support. Here too, the way I
implemented it is different than numpy.distutils. Instead of harcoding
flags depending on versions, I use code snipets, to get link options,
mangling, and so on. For example, if you call the following in a Sconscript:
config.CheckF77Mangling()
This will try to get the mangling of the F77 compiler, and if
sucessfull, will set a python function mangle, which returns the fortran
name from the 'common' name. For example, for g77, mangle('dgemm') will
returns dgemm_. Since the mangling is found at runtime, this can be used
for not yet known compilers.
Next
This was quite a lot of work, and I wanted to get something working as
fast as possible. As such, the code is not always clean. Once I am
satisfied with the global 'feel' of the new build, I will sanitize the
API and implement the doc. Still, I believe the code to be much easier,
simpler and robust than distutils (everything, from fortran support to
libraries checkers is less than 2000 lines of python code, including
tests; of course, this is largely thanks to scons).
The thing I am the less sure about is how to combine checkers in
meta-checkers:
Re: [Numpy-discussion] fortran array storage question
Hallo! This depends on what you are trying to do, but generally, I find that if you can afford it memory-wise, it is much faster to just get a C contiguous array if you treat your C array element per element. If you Yes, but the problem is that this data is very big (up to my memory limit) and in python I only want to read it for debugging purpose. And if I want to make a copy to a C contiguous array I can always do it like: my_cstyle_array = my_method().copy() Thanks, LG Georg ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in ndarray
On 10/26/07, Robert Crida [EMAIL PROTECTED] wrote: Hi again I watch the VmSize of the process using eg top or ps If a is a list then it remains constant. If a is an ndarray as shown in the example, then the VmSize grows quite rapidly. Actually, I did a typo while copying your example. I can confirm the memory leak (happen in PyObject_Malloc). cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] vectorizing loops
On 10/26/07, Sebastian Haase [EMAIL PROTECTED] wrote: On 10/26/07, David Cournapeau [EMAIL PROTECTED] wrote: P.S: IMHO, this is one of the main limitation of numpy (or any language using arrays for speed; and this is really difficult to optimize: you need compilation, JIT or similar to solve those efficiently). This is where the scipy - sandbox numexpr project comes in - if I'm not misaken http://www.scipy.org/SciPyPackages/NumExpr Description The scipy.sandbox.numexpr package supplies routines for the fast evaluation of array expressions elementwise by using a vector-based virtual machine. It's comparable to scipy.weave.blitz (in Weave), but doesn't require a separate compile step of C or C++ code. I hope that more noise around this will result in more interest and subsequentially result in more support. I think numexpr might be one of the most powerful ideas in numpy / scipy recently. Did you know about numexpr - David ? Sadly, I don't think numexpr will help here; It basically handles the same cases as numpy; only faster because it can avoid a lot of temporaries. I think he's going to need Psyco, Pyrex, Weave or similar. -- . __ . |-\ . . [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in ndarray
On 10/26/07, David Cournapeau [EMAIL PROTECTED] wrote: On 10/26/07, Robert Crida [EMAIL PROTECTED] wrote: Hi again I watch the VmSize of the process using eg top or ps If a is a list then it remains constant. If a is an ndarray as shown in the example, then the VmSize grows quite rapidly. Actually, I did a typo while copying your example. I can confirm the memory leak (happen in PyObject_Malloc). The problem *may* be within the ufunc machinery. I am still investigating, but since we can reproduce the problem with such a simple code, it should not be difficult to track down the problem with high level tools such as valgrind. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in ndarray
I can confirm the same behaviour with numpy '1.0.4.dev4271' on OS X 10.4with python 2.5.1 (installer from python.org). For me the memory used by the python process grows at about 1MB/sec. The memory isn't released when the loop is canceled. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Announce] numpy.scons , ALPHA version
What does NOT work == One important target missing is windows 64, but this should not be too difficult to solve. There are still many corner cases not yet solved (in particular some windows things, most libraries path cannot yet be overriden in site.cfg); also, I do not attempt to be 100 % backward compatible with the current build system (that for a given environment, you won't necessarily have the same build options). Some things are not yet supported either (overriding libraries with env variables, for example, is not supported). I would add that MSVC 2005 (windows32) does not work either as it needs a modification in the SharedLibrary builder (which I can provide if you can check the version of MSVC). This would enable people that cannot compile extensions with distutils to compile them with a compiler they can freely download on the net. -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fortran array storage question
Anne Archibald wrote: On 26/10/2007, Georg Holzmann [EMAIL PROTECTED] wrote: if in that example I also change the strides: int s = tmp-strides[1]; tmp-strides[0] = s; tmp-strides[1] = s * dim0[0]; Then I get in python the fortran-style array in right order. This is the usual way. More or less, at least. numpy is designed from the start to handle arrays with arbitrary striding; this is how slices are implemented, for example. There will be no major performance hit from numpy code itself. The actual organization of data in memory will of course affect the speed at which your code runs. The flags, as you discovered, are just a performance optimization, so that code that needs arrays organized as C- or FORTRAN-standard doesn't need to check the strides every time. I don't think numpy's loops - for example in ones((100,100))+eye(100) - are smart about doing operations in an order that makes cache-coherent use of memory. The important exception is the loops that use ATLAS, which I think is mostly the dot() function. There is an optimization where-in the inner-loops are done over the dimension with the smallest stride. What other cache-coherent optimizations do you recommend? -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C-API for non-contiguous arrays
David Cournapeau wrote: Oliver Kranz wrote: Hi, I am working on a Python extension module using of the NumPy C-API. The extension module is an interface to an image processing and analysis library written in C++. The C++ functions are exported with boos::python. Currently I am implementing the support of three-dimensional data sets which can consume a huge amount of memory. The 3D data is stored in a numpy.ndarray. This array is passed to C++ functions which do the calculations. In general, multi-dimensional arrays can be organized in memory in four different ways: 1. C order contiguous 2. Fortran order contiguous 3. C order non-contiguous 4. Fortran order non-contiguous Am I right that the NumPy C-API can only distinguish between three ways the array is organized in memory? These are: 1. C order contiguous e.g. with PyArray_ISCONTIGUOUS(arr) 2. Fortran order contiguous e.g. with PyArray_ISFORTRAN(arr) 3. non-contiguous e.g. with !PyArray_ISCONTIGUOUS(arr) !PyArray_ISFORTRAN(arr) So there is no way to find out if a non-contiguous array has C order or Fortran order. The same holds for Python code e.g. by use of the flags. a.flags.contiguous a.flags.fortran This is very important for me because I just want to avoid to copy every non-contiguous array into a contiguous array. This would be very inefficient. But I can't find an other solution than copying the array. It is inefficient depending on what you mean by inefficient. Memory-wise, copying is obviously inefficient. But speed-wise, copying the array into a contiguous array in C order is faster in most if not all cases, because of memory access times. You may want to read the following article from Ulrich Drepper on memory and cache: http://lwn.net/Articles/252125/ That's an interesting note. We already thought about this. At the moment, we decided to consequently avoid copying in our apecial case. It's not unusal to work with data sets consuming about 1 GB of memory. In the case of arrays not being in contiguous C order we have to live with the inefficiency in speed. Cheers, Oliver ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Request for advice: project to get NumPy working in IronPython
Travis E. Oliphant wrote: Giles Thomas wrote: Hi, At Resolver Systems, we have a product that is written in IronPython - the .NET Python implementation - and allows users to use that language to script a spreadsheet-like interface. Because they're using IronPython, they can access their existing .NET objects and libraries, which has worked out really well for us and for them. But there's an increasing number of users who would like access to CPython C extensions - in particular, NumPy. An IronPython compatible version of NumPy would be great.Of course it could be done by using C# to write NumPy, but I'm not sure that this would really be any less work than creating a glue layer that allowed most (or all) C-Python extensions to work with IronPython. So can anyone inform will IronPython have bridge to NumPy or something else? And will it be available in nearest future or some weeks/months/years are required? (I'm interested in scikits.openopt being available for IronPython as well, and it requires numpy) And what about pylab for IronPython? Is it work already now, or will be available in nearest future, or the situation is undefined? Regards, D. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Request for advice: project to get NumPy working in IronPython
An IronPython compatible version of NumPy would be great.Of course it could be done by using C# to write NumPy, but I'm not sure that this would really be any less work than creating a glue layer that allowed most (or all) C-Python extensions to work with IronPython. I'm curious about why all the discussion is about putting Python and its extensions on top of C# and very little discussion about just using C#-based tools as an extension from CPython. Python .NET is a great example of what I'm referring to. The C# language and the CLR does solve some problems, but it does not solve all the problems related to scientific computing that it could. In particular, I wish it's cross-platform visibility where higher. Mono is a great start, but there are a lot of C# libraries that just don't get made to work on Linux or Mac OS X. The task of moving scipy to sit on top of the CLR seems rather large without some automatic tool to allow calling CPython extensions from the CLR that works in a cross-platform way. I don't really see the benefit that the CLR offers (unless all the hype is just so you can write code that runs in a browser --- in which case, are you really going to run matrix inversion on the CLR in a browser?) How does legacy code interact with the magic of the CLR? What are people's opinions about the value of NumPy and SciPy on the CLR? -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Request for advice: project to get NumPy working in IronPython
What are people's opinions about the value of NumPy and SciPy on the CLR? As someone who uses Numpy/Scipy almost exclusively on Linux workstations or on clusters (in coordination with lots of C code), I wouldn't value NumPy and SciPy on the CLR at all. I am kind of curious, though, to see how many people _would_ think it would be usefull S -- Scott M. RansomAddress: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: [EMAIL PROTECTED] Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Request for advice: project to get NumPy working in IronPython
On 10/26/07, Travis E. Oliphant [EMAIL PROTECTED] wrote: An IronPython compatible version of NumPy would be great.Of course it could be done by using C# to write NumPy, but I'm not sure that this would really be any less work than creating a glue layer that allowed most (or all) C-Python extensions to work with IronPython. I'm curious about why all the discussion is about putting Python and its extensions on top of C# and very little discussion about just using C#-based tools as an extension from CPython. Python .NET is a great example of what I'm referring to. The C# language and the CLR does solve some problems, but it does not solve all the problems related to scientific computing that it could. In particular, I wish it's cross-platform visibility where higher. Mono is a great start, but there are a lot of C# libraries that just don't get made to work on Linux or Mac OS X. The task of moving scipy to sit on top of the CLR seems rather large without some automatic tool to allow calling CPython extensions from the CLR that works in a cross-platform way. I don't really see the benefit that the CLR offers (unless all the hype is just so you can write code that runs in a browser --- in which case, are you really going to run matrix inversion on the CLR in a browser?) How does legacy code interact with the magic of the CLR? What are people's opinions about the value of NumPy and SciPy on the CLR? If we were all Microsoft, all the time, it might be worth it. I run Linux myself and haven't installed Windows at home in years, so for me it is a non-starter. Keep it in C and target generic python. Portability and compiler independence is the goal. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak in ndarray, more info
On 10/26/07, Robert Crida [EMAIL PROTECTED] wrote: Hi all I recently posted about a memory leak in numpy and failed to mention the version. The leak manifests itself in numpy-1.0.3.1 but is not present in numpy-1.0.2 The following code reproduces the bug: import numpy as np a = np.array([1.0, 2.0, 3.0]) while True: b = str(a) What happens above is that is repeatedly converted to a string. The process size grow quite rapidly. Has anyone else come across this? Where do I look to try to correct it? I find the same leak: Python version: 2.5.1 numpy version: 1.0.4.dev4081 Regards, Kurt Thanks Robert ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Request for advice: project to get NumPy working in IronPython
What are people's opinions about the value of NumPy and SciPy on the CLR? If anything, wouldn't the big win (if it's a win at all) be to get NumPy/SciPy working on top of the JVM (as T. Hochber tried)? This way it's pretty much universally portable. I know Jython isn't as up to speed as IronPython, but the folks in Jython land are doing a pretty good job of it these days. Their next target CPython compatibility will definitely be good enough for NumPy since it will be at least CPythoon 2.3 ... maybe even 2.5. -steve ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fortran array storage question
On 26/10/2007, Travis E. Oliphant [EMAIL PROTECTED] wrote: There is an optimization where-in the inner-loops are done over the dimension with the smallest stride. What other cache-coherent optimizations do you recommend? That sounds like a very good first step. I'm far from an expert on this sort of thing, but here are a few ideas at random: * internally flattening arrays when this doesn't affect the result (e.g. ones((10,10))+ones((10,10))) * prefetching memory: in a C application I recently wrote, explicitly prefetching data for interpolation cut my runtime by 30%. This includes telling the processor when you're done with data so it can be purged from the cache. * aligning (some) arrays to 8- 16- 32- or 64-byte boundaries so that they divide nicely into cache lines * using MMX/SSE instructions when available * combining ufuncs so that computations can keep the CPU busy while it waits for data to come in from main RAM (I realize that this is properly the domain of numexpr) * using ATLAS- or FFTW-style autotuning to determine the fastest ways to structure computations (again more relevant for significant expressions rather than simple ufuncs) * reducing use of temporaries in the interest of reducing traffic to main memory * openmp parallel operations when this actually speeds up calculation I realize most of these are a lot of work, and some of them are probably in numpy already. Moreover without using an expression parser it's probably not feasible to implement others. But an array language offers the possibility that a runtime can implement all sorts of optimizations without effort on the user's part. Anne ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
