[Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Ke Sun
Dear all, I have written the following function to compute the square distances of a large matrix (each sample a row). It compute row by row and print the overall progress. The progress output is important and I didn't use matrix multiplication. I give as input a 70,000x800 matrix. The output

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Matthew Brett
Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a large matrix (each sample a row). It compute row by row and print the overall progress. The progress output is important and I didn't

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Julian Taylor
your computation is symmetric so you only need to compute the upper or lower triangle which will save both memory and time. On Tue, Oct 8, 2013 at 10:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a large matrix

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Henry Gomersall
On 08/10/13 09:06, Ke Sun wrote: I give as input a 70,000x800 matrix. The output should be a 70,000x70,000 matrix. The program runs really slow (16 hours for 1/3 progress). And it eats 36G memory (fortunately I have enough). At this stage I'd be asking myself what I'm trying to achieve and why

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Henry Gomersall
On 08/10/13 09:49, Matthew Brett wrote: On Tue, Oct 8, 2013 at 1:06 AM, Ke Sunsunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a large matrix (each sample a row). It compute row by row and print the overall progress. The

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Julian Taylor
Out of interest, how did you do this with matrix multiplication? http://stackoverflow.com/a/4856692 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Ke Sun
On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a large matrix (each sample a row). It compute row by row and print

[Numpy-discussion] Question about typenum

2013-10-08 Thread Valentin Haenel
Hi, I have a quick question about typenum. Certain functions, like `PyArray_SimpleNewFromData` `PyArray_SimpleNew` take a typeenum argument. Is there any way to go from typeenum to something that can be passed to the dtype constructor, like mapping 12 - 'f8'? thanks, V-

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Julian Taylor
On Tue, Oct 8, 2013 at 1:38 PM, Ke Sun sunk...@gmail.com wrote: On a machine I had access to it took about 20 minutes. How? I am using matrix multiplication (the same code as http://stackoverflow.com/a/4856692) and it runs for around 18 hours. make sure you are using an optimized BLAS

[Numpy-discussion] Python function for line intersection??

2013-10-08 Thread Happyman
Hello, I am having trouble with solving line intersection problem. I would solve it using standard functions which may already exist in Python library. Let me explain it below: I have sets of lines A(x,y) and B(x,y) and one map as a matrix form, also with its resolution dx, dy which can be

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Jaime Fernández del Río
On Tue, Oct 8, 2013 at 4:38 AM, Ke Sun sunk...@gmail.com wrote: On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a

Re: [Numpy-discussion] Question about typenum

2013-10-08 Thread Richard Hattersley
Hi Valentin, On 8 October 2013 13:23, Valentin Haenel valen...@haenel.co wrote: Certain functions, like `PyArray_SimpleNewFromData` `PyArray_SimpleNew` take a typeenum Is there any way to go from typeenum to something that can be passed to the dtype constructor, like mapping 12 - 'f8'? If

Re: [Numpy-discussion] Question about typenum

2013-10-08 Thread Valentin Haenel
Hi Richard, * Richard Hattersley rhatters...@gmail.com [2013-10-08]: On 8 October 2013 13:23, Valentin Haenel valen...@haenel.co wrote: Certain functions, like `PyArray_SimpleNewFromData` `PyArray_SimpleNew` take a typeenum Is there any way to go from typeenum to something that can be

Re: [Numpy-discussion] Bug in numpy.correlate documentation

2013-10-08 Thread Richard Hattersley
Hi Bernard, Looks like you're on to something - two other people have raised this discrepancy before: https://github.com/numpy/numpy/issues/2588. Unfortunately, when it comes to resolving the discrepancy one of the previous comments takes the opposite view. Namely, that the docstring is correct

Re: [Numpy-discussion] Question about typenum

2013-10-08 Thread Richard Hattersley
On 8 October 2013 19:56, Valentin Haenel valen...@haenel.co wrote: I ended up using: PyArray_TypeObjectFromType from cython so: np.dtype(cnp.PyArray_TypeObjectFromType(self.ndtype)).str Maybe i can avoid the np.dtype call, when using PyArray_Descr? In short: yes.

Re: [Numpy-discussion] Question about typenum

2013-10-08 Thread Valentin Haenel
* Richard Hattersley rhatters...@gmail.com [2013-10-08]: On 8 October 2013 19:56, Valentin Haenel valen...@haenel.co wrote: I ended up using: PyArray_TypeObjectFromType from cython so: np.dtype(cnp.PyArray_TypeObjectFromType(self.ndtype)).str Maybe i can avoid the np.dtype call,

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Matthew Brett
Hi, On Tue, Oct 8, 2013 at 4:38 AM, Ke Sun sunk...@gmail.com wrote: On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Matthew Brett
Hi, On Tue, Oct 8, 2013 at 12:44 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Tue, Oct 8, 2013 at 4:38 AM, Ke Sun sunk...@gmail.com wrote: On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun sunk...@gmail.com wrote: Dear