Re: [Numpy-discussion] Efficient square distance computation

2013-10-10 Thread Ke Sun
On a machine I had access to it took about 20 minutes. How? I am using matrix multiplication (the same code as http://stackoverflow.com/a/4856692) and it runs for around 18 hours. make sure you are using an optimized BLAS library. e.g. with openblas I could also do this size in about

[Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Ke Sun
Dear all, I have written the following function to compute the square distances of a large matrix (each sample a row). It compute row by row and print the overall progress. The progress output is important and I didn't use matrix multiplication. I give as input a 70,000x800 matrix. The output

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Matthew Brett
Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a large matrix (each sample a row). It compute row by row and print the overall progress. The progress output is important and I didn't

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Julian Taylor
your computation is symmetric so you only need to compute the upper or lower triangle which will save both memory and time. On Tue, Oct 8, 2013 at 10:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a large matrix

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Henry Gomersall
On 08/10/13 09:06, Ke Sun wrote: I give as input a 70,000x800 matrix. The output should be a 70,000x70,000 matrix. The program runs really slow (16 hours for 1/3 progress). And it eats 36G memory (fortunately I have enough). At this stage I'd be asking myself what I'm trying to achieve and why

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Henry Gomersall
On 08/10/13 09:49, Matthew Brett wrote: On Tue, Oct 8, 2013 at 1:06 AM, Ke Sunsunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a large matrix (each sample a row). It compute row by row and print the overall progress. The

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Julian Taylor
Out of interest, how did you do this with matrix multiplication? http://stackoverflow.com/a/4856692 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Ke Sun
On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a large matrix (each sample a row). It compute row by row and print

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Julian Taylor
On Tue, Oct 8, 2013 at 1:38 PM, Ke Sun sunk...@gmail.com wrote: On a machine I had access to it took about 20 minutes. How? I am using matrix multiplication (the same code as http://stackoverflow.com/a/4856692) and it runs for around 18 hours. make sure you are using an optimized BLAS

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Jaime Fernández del Río
On Tue, Oct 8, 2013 at 4:38 AM, Ke Sun sunk...@gmail.com wrote: On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Matthew Brett
Hi, On Tue, Oct 8, 2013 at 4:38 AM, Ke Sun sunk...@gmail.com wrote: On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun sunk...@gmail.com wrote: Dear all, I have written the following function to compute the square distances of a

Re: [Numpy-discussion] Efficient square distance computation

2013-10-08 Thread Matthew Brett
Hi, On Tue, Oct 8, 2013 at 12:44 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Tue, Oct 8, 2013 at 4:38 AM, Ke Sun sunk...@gmail.com wrote: On Tue, Oct 08, 2013 at 01:49:14AM -0700, Matthew Brett wrote: Hi, On Tue, Oct 8, 2013 at 1:06 AM, Ke Sun sunk...@gmail.com wrote: Dear