Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Hi Pietro I would just like to check if you had a change to look at the problem described in http://trac.osgeo.org/grass/ticket/2629? Cheers Paulo On 16-03-15 22:01, Paulo van Breugel wrote: Great, I'll do that On Mon, Mar 16, 2015 at 9:56 PM, Pietro > wrote

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Great, I'll do that On Mon, Mar 16, 2015 at 9:56 PM, Pietro wrote: > Hi Paulo, > > On Mon, Mar 16, 2015 at 9:23 PM, Paulo van Breugel > wrote: > >> Paulo van Breugel wrote: > >> Alternatively, it might indicate an issue in the PyGrass library; that > >> code is relatively new. > > > > > > OK,

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Hi Paulo, On Mon, Mar 16, 2015 at 9:23 PM, Paulo van Breugel wrote: >> Paulo van Breugel wrote: >> Alternatively, it might indicate an issue in the PyGrass library; that >> code is relatively new. > > > OK, hopefully the PyGrass gurus can have a look at it. Ofcourse I will look into it, but sinc

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Mon, Mar 16, 2015 at 3:30 PM, Glynn Clements wrote: > > Paulo van Breugel wrote: > > > When running the tiled_function function you wrote [1] I am getting the > > error below. > > > > {{{ > > tiled_function(raster_inputs=ref, raster_output="out_mah1", > > func=mahalanobis_distances) > > }}} >

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Paulo van Breugel wrote: > When running the tiled_function function you wrote [1] I am getting the > error below. > > {{{ > tiled_function(raster_inputs=ref, raster_output="out_mah1", > func=mahalanobis_distances) > }}} > > ERROR: No null file for > > Any idea what is wrong? That error messa

[GRASS-dev] Compute mahalanobis distance using Scipy

Hi Pietro, When running the tiled_function function you wrote [1] I am getting the error below. {{{ tiled_function(raster_inputs=ref, raster_output="out_mah1", func=mahalanobis_distances) }}} ERROR: No null file for Any idea what is wrong? Best wishes, Paulo [1] http://lists.osgeo.org/pipe

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Paulo van Breugel wrote: > > It may be possible to avoid this issue using numpy.tensordot, e.g. > > > > mahdist = np.sum(np.tensordot(VI, delta, (1,0)) * delta,axis=0) > > > > I don't know for sure whether this actually uses less memory. It could > > do, as it rearranges the matrices so that the

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Hi, On Tue, Feb 17, 2015 at 10:44 AM, Paulo van Breugel wrote: > Can raster_input be multiple rasters? yes, I've define a new split_rasters function, that should does what you want, note, the code is not tested, just to provide a rough idea. {{{ import numpy as np from grass.pygrass.gis.region

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Tue, Feb 17, 2015 at 7:22 AM, Pietro wrote: > On Mon, Feb 16, 2015 at 10:50 PM, Paulo van Breugel > wrote: > > On Sat, Feb 14, 2015 at 7:29 PM, Vaclav Petras > wrote: > >> On Sat, Feb 14, 2015 at 11:47 AM, Paulo van Breugel > >> wrote: > >>> For a quick solution, what about using r.tile to

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Brilliant, thanks! On Tue, Feb 17, 2015 at 7:22 AM, Pietro wrote: > On Mon, Feb 16, 2015 at 10:50 PM, Paulo van Breugel > wrote: > > On Sat, Feb 14, 2015 at 7:29 PM, Vaclav Petras > wrote: > >> On Sat, Feb 14, 2015 at 11:47 AM, Paulo van Breugel > >> wrote: > >>> For a quick solution, what ab

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Mon, Feb 16, 2015 at 10:50 PM, Paulo van Breugel wrote: > On Sat, Feb 14, 2015 at 7:29 PM, Vaclav Petras wrote: >> On Sat, Feb 14, 2015 at 11:47 AM, Paulo van Breugel >> wrote: >>> For a quick solution, what about using r.tile to split the input data in >>> tiles and compute the mahalanobis d

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Sat, Feb 14, 2015 at 7:29 PM, Vaclav Petras wrote: > > On Sat, Feb 14, 2015 at 11:47 AM, Paulo van Breugel < > p.vanbreu...@gmail.com> wrote: > >> >> For a quick solution, what about using r.tile to split the input data in >> tiles and compute the mahalanobis distance per tile. >> > > See PyGR

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

That would be great, I would like to learn and help doing that! El 16/02/2015 12:07, "Paulo van Breugel" escribió: > Hi Javier, > > Thanks. I need to test whether the mahalanobis function you provided is > faster than the one I used (which Glynn wrote, based on the numpy function > I believe); I

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Hi Javier, Thanks. I need to test whether the mahalanobis function you provided is faster than the one I used (which Glynn wrote, based on the numpy function I believe); I think I can use both together with the function to run it in parallel, so it would be a matter of selecting the fastest one (o

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Hi Paulo, to use it with python just copy the mahalanobis function definitions, import all necessary libraries and use it as in line 597 (the covariance matrix and the mean are computed just before this line). I am sorry that the code is not documented yet. I am not sure if this will solve the lar

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Hi Javier, This looks really useful, thanks! My python skills are very limited, so I will have to look at it better to see what I can understand. But to start, the parallel function, does that also solve the handling of potentially very large raster layers? Btw, this does look like a very useful

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Hi Paulo, you can see an implementation of the mahalanobis distance computed in parallel (using all computer processors) for a single image here: https://github.com/javimarlop/eHabpy/blob/master/ehab.py see lines 32-86 and 597. I hope it helps! Cheers, Javier On Sat, Feb 14, 2015 at 7:29 PM, Va

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Sat, Feb 14, 2015 at 11:47 AM, Paulo van Breugel wrote: > > For a quick solution, what about using r.tile to split the input data in > tiles and compute the mahalanobis distance per tile. > See PyGRASS GridModule class which will do tiling (and a lot of other things) for you. http://grass.os

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Sat, Feb 14, 2015 at 4:38 AM, Glynn Clements wrote: > > Paulo van Breugel wrote: > > > >> With memmap you still have a limits of 2Gb I guess, you should try: > dask > > >> > > > > > Just reading the memmap manual page, where it reads: "Memory-mapped > arrays > > use the Python memory-map objec

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Paulo van Breugel wrote: > >> With memmap you still have a limits of 2Gb I guess, you should try: dask > >> > > > Just reading the memmap manual page, where it reads: "Memory-mapped arrays > use the Python memory-map object which (prior to Python 2.5) does not allow > files to be larger than a ce

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Fri, Feb 13, 2015 at 10:13 AM, Paulo van Breugel wrote: > Hi Pietro, > > Thanks for the suggestion, I will have a look at the documentation. > > Paulo > > On Fri, Feb 13, 2015 at 10:09 AM, Pietro wrote: > >> Dear Paulo, >> >> On Fri, Feb 13, 2015 at 9:57 AM, Paulo van Breugel >> wrote: >> >

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Hi Pietro, Thanks for the suggestion, I will have a look at the documentation. Paulo On Fri, Feb 13, 2015 at 10:09 AM, Pietro wrote: > Dear Paulo, > > On Fri, Feb 13, 2015 at 9:57 AM, Paulo van Breugel > wrote: > > I guess this is because the calculations are done in-memory? Any way to > > av

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Dear Paulo, On Fri, Feb 13, 2015 at 9:57 AM, Paulo van Breugel wrote: > I guess this is because the calculations are done in-memory? Any way to > avoid this memory problem when using large data sets (something like working > with memmap objects?) With memmap you still have a limits of 2Gb I gues

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Hi Glyn, Your suggestions to compute the mahalanobis distance work perfect. However, I have a follow up question. For large data sets, I am getting a MemoryError when running "mahdist = np.sum(np.sum(delta[None,:,:,:] * VI[:,:,None,None],axis=1) * delta,axis=0)" # pro is list with raster names

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Paulo van Breugel wrote: > > The second version: > > > > > > VI = np.linalg.inv(covar) > > > > delta = dat_ref - stat_mean[:,None,None] > > > > m = np.sum(np.sum(delta[:,:,:,None] * delta[:,:,None,:] * > > > > VI[None,None,:,:],axis=-1),axis=-1) > > > > stat_mah = np.sqrt(m) > >

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Mon, Feb 2, 2015 at 3:40 PM, Paulo van Breugel wrote: > > > On Mon, Feb 2, 2015 at 1:38 PM, Glynn Clements > wrote: > >> >> Paulo van Breugel wrote: >> >> > > The main speed-up will come from "inlining" distance.mahalanobis(), >> > > which is essentially: >> > > >> > > delta = u - v >> >

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Mon, Feb 2, 2015 at 1:38 PM, Glynn Clements wrote: > > Paulo van Breugel wrote: > > > > The main speed-up will come from "inlining" distance.mahalanobis(), > > > which is essentially: > > > > > > delta = u - v > > > m = np.dot(np.dot(delta, VI), delta) > > > return np.sqrt(m) > > >

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Paulo van Breugel wrote: > > The main speed-up will come from "inlining" distance.mahalanobis(), > > which is essentially: > > > > delta = u - v > > m = np.dot(np.dot(delta, VI), delta) > > return np.sqrt(m) > > > > np.dot(v, m) is equivalent to np.sum(v * m,axis=0), and np.dot(u,v) i

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Fri, Jan 30, 2015 at 3:39 PM, Glynn Clements wrote: > > Paulo van Breugel wrote: > > > I would like to compute a raster layer with for each raster cell the > > mahalanobis distance to the centre of the environmental space formed by > all > > reference data points (raster cells). In R this can

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On Fri, 30 Jan 2015 10:59 Moritz Lennert wrote: On 29/01/15 18:30, Paulo van Breugel wrote: > I would like to compute a raster layer with for each raster cell the > mahalanobis distance to the centre of the environmental space// formed > by all reference data points (raster cells). In R this can

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Thanks! I am traveling right now so will look at this as soon as I am back, but this looks promising. Much appreciated. On Fri, 30 Jan 2015 15:40 Glynn Clements wrote: > > Paulo van Breugel wrote: > > > I would like to compute a raster layer with for each raster cell the > > mahalanobis distance

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

Paulo van Breugel wrote: > I would like to compute a raster layer with for each raster cell the > mahalanobis distance to the centre of the environmental space formed by all > reference data points (raster cells). In R this can be done as explained > here [1]. > > . I would like to do this using

Re: [GRASS-dev] Compute mahalanobis distance using Scipy

On 29/01/15 18:30, Paulo van Breugel wrote: I would like to compute a raster layer with for each raster cell the mahalanobis distance to the centre of the environmental space// formed by all reference data points (raster cells). In R this can be done as explained here [1]. . I would like to do t

[GRASS-dev] Compute mahalanobis distance using Scipy

I would like to compute a raster layer with for each raster cell the mahalanobis distance to the centre of the environmental space formed by all reference data points (raster cells). In R this can be done as explained here [1]. . I would like to do this using python only (no dependency on R). I ca