Hi again, I am glad to hear the approach seemed useful.
Regarding resolution: my point was, that you need to use a high enough resolution for e.g. a 3m buffer. Keep in mind that r.neighbors only allows odd numbers for diameter of a moving window. And with a small moving window, the neighbourhood will not really look like a circle… So just be aware that it is an approximation in many regards… BTW. I just started working on porting the v.what.rast.buffer addon to Python/pygrass/GRASS 7 (which in fact will become almost a rewrite as I add some new functionality). It will most likely not have the performance for 700k points (at least not if you expect fast results), but I can see if I manage to include the r.neigbors approach as an alternative way of buffer statistics (for point input). Cheers, Stefan From: Markus Metz <[email protected]> Sent: lørdag 24. mars 2018 19.03 To: Stefan Blumentrath <[email protected]> Cc: Mehrdad Varedi <[email protected]>; [email protected] Subject: Re: [GRASS-user] Need Help with speeding up v.import and reducing/removing null values from v.rast.stat results On Sat, Mar 24, 2018 at 10:36 AM, Stefan Blumentrath <[email protected]<mailto:[email protected]>> wrote: > > Hei again Mehrdad, > > A different appoach to tackle your problem could be to: > 1) run r.neighbors with a circular window that has a size according to your > buffer distance while computing the statistic of interest > 2) use v.what.rast or r.what to get neighbourhood values at your point > locations... Wow, that's a really elegant solution! Could you please add this to the manual of v.what.rast? Thanks! > > That way you can circumvent overlap issues and for the points you do not even > have to build topology (or can use a linked OGR map (v.external), if I am not > mistaken)... > > In any case you would have to carefully consider the resolution for your > analysis based on the required precision... Any reason why the resolution of the raster to be queried should not be used? Markus M > > Cheers > > Stefan > > > > > > From: Markus Metz > <[email protected]<mailto:[email protected]>> > Sent: fredag 23. mars 2018 17.57 > To: Mehrdad Varedi <[email protected]<mailto:[email protected]>> > Cc: Stefan Blumentrath > <[email protected]<mailto:[email protected]>>; > [email protected]<mailto:[email protected]> > > > Subject: Re: [GRASS-user] Need Help with speeding up v.import and > reducing/removing null values from v.rast.stat results > > > > > > On Fri, Mar 23, 2018 at 4:30 PM, Mehrdad Varedi > <[email protected]<mailto:[email protected]>> wrote: > > > > Thank you very much Stefan and Moritz for your attention and replies. > > I found that you both exactly understand where is the problem. Let me be > > more specific about my primary issue. > > > > - v.buffer and v.import (v.in.ogr) - They freez. > > I found that my problem was reported in a post: #2185 new defect > > Painfully Slow 'v.in.ogr' Vector Import > > Here is the link: https://trac.osgeo.org/grass/ticket/2185#no1 > > > > Now two questions: > > > > 1 - Two patches has been offered in this post that apparently has solved > > the problem. I really don't know how to use patches and when they should be > > executed or compiled. Would you please refer me to some resources to > > educate myself to how to use the patches, where to get the compiler, etc. > > > > 2 - Are these patches applied already in GRASS 7.4.0? > > The issue in #2185 has been fixed already, but differently. The suggested > patches were fixing symptoms but not the cause. > > > > > - Some background about the data I have: > > > > As I said there are 700,000 points , the distance between the points are 3 > > meters. I create buffers with the size of 3, 7, 20, and 50 meters around > > each point. Then will use v.rast.stats to calculate statistics on rasters > > with the resolution of 1 square meter. > > > > I could import the buffer layers created in QGIS using fixed distance > > buffer function and import them for the 3 and 7 meter buffer layers. then > > v.rast.stats could complete the calculations after 2 hours for the 3 meter > > buffer and after 4.5 hours for the 7 meter layer. > > > > v.import for the 20 meter buffer was frozen on "Breaking Boundaries ... " > > step. I have provided the log of messages that I get. all of the steps is > > done in few minutes and it stuck on the "Breaking Boundaries ..." for more > > than 12 hours for the 20 meter buffer. ( I am sure it would be the same for > > the 50 meter buffer) > > If the buffer radius is a bit larger than half the distance between points, > you will get some overlapping buffers, but that should not be too bad. If the > buffer radius is much larger than the distance between points, e.g. distance > between points = 3 and buffer distance = 50, the result will be a topological > challenge with multiple overlapping buffers. > > Alternatively you can approximate a common buffer for all points by 1) > creating a concave hull with v.concave.hull [0], 2) buffering that hull. > > HTH, > > Markus M > > > [0] https://grass.osgeo.org/grass74/manuals/addons/v.concave.hull.html > > > > > > > (Thu Mar 22 22:41:46 2018) > > v.import > > input=E:\Projects-Analysis\London\Data\Output\P3ntsOnMains\Buffers\LondonBuffers > > layer=Buffer20_P3intsOnMains output=Buffer20_P3intsOnMains --overwrite > > Check if OGR layer <Buffer20_P3intsOnMains> contains polygons... > > WARNING: Vector map <Buffer20_P3intsOnMains> already exists and will be > > overwritten > > WARNING: Table <Buffer20_P3intsOnMains> linked to vector map > > <Buffer20_P3intsOnMains> does not exist > > Creating attribute table for layer <Buffer20_P3intsOnMains>... > > Column name <cat> renamed to <cat_> > > Importing 706628 features (OGR layer <Buffer20_P3intsOnMains>)... > > ----------------------------------------------------- > > Registering primitives... > > 706628 primitives registered > > 14839188 vertices registered > > Number of nodes: 681288 > > Number of primitives: 706628 > > Number of points: 0 > > Number of lines: 0 > > Number of boundaries: 706628 > > Number of centroids: 0 > > Number of areas: - > > Number of isles: - > > ----------------------------------------------------- > > Cleaning polygons > > ----------------------------------------------------- > > Breaking polygons... > > Breaking polygons (pass 1: select break points)... > > Breaking polygons (pass 2: break at selected points)... > > ----------------------------------------------------- > > Removing duplicates... > > ----------------------------------------------------- > > Breaking boundaries... > > > > > > > > In another try when the distance between my points were 12 meters, > > everything was working relatively fast and with no problem. increasing the > > density of points (3 meter distance) has resulted in a very long delay, so > > much that I think it is in an infinite loop. > > > > I appreciate for your help and looking forward hearing from you. > > > > Kind regards, > > > > Mehrdad > > > > > > > > Mehrdad Varedi, M.A.Sc<http://M.A.Sc>. > > Cell: +1 (519)722-7057 > > > > On Fri, Mar 23, 2018 at 9:15 AM, Stefan Blumentrath > > <[email protected]<mailto:[email protected]>> wrote: > >> > >> Hei Mehrdad, > >> > >> If your buffers are overlapping, topology will become an issue, as Moritz > >> pointed out. > >> Johannes Radinger once wrote a workaround for a specific use case that > >> might be of help. See [1]. No idea if it would be possible to come up with > >> something more generic for v.rast.stats… > >> > >> In v.rast.stas NAs occur most likely due to vector areas that are not > >> rasterized, which usually happens when polygons are relatively small / > >> narrow compared to resolution of the raster map. > >> > >> However, during community sprint these days in Bonn, Martin Landa added > >> centroids to the rasterization algorithm (v.to.rast) that v.rast.stats > >> uses, than can help to reduce the number of NAs. > >> > >> Also during community sprint, I created a patch for v.rast.stats [2] that > >> aims at improving the performance of the module by allowing multiple > >> raster input (it needs testing). > >> > >> There are possible further improvements, but these require some more > >> discussion with the other devs regarding implementation. > >> > >> Cheers > >> Stefan > >> > >> 1: https://lists.osgeo.org/pipermail/grass-user/2017-June/076670.html > >> 2: https://trac.osgeo.org/grass/ticket/3523 > >> > >> -----Original Message----- > >> From: grass-user > >> <[email protected]<mailto:[email protected]>> > >> On Behalf Of Moritz Lennert > >> Sent: fredag 23. mars 2018 10.52 > >> To: Mehrdad Varedi <[email protected]<mailto:[email protected]>>; > >> [email protected]<mailto:[email protected]> > >> Subject: Re: [GRASS-user] Need Help with speeding up v.import and > >> reducing/removing null values from v.rast.stat results > >> > >> On 22/03/18 22:40, Mehrdad Varedi wrote: > >> > Hi Everyone, > >> > > >> > I am trying to import few layers with nearly 700,000 buffer areas > >> > around the same number of points. Fr each layer t takes more than 4-5 > >> > hours and most of the time, it just stops in the middle of the process. > >> > Any idea that which kind of layers can be imported with a faster speed? > >> > I need o load 4-5 of such layers and it is now 5 days that I am busy > >> > with building a buffer or importing the results. > >> > > >> > Actually I tried v.buffer and it took more than a day until I stopped > >> > the procedure, although I could create the layer with Fixed distance > >> > buffer function in QGIS in 10 minutes. > >> > Any idea to accelerate the process of v.buffer ot v.import in GRASS? > >> > >> I would guess that the difference comes from the difference in output > >> format: QGIS will just create non-topological, overlapping buffers (unless > >> you ask it to create unique buffers) while GRASS has to take care of all > >> the topological issues in order to create a topologically clean vector map. > >> > >> The same is true both for buffer creation and for import. > >> > >> > > >> > Also I want to calculate statistics in each of the buffers and have > >> > the same issue with v.rast.stats function > >> > >> You might get faster results rasterizing your buffers and using r.univar > >> with the zones= parameter. You can then import the resulting text file > >> into your database and link it to the vector map, as long as you make sure > >> that the pixel values in the zone map correspond to the cat values of the > >> vector map. > >> > >> > > >> > Another issue with this function is a lot of NAs in the result. No > >> > hole exists in the raster files and it should be something with > >> > snapping I guess which results in removal of many points. > >> > >> You would have to be a bit more precise about what exactly you are doing > >> and what exactly the results are. Are you talking about points or polygons > >> ? > >> > >> Moritz > >> > >> > >> _______________________________________________ > >> grass-user mailing list > >> [email protected]<mailto:[email protected]> > >> https://lists.osgeo.org/mailman/listinfo/grass-user > > > > > > > > _______________________________________________ > > grass-user mailing list > > [email protected]<mailto:[email protected]> > > https://lists.osgeo.org/mailman/listinfo/grass-user
_______________________________________________ grass-user mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/grass-user
