Pierre Roudier wrote: > Dear all, > > I tried the very same thing on the stable version of GRASS (6.4 RC7 - > which AFAIK is the stable 6.4), on the very same machine.
The stable 6.4 version is 6.4.0, the recommended 6.4 version is 6.4.1 > Unfortunately, I still an error. On the "Break boundaries" state > during the imprt, it is stuck for a while at 4gb RAM used. Then for > some reason the memory consumption becomes much bigger (maximum > actually - 6.2GB+2Gb swap) and the process gets killed. > Weird. I assume you want to import the Land Cover DataBase version 2 for the Northern Island of New Zealand in the New Zealand Map Grid projection. I have imported the Land Cover DataBase version 2 for all of New Zealand in the New Zealand Transverse Mercator projection with more than twice as many polygons as in the Northern Island only, and memory peak was at about 4GB. Judging by the cleaning done by v.in.ogr --v, the shapefile I imported was clean. For not so clean shapefiles where adjacent polygons are not 100% adjacent but are slightly overlapping or leave small gaps in between them, memory consumption can go up considerably. I have two suggestions: 1. Use the snapping option of v.in.ogr with snap=1 and have verbose output with --v. Considering that the LCDB2 was created from Landsat ETM7+ imagery, the spatial detail can not be finer than 28.5m and you may be able to increase the snapping threshold to values larger than 1 without snapping away polygons. 2. Use the lcdb2 version I used, available at koordinates.com [1]. Optionally set the region first to the Northern Island and then import with v.in.ogr -r > Any pointers would be much appreciated. Otherwise, maybe I should open > a bug for that? > I don't think this is a bug. Memory consumption can get quite high when processing larger vectors with topology in grass. I know which are the memory-hungry components and want to reduce memory consumption, but that will happen only in grass 7, and most probably not this year. Markus M [1] http://koordinates.com/layer/1072-land-cover-database-version-2-lcdb2/ > > 2010/11/28 Pierre Roudier <[email protected]>: >> Thanks for your answers. Yes, this is a 64bits OS. >> >> 2010/11/27 Markus Metz <[email protected]>: >>> On Fri, Nov 26, 2010 at 3:14 AM, Pierre Roudier >>> <[email protected]> wrote: >>>> Dear list, >>>> >>>> I am trying to load a pretty big shapefile on GRASS (6Mb +, 211621 >>>> features) using v.in.ogr, but each attempt fails at the 'break >>>> boundaries'' stage (at 98%!): >>>> >>>> GRASS 7.0.svn (NZTM2000):~ > v.in.ogr >>>> dsn=~/Documents/DATA/LCDB2/ni_nzmg.shp out=ni_lcdb2 --o >>>> Projection of input dataset and current location appear to match >>>> Layer: ni_nzmg >>>> Counting polygons for 211621 features... >>>> Importing map 211621 features... >>>> 100% >>>> ----------------------------------------------------- >>>> Building topology for vector map <ni_lcdb2_...@ni>... >>>> Registering primitives... >>>> 334611 primitives registered >>>> 18459167 vertices registered >>>> Number of nodes: 211681 >>>> Number of primitives: 334611 >>>> Number of points: 0 >>>> Number of lines: 0 >>>> Number of boundaries: 334611 >>>> Number of centroids: 0 >>>> Number of areas: - >>>> Number of isles: - >>>> ----------------------------------------------------- >>>> WARNING: Cleaning polygons, result is not guaranteed! >>>> ----------------------------------------------------- >>>> Break polygons: >>>> 100% >>>> 100% >>>> ----------------------------------------------------- >>>> Remove duplicates: >>>> 100% >>>> ----------------------------------------------------- >>>> Break boundaries: >>>> ERROR: G_calloc: unable to allocate 50 * 8 bytes of memory at >>>> allocation.c:82 >>>> >>>> The shp loads without any problems on QGIS. >>>> >>> >>>> OS: Opensuse 11.3, with 7Gb RAM. >>> Is this a 64 bit OS? >>> >>> I have previously successfully imported larger shapefiles, more than >>> double in size: >400,000 areas, >40 million vertices, peak memory >>> consumption was close to 4GB. The import of your shapefile would >>> approximately require about 2GB of memory, which should not be a >>> problem with 7GB available, as long as it's all 64 bit and not 32 bit. >>> >>> Markus M >>> >> > _______________________________________________ grass-user mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-user
