[GRASS-user] Help with defining DB Drivers for input and outputs
Within the same mapset, is it possible to specify SQLITE as the database for the output layer, whilst using a map as an input that has a PG driver? I’ve read the manual and tried a few approaches, but I found the outputs still get written to PG. May be I’ve missed a critical step? See messages below. I also experimented with v.add.table and v.add.columns to the empty map, to no avail. I want the outputs written to sqlite, because v.out.ogr and v.out.postgis when the attributes are stored in PostgreSQL seems unworkably slow (which may be a pg_conf issue but my DB is performing well at scale for non-GRASS parallelised workflows)? I want to keep the input attributes still in PostgreSQL because its the road table, and the fwdcost and bwdcost columns get updated by data from non-GRASS schema. I need the fastest pipeline performance possible for when I scale the solution out (which will be done using GNU Parallel). Right now I’m stuck between a rock and a hard place. Really hoping someone can help me out… :-) Thanks Mark #CREATE EMPTY MAP TO HOLD THE OUTPUTS g.remove -f type=vector pattern=temp** layername=temp_8 v.edit --overwrite tool=create map=$layername v.db.connect map=$layername table=$layername driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db’ v.info $layername #RUN THE ANALYSIS, WHERE THE INPUT LAYER ATTRIBUTES RESIDE IN POSTGRESQL v.net.allpairs --overwrite in=nsw_road_network_final_connected cats=1-100 out=$layername afcolumn=fwdcost abcolumn=bwdcost # NO GOOD - THE OUTPUT ATTRIBUTE TABLE STILL APPEARS IN PGL ## GRASS 7.0.0svn (nodeclean):~ v.edit --overwrite tool=create map=$layername Building topology for vector map temp_8@PERMANENT... Registering primitives... 0 primitives registered 0 vertices registered Building areas... 0 areas built 0 isles built Attaching islands... Attaching centroids... Number of nodes: 0 Number of primitives: 0 Number of points: 0 Number of lines: 0 Number of boundaries: 0 Number of centroids: 0 Number of areas: 0 Number of isles: 0 v.edit complete. GRASS 7.0.0svn (nodeclean):~ v.db.connect map=$layername table=$layername driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db' WARNING: Table temp_8 does not exist in database $GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db The table temp_8 is now part of vector map temp_8 and may be deleted or overwritten by GRASS modules DBMI-SQLite driver error: Unable to create index: create unique index if not exists temp_8_cat on temp_8 ( cat ) no such table: main.temp_8 DBMI-SQLite driver error: Unable to create index: create unique index if not exists temp_8_cat on temp_8 ( cat ) no such table: main.temp_8 WARNING: Cannot create index Select privileges were granted on the table GRASS 7.0.0svn (nodeclean):~ v.db.connect -p map=$layername Vector map temp_8 is connected by: layer 1/temp_8 table temp_8 in database /var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db through driver sqlite with key cat GRASS 7.0.0svn (nodeclean):~ v.info $layername ++ | Name:temp_8| | Mapset: PERMANENT | | Location:nodeclean | | Database:/var/tmp | | Title: | | Map scale: 1:1 | | Name of creator: root | | Organization: | | Source date: Wed Feb 11 22:49:13 2015 | | Timestamp (first layer): none | || | Map format: native| || | Type of map: vector (level: 2) | || | Number of points: 0 Number of centroids: 0 | | Number of lines:0 Number of boundaries: 0 | | Number of areas:0 Number of islands:0 | || | Map is 3D: No | | Number of dblinks: 1| || | Projection: Albers Equal Area
Re: [GRASS-user] Help with defining DB Drivers for input and outputs
On 12/02/15 00:12, Mark Wynter wrote: Within the same mapset, is it possible to specify SQLITE as the database for the output layer, whilst using a map as an input that has a PG driver? I’ve read the manual and tried a few approaches, but I found the outputs still get written to PG. May be I’ve missed a critical step? See messages below. I also experimented with v.add.table and v.add.columns to the empty map, to no avail. You should use db.connect to set your default settings to SQLite, then run your command. By overwriting your map in the v.net.allpairs call you overwrite everything, including any db connections set before and you use the default mapset connection settings which seem to be set to PG. Moritz I want the outputs written to sqlite, because v.out.ogr and v.out.postgis when the attributes are stored in PostgreSQL seems unworkably slow (which may be a pg_conf issue but my DB is performing well at scale for non-GRASS parallelised workflows)? I want to keep the input attributes still in PostgreSQL because its the road table, and the fwdcost and bwdcost columns get updated by data from non-GRASS schema. I need the fastest pipeline performance possible for when I scale the solution out (which will be done using GNU Parallel). Right now I’m stuck between a rock and a hard place. Really hoping someone can help me out… :-) Thanks Mark #CREATE EMPTY MAP TO HOLD THE OUTPUTS g.remove -f type=vector pattern=temp** layername=temp_8 v.edit --overwrite tool=create map=$layername v.db.connect map=$layername table=$layername driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db’ v.info $layername #RUN THE ANALYSIS, WHERE THE INPUT LAYER ATTRIBUTES RESIDE IN POSTGRESQL v.net.allpairs --overwrite in=nsw_road_network_final_connected cats=1-100out=$layername afcolumn=fwdcost abcolumn=bwdcost # NO GOOD - THE OUTPUT ATTRIBUTE TABLE STILL APPEARS IN PGL ## GRASS 7.0.0svn (nodeclean):~ v.edit --overwrite tool=create map=$layername Building topology for vector map temp_8@PERMANENT... Registering primitives... 0 primitives registered 0 vertices registered Building areas... 0 areas built 0 isles built Attaching islands... Attaching centroids... Number of nodes: 0 Number of primitives: 0 Number of points: 0 Number of lines: 0 Number of boundaries: 0 Number of centroids: 0 Number of areas: 0 Number of isles: 0 v.edit complete. GRASS 7.0.0svn (nodeclean):~ v.db.connect map=$layername table=$layername driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db' WARNING: Table temp_8 does not exist in database $GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db The table temp_8 is now part of vector map temp_8 and may be deleted or overwritten by GRASS modules DBMI-SQLite driver error: Unable to create index: create unique index if not exists temp_8_cat on temp_8 ( cat ) no such table: main.temp_8 DBMI-SQLite driver error: Unable to create index: create unique index if not exists temp_8_cat on temp_8 ( cat ) no such table: main.temp_8 WARNING: Cannot create index Select privileges were granted on the table GRASS 7.0.0svn (nodeclean):~ v.db.connect -p map=$layername Vector map temp_8 is connected by: layer 1/temp_8 table temp_8 in database /var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db through driver sqlite with key cat GRASS 7.0.0svn (nodeclean):~ v.info $layername ++ | Name:temp_8 | | Mapset: PERMANENT | | Location:nodeclean | | Database:/var/tmp | | Title: | | Map scale: 1:1 | | Name of creator: root | | Organization: | | Source date: Wed Feb 11 22:49:13 2015 | | Timestamp (first layer): none | || | Map format: native | || | Type of map: vector (level: 2) | | | | Number of points: 0 Number of centroids: 0 | | Number of lines:0 Number of boundaries: 0 | | Number of areas:0 Number of islands:0 | | | | Map is 3D: No | | Number of dblinks: 1 | | | | Projection: Albers Equal Area | | | | N: 0S: 0 | | E: 0W: 0 | | | | Digitization threshold: 0 | | Comment: | | | ++ GRASS 7.0.0svn (nodeclean):~ v.net.allpairs --overwrite in=nsw_road_network_final_connected cats=1-100 out=$layername afcolumn=fwdcost abcolumn=bwdcost
[GRASS-user] SQLite Error
Thanks Moritz. I am getting an SQLite error when trying to run v.net.allpairs. This is my latest workflow, incl. my checks and output messages of each step. #(1) CREATE A COPY OF MY ROAD NETWORK, AND DROP THE ATTRIBUTE TABLE (CURRENTLY SET WITH PG DRIVER) db.connect driver=pg database=host=localhost,dbname=osm_au db.login user=postgres [pass=password] db.connect -p db.tables -p g.copy --overwrite vect=nsw_road_network_final_connected,nsw_road_network_final_connected_copy v.db.droptable nsw_road_network_final_connected_copy v.db.droptable nsw_road_network_final_connected_copy -f # Outputs confirm my copied map no longer is connected to an attribute table Dropping table grass.nsw_road_network_final_connected_copy... Current attribute table link(s): Map nsw_road_network_final_connected_copy is not connected to a database #(2) RESET THE DATABASE DRIVER FOR THE CURRENT MAPSET TO SQLITE db.connect driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db' schema=grass db.connect -p #Output driver: sqlite database: /var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db schema: grass group: #(3) COPY THE ATTRIBUTE TABLE OF THE ORIGINAL ROAD NETWORK IN POSTGRESQL TO THE NEW SQLITE DB db.copy from_driver=pg from_database=host=localhost,dbname=osm_au from_table=grass.nsw_road_network_final_connected to_driver=sqlite to_database=/var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db to_table=nsw_road_network_final_connected_sqlite db.copy from_driver=pg from_database=host=localhost,dbname=osm_au from_table=grass.nsw_tz_centroids_nodes to_driver=sqlite to_database=/var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db to_table=nsw_tz_centroids_nodes_sqlite db.tables -p #Output confirms table successfully copied nsw_road_network_final_connected_sqlite nsw_tz_centroids_nodes_sqlite #(4) CONNECT THE COPIED ATTRIBUTE TABLES TO THE COPIED MAP IN (1) v.db.connect map=nsw_road_network_final_connected_copy table=nsw_road_network_final_connected_sqlite layer=1 v.db.connect map=nsw_road_network_final_connected_copy table=nsw_tz_centroids_nodes_sqlite layer=2 v.category nsw_road_network_final_connected_copy op=report v.category nsw_road_network_final_connected_copy op=report Layer/table: 1/nsw_road_network_final_connected_sqlite type countminmax point 0 0 0 line 467531 1 465943 boundary 0 0 0 centroid 0 0 0 area 0 0 0 face 0 0 0 kernel 0 0 0 all 467531 1 465943 Layer: 2 type countminmax point 3514 1 3514 line 0 0 0 boundary 0 0 0 centroid 0 0 0 area 0 0 0 face 0 0 0 kernel 0 0 0 all 3514 1 3514 v.info -c nsw_road_network_final_connected_copy layer=1 Displaying column types/names for database connection of layer 1: INTEGER|cat INTEGER|lid INTEGER|osm_id CHARACTER|name INTEGER|bridge INTEGER|tunnel INTEGER|oneway CHARACTER|type INTEGER|tid CHARACTER|poa_code INTEGER|reversed INTEGER|open DOUBLE PRECISION|length DOUBLE PRECISION|fwdcost DOUBLE PRECISION|bwdcost v.info -c nsw_road_network_final_connected_copy layer=2 Displaying column types/names for database connection of layer 2: INTEGER|cat INTEGER|tz_code11 CHARACTER|tz_name11 INTEGER|sa2_main11 INTEGER|sa3_code11 #(5) Run v.net.allpairs…. g.remove -f type=vector pattern=temp** v.net.allpairs --overwrite in=nsw_road_network_final_connected_copy cats=1-10 out=temp_8 afcolumn=fwdcost abcolumn=bwdcost DBMI-SQLite driver error: Error in sqlite3_prepare(): unknown database grass DBMI-SQLite driver error: Error in sqlite3_prepare(): unknown database grass ERROR: Unable to create table: 'create table grass.temp_8_1 ( cat integer, from_cat integer, to_cat integer, cost double precision)' Segmentation fault On 12/02/15 00:12, Mark Wynter wrote: Within the same mapset, is it possible to specify SQLITE as the database for the output layer, whilst using a map as an input that has a PG driver? I?ve read the manual and tried a few approaches, but I found the outputs still get written to PG. May be I?ve missed a critical step? See messages below. I also experimented with v.add.table and v.add.columns to the empty map, to no avail. You should use db.connect to set your default settings to SQLite, then run your command. By overwriting your map in the v.net.allpairs call you overwrite everything, including any db connections set before and you use the default mapset connection settings which seem to be set to PG. Moritz ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] v.out.postgis and v.out.ogr - inconsistencies in output
As context, I’m running analysis using v.net.allpairs, whereby vector geometries are often common to more than 1 origin-destination journey path, hence the line segments appear multiple times but with unique cat values, and from_cat and to_cat combinations. When exporting the output layer of v.net.allpairs back into PostGIS, only the third method returns the correct number of geometry features - and this requires first exporting to shapefile and then importing using ogr2ogr. Methods (1) and (2) discard line strings that appear more than once (despite them having unique cat values). This is not what I was expecting. The third method correctly treats each linestring feature at unique, hence assigns a unique ogc_fid to each feature. Are there flags for methods (1) and (2) that deal with this issue - or is method (3) the only feasible option? (1) v.out.postgis --o input=temp_8 output=PG:host=localhost dbname=osm_au user=postgres password=password olayer=jtw.temp_8_pg options=GEOMETRY_NAME=wkb_geometry,SRID=3577 134 Features Written (2) v.out.ogr --o input=temp_8 type=line output=PG:host=localhost dbname=osm_au user=postgres password=password output_layer=jtw.temp_8_ogr format=PostgreSQL 134 Features Written (3) v.out.ogr --o input=temp_8 output=/var/tmp/temp_8 type=line ogr2ogr -overwrite -f PostgreSQL PG:host=localhost dbname=osm_au user=postgres password=password /var/tmp/temp_8/temp_8.shp -nln jtw.temp_8_ogr -s_srs EPSG:3577 -t_srs EPSG:3577 -a_srs EPSG:3577 -nlt LINESTRING 1323 Features Written Kind regards Mark ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] New error at startup
I was making progress with the r.stream.basins issue until I went to edit my vector pour-points layer to move them over the highest flow accumulation. At this point I got an error (that I didn't capture) that crashed the GUI, something about being unable to find the extents of either the layer or the workspace and to run g.region. It wouldn't display properly at all until it was restarted and layers were added manually. Running g.region didn't help at all. Now at startup I get this error: Launching wxpython GUI in the background, please wait... GRASS 7.0.0RC1 (HenrysGRASSLocation):~ 2015-02-11 17:02:51.867 Python[7640:f0f] CFURLCreateWithString was passed this invalid URL string: '/System/Library/CoreServices/CommonCocoaPanels.bundle' (a file system path instead of an URL string). The URL created will not work with most file URL functions. CFURLCreateWithFileSystemPath or CFURLCreateWithFileSystemPathRelativeToBase should be used instead. Things display fine now but editing points still doesn't work, though it hasn't crashed the GUI again. It was throwing errors in the command console when I tried to edit which I didn't capture, something about not finding needed parts. I reinstalled GRASS thinking that might help, which did help with the command console errors but didn't help with the editing problems or the error at startup. I did run a Python script some time before this, thinking it would solve my basins problem, which worked and worked for about 20 minutes according to activity monitor but didn't ever produce anything either in the form of written files, output or errors. The script is here: http://gis.stackexchange.com/questions/88945/how-to-do-the-loop-when-using-grass-module-r-water-outlet Could this script have changed the Python file paths somehow? If so, how do I change them back? I'm on a Mac running OSX 10.7.5 with GRASS version 7.0 Any help would be greatly appreciated. Thanks Jens Hegg ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] new addon: v.ellipse
Dear GRASS users, I would like to announce a new addon module v.ellipse [1] to compute the best-fitting ellipse for given vector data. The module has been written by Tereza Fiedlerova (cc'ed) at the CTU in Prague. Enjoy! Regards, Martin [1] http://grass.osgeo.org/grass70/manuals/addons/v.ellipse.html -- Martin Landa http://geo.fsv.cvut.cz/gwiki/Landa http://gismentors.eu/mentors/landa ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] GRASS pygrass - composite
Dear Pierric, On Wed, Feb 11, 2015 at 10:02 AM, Pierric de Laborie pierric.delabo...@gmail.com wrote: I don't understand why Grass seems to get stuck on the following Python command : r.composite(red=B4,green=B3,blue=B2,output=mycomposite,quiet=QUIET,overwrite=OVR) Have you tried to run the same command in BASH in a normal GRASS shell? Does it work? if yes, can you run a python interpreter inside the GRASS shell and see if calling the module from pygrass it works or not. Just some ideas to understand where could be the problem. All the best Pietro ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] GRASS pygrass - composite
Hello, I am using Grass 7.0 through Python and Pygrass (from outside Grass without starting it) on Linux (Debian GNU/Linux 7.6 (wheezy)) . I don't understand why Grass seems to get stuck on the following Python command : *r.composite(red=B4,green=B3,blue=B2,output=mycomposite,quiet=QUIET,overwrite=OVR)* Here is the information message coming from Grass : *Creating color table for output raster map...* *0..6..12..18..25..31..37..43..50..56..62..68..75..81..87..93..100* *Writing raster map mycomposite...* *0..* I checked the process and memory usage and apparently Grass uses about 100% of the CPU time and only uses half of the memory available. Would there be a way to have more debug / warning information to understand what Grass is doing? Thanks ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] GRASS pygrass - composite
I have just tried to start the same command from the GRASS shell (same machine and same GRASS database and location) and it gets stuck also. GRASS 7.0.0svn (222):~/Grass_test r.composite red=B5@PERMANENT green=B4@PERMANENT blue=B3@PERMANENT output=mycompo Creating color table for output raster map... 0..6..12..18..25..31..37..43..50..56..62..68..75..81..87..93..100 Writing raster map mycompo... 0.. Running the same command in GRASS GUI (command shell) on Windows works fine and creates a composite in a few seconds. I am starting to think that it is due to the way I installed Grass on this Linux machine (I used the following link http://grasswiki.osgeo.org/wiki/Compile_and_Install#GRASS_7_on_Debian_Wheezy ) Thanks 2015-02-11 10:42 GMT+01:00 Pietro peter.z...@gmail.com: Dear Pierric, On Wed, Feb 11, 2015 at 10:02 AM, Pierric de Laborie pierric.delabo...@gmail.com wrote: I don't understand why Grass seems to get stuck on the following Python command : r.composite(red=B4,green=B3,blue=B2,output=mycomposite,quiet=QUIET,overwrite=OVR) Have you tried to run the same command in BASH in a normal GRASS shell? Does it work? if yes, can you run a python interpreter inside the GRASS shell and see if calling the module from pygrass it works or not. Just some ideas to understand where could be the problem. All the best Pietro ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] percentage of class in raster within polygons
Hi, I have to calculate the percentage of given class of a raster within the polygons given in a vector (a regular grid). Since I'm dealing with large files and operations repeated in a chain, i should be focusing on what is the optimal (fastest) option to do that. I could vectorize the raster and use v.overlay and v.to.db, or using v.rast.stats i can count out the number of cells (should i use number for that or sum?) and then divide per area of the grid. Any better idea? Thank you in advance -- Best regards, Dr. Margherita DI LEO Scientific / technical project officer European Commission - DG JRC Institute for Environment and Sustainability (IES) Via Fermi, 2749 I-21027 Ispra (VA) - Italy - TP 261 Tel. +39 0332 78 3600 margherita.di-...@jrc.ec.europa.eu Disclaimer: The views expressed are purely those of the writer and may not in any circumstance be regarded as stating an official position of the European Commission. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
