Re: [GRASS-user] v.isochrones : error message
from a practical standpoint, it looks like in your image the origin point for the isolines is central Paris, not the yellow dot to north. By way of comparison with your image, what do results look like if you use v.net.iso? This will return lines with each ISO cost value, not a surface. It thus is a two step process to create a surface (potentially giving you more options), but it might help to isolate the cause of the problem. So with max speed limits, you mean you updated the forward and backward cost columns, taking into account length too? When i modify the resolution nres/ewres from 1 to 150, i have no longer error message. But when i try a process with a resolution with a value 150, the process is endless... According to you, the results generated at this resolution (150) would be relevant? valid? I loaded the output vector into qgis. The results are seem incoherent. The curves show that we can go in 15 minutes on the other side of Paris! In reality, From my starting point we would go much further toward toward north than south west... I use OSM data. I edited and populated the max speed field thanks to the value of another filed (type of road). How to explain this incoherence? Due to v.isochrones? Due to resolution? Due to input maxspeed values ? I am annoyed because it seems impossible to generate the results for a better resolution than 150. Thanks. I paste below a SS representing results. http://osgeo-org.1560.x6.nabble.com/file/n529/SS_v.jpg ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] v.net tools with polygons
Just recalling the v.net techniques I used on a mining project a year ago where we had to optimize the infrastructure build and operational logistics for over 2000 drill sites, feeding several downstream processing plants. V.net.distance in reverse will do what you need if each trip is being made independently (farmers act as individual agents and cart their own goods to the processing plant). If a transport haulage firm calls on multiple farms as part of the same trip, then traveling sales person might be relevant to your needs. Situations may also arise where you need to use v.net.spanningtree or Steiner. For example, if you need to build roads or pipelines connecting various facilities, or supply aggregation points, then this problem needs to be solved, before you apply v.net.distance, or v.net.iso which assume the network infrastructure is already built. If the network infrastructure is already built, and the agents act independently with no supply chain aggregation, then V.net.iso is an alternative to v.net.distance if all you need is the travel cost contours radiating out from the processing plants and not the route paths themselves. if you have numerous processing plants, you can run v.net.iso for each plant. Then you combine cost surfaces by taking the plant with the least cost value, giving you a minimum cost surface. With v.net.iso, your decision model doesn't always have to assume the farmer will choose the plant with the lowest transport cost. It pays to keep each v.net.iso cost surface, because sometimes plants have capacity limitations - in which case agents may choose to incur a higher transport cost and still be profit maximizing. the v.net.iso approach allows you to consider the economics of lots of different processing plant and network configuration scenarios without having to always re-run v.net.distance every time. Sorry if this sounds overwhelming - but v.net should give you all the tools you need. It's a case of knowing what tool for what job. My advice is to be really clear on your modeling assumptions that underlie each method. Sing out if you want to test your methodology on me. Just email me.. - mark Sent from my iPhone On 4 Mar 2015, at 7:48 am, Mark Wynter m...@dimensionaledge.com wrote: Hi Daniel, I've done something similar - I call it off road routing, and uses a regular lattice of nodes and arcs. You can then constrain the off road network by closing arcs that cross major watercourses, fence lines or where the terrain or vegetation is non navigatable. For farms, I added paddock boundaries as rings, as well as gate nodes that constrain movement between paddocks. In essence you build a network topology that reflects the off-road aspect of your network. Relevant to mining as well as agriculture. Ok Moritz, Thanks for the tips. I'll try to go the centroids way Cheers Daniel On Tue, Mar 3, 2015 at 5:35 AM, Moritz Lennert mlenn...@club.worldonline.be wrote: On 02/03/15 21:39, Daniel Victoria wrote: Hi list, I'm beginning to learn and use the v.net http://v.net tools in Grass in order to evaluate the distance from several crop fields to a processing plant. I've successfully build the road network with the end nodes but now I'm in doubt. My starting points in the analysis are crop fields, which are polygons. So what is the best (or most common) practice? 1) Use the field centroids as starting nodes? 2) Add field polygon boundaries to the network and run v.net.distance backwards (from mill to fields)? 3) Some other option? I don't think that there is a best practice for this. It all depends on your application and the desired outcome. Do you want average time/distance from anywhere in the field to the plant ? Then probably the centroid is ok. Or do you want distance from the point of the field that is closest to the network ? Then you could get the coordinates of that point through v.distance (with upload=to_x,to_y) and use these points as nodes. Also, if I'm to add the field boundaries to the network, how would I go about it? Should I first v.patch the field with the roads layer and then run v.net http://v.net? Adding field boundaries still does not answer the question of where to put the start/stop point of your paths... If you want to add them to the network then yes, patching would be the best option, AFAIK. Moritz * ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] v.net tools with polygons
Hi Daniel The modeling approach I took used a combination of grass and postgis. I ran v.net.iso for every processing plant - or possible processing plant - and every farm centroid. At this point, I did not worry about capacity limitations of the plants - you can assume each is unconstrained. I then imported each v.net.iso output layer into a single results table in postgis, assigning the plant id as an additional field. Also in postgis I created a regular grid (hex grid specifically) and computed an average travel cost for each grid cell grouped by plant id. the reason for using a regular grid is that we can generate a cost value based on a consistent grid cell geometry, whereas v.net.iso breaks linestrings on the iso contours, and therefore line breaks vary for each v.net.iso output. To create a hybrid or least cost surface, you use a SQL min aggregate window function partitioned by grid cell id - from the grid table LEFT JOIN on results table. At this point, you can now start to consider the impact of capacity limitations, or you can add additional plant processing costs. I also do this in PostgreSQL/postgis because it's easy to consider new hybrid cost surfaces by adding additional non- transport costs and capacity limits, as well as a farm production value to the grid cells that contain the farm centroids.You can then use rank over() and case statements to assign grid cells to plants with the lowest cost up to the capacity limits of the plant. Agricultural production is generally seasonal, hence you can generate hybrid cost surfaces based on different crops, yields, output volumes etc without it affecting your unit travel costs. I find the economic modeling component easier in SQL than trying to do economic optimization in a GIS tool not designed for that. HTH Mark Sent from my iPhone On 4 Mar 2015, at 9:48 pm, Daniel Victoria daniel.victo...@gmail.com wrote: Hi Mark, Thanks for your tips. The use of a regular lattice is very interesting. But I won't need anything that fancy so v.net.iso will most likely fill my needs. I'm particularly interested in what you said about capacity limitations and transport costs. How would you go about modeling that? Change the node costs? Thanks Daniel ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] v.net tools with polygons
Hi Daniel, I've done something similar - I call it off road routing, and uses a regular lattice of nodes and arcs. You can then constrain the off road network by closing arcs that cross major watercourses, fence lines or where the terrain or vegetation is non navigatable. For farms, I added paddock boundaries as rings, as well as gate nodes that constrain movement between paddocks. In essence you build a network topology that reflects the off-road aspect of your network. Relevant to mining as well as agriculture. Ok Moritz, Thanks for the tips. I'll try to go the centroids way Cheers Daniel On Tue, Mar 3, 2015 at 5:35 AM, Moritz Lennert mlenn...@club.worldonline.be wrote: On 02/03/15 21:39, Daniel Victoria wrote: Hi list, I'm beginning to learn and use the v.net http://v.net tools in Grass in order to evaluate the distance from several crop fields to a processing plant. I've successfully build the road network with the end nodes but now I'm in doubt. My starting points in the analysis are crop fields, which are polygons. So what is the best (or most common) practice? 1) Use the field centroids as starting nodes? 2) Add field polygon boundaries to the network and run v.net.distance backwards (from mill to fields)? 3) Some other option? I don't think that there is a best practice for this. It all depends on your application and the desired outcome. Do you want average time/distance from anywhere in the field to the plant ? Then probably the centroid is ok. Or do you want distance from the point of the field that is closest to the network ? Then you could get the coordinates of that point through v.distance (with upload=to_x,to_y) and use these points as nodes. Also, if I'm to add the field boundaries to the network, how would I go about it? Should I first v.patch the field with the roads layer and then run v.net http://v.net? Adding field boundaries still does not answer the question of where to put the start/stop point of your paths... If you want to add them to the network then yes, patching would be the best option, AFAIK. Moritz * ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] v.net parallelisation issues
Hi Moritz, Stefan (and Marcus if you’re around?) Every day is a new day. Not IOPs, Not memory, Not CPU… Hmm, how about reboot... As you have suspected, I get no benefit from additional CPUs. Are you sure the problem is CPU-bound ? I started by testing v.net (the maintenance module) in parallel - and observed pretty much linear scaling in performance over and above 2 parallel jobs. So I then proceeded to test v.net.distance… and sure enough, it is now scaling. Really different looking CPU performance profiles to yesterday, when it wasn’t scaling. On top, all the v.net CPUs are humming at 50%, the other 50% used by pg processes. Below are some iostat profiles of the different parallel test. So v.net.distance does parallelise nicely. The neatest way is to launch GNU parallel from within a single grass_batch_job. I will write up on the grass wiki in the coming 2 weeks. Mark :-) ---V.NET --- Single job v.net iostat 1 TOTAL SCRIPT TIME: 284 avg-cpu: %user %nice %system %iowait %steal %idle 15.920.00 35.320.500.00 48.26 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 18.00 8.00 744.00 8744 xvdj 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 20.000.00 34.000.500.00 45.50 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 9.90 0.00 633.66 0640 xvdj 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 22.500.00 35.500.500.00 41.50 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 9.00 0.00 592.00 0592 xvdj 0.00 0.00 0.00 0 0 Two parallel jobs v.net iostat 1 TOTAL SCRIPT TIME: 397 avg-cpu: %user %nice %system %iowait %steal %idle 32.160.00 67.840.000.000.00 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 19.00 0.00 1024.00 0 1024 xvdj 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 39.500.00 60.500.000.000.00 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 19.00 0.00 912.00 0912 xvdj 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 35.500.00 64.500.000.000.00 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 16.00 0.00 1024.00 0 1024 xvdj 0.00 0.00 0.00 0 0 Four parallel jobs v.net iostat 1 TOTAL SCRIPT TIME: 388 avg-cpu: %user %nice %system %iowait %steal %idle 32.920.00 67.080.000.000.00 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 31.00 0.00 1952.00 0 1952 xvdj 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 33.830.00 66.170.000.000.00 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 35.00 0.00 2016.00 0 2016 xvdj 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 32.920.00 67.080.000.000.00 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 61.00 0.00 4032.00 0 4032 xvdj 0.00 0.00 0.00 0 0 ---V.NET.DISTANCE --- Single job v.net.distance iostat 1 TOTAL SCRIPT TIME: 88 avg-cpu: %user %nice %system %iowait %steal %idle 4.000.00 21.000.000.00 75.00 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 0.00 0.00 0.00 0 0 xvdj 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 7.750.00 21.750.000.00 70.50 Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvde 0.00 0.00 0.00 0 0 xvdj 0.00 0.00 0.00 0 0 avg-cpu: %user %nice %system %iowait %steal %idle 6.270.00 22.560.000.00 71.18 Device:tps Blk_read/s Blk_wrtn/s
Re: [GRASS-user] v.net parallelisation issues
Hi Moritz With the second approach (the code I shared in my post), I have 3500 discrete jobs, and I set the number of batches equal to the number of CPUs. Each batch job is despatched to a cpu, where it then pulls from a queue of job id’s that are processed in serial within each batch job. The thinking behind this approach was to allocate jobs across available CPUs as separate batch processes. The other and preferred approach is to launch 1 batch job, and then GNU parallel draws down from the list of 3500 jobs, assigning jobs to worker functions as CPUs become available. This code pattern I’ve had much success with parallelising PostGIS queries etc. As you have suspected, I get no benefit from additional CPUs. Unfortunately I don’t have time on my side, and parallelisation is critical. A fallback is to spin up a cluster of 16 x 2 CPU machines and pre-allocate job-ids to machines, and then write the results back to the master node - but this is not ideal and pathway I am reticent about going down. Do you know anyone who may have attempted to parallelise v.net? I guess the most important question right now is - is it possible to do poor man’s parallelisation with v.net? Anyone? Mark On 13 Feb 2015, at 7:56 pm, Moritz Lennert mlenn...@club.worldonline.be wrote: On 13/02/15 08:39, Mark Wynter wrote: I’ve encountered a bottleneck somewhere with v.net http://v.net when scaling out with GNU Parallel… not sure if its an underlying issue with v.net http://v.net or the way I’m calling the batch jobs? I’ve got 32 CPUs and commensurate RAM. What I’m observing is v.net http://v.net CPU utilisation dropping off in accordance with number of jobs running. And this means that you don't get any gain in duration ? Could it be that as you divide into more batches each batch is smaller and thus needs less CPU ? Moritz ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] v.net parallelisation issues
Getting closer to the bottom of this. On AWS, problems occur once I scale beyond 16 CPUs. c3.4xlarge (16 CPUs, 30GB memory), v.net scales linearly c3.8xlarge (32 CPUs, 60GB memory), v.net doesn't scale AT ALL - the same as having single CPU. I don’t believe its a pg issue, as I run loads of 32 CPU parallel scripts with postgresql as a backend Also, I noticed the same drop off issue with 32 CPUs when I had sqlite as the db driver in grass. For now, I can work with 16 CPUs. Time to enlist the help of a sysadmin to find the bottleneck. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] SQLite Error
Hi Moritz - the SQLite DB seems to be inheriting the schema from the previous PG driver? Even if I db.dropdb…. I don’t encounter this if I create a new location… e.g. test GRASS 7.0.0svn (nodeclean):~ db.dropdb database='/var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db' GRASS 7.0.0svn (nodeclean):~ db.connect driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db' GRASS 7.0.0svn (nodeclean):~ db.connect -p driver: sqlite database: /var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db schema: grass group: grass70 -text -c EPSG:3577 /var/tmp/test GRASS 7.0.0svn (test):~ db.connect -p driver: sqlite database: /var/tmp/test/PERMANENT/sqlite/sqlite.db schema: group: On 12 Feb 2015, at 7:05 pm, Moritz Lennert mlenn...@club.worldonline.be wrote: On 12/02/15 02:14, Mark Wynter wrote: #(2) RESET THE DATABASE DRIVER FOR THE CURRENT MAPSET TO SQLITE db.connect driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db' schema=grass db.connect -p #Output driver: sqlite database: /var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db schema: grass group: [...] #(5) Run v.net.allpairs…. g.remove -f type=vector pattern=temp** v.net.allpairs --overwrite in=nsw_road_network_final_connected_copy cats=1-10 out=temp_8 afcolumn=fwdcost abcolumn=bwdcost DBMI-SQLite driver error: Error in sqlite3_prepare(): unknown database grass DBMI-SQLite driver error: Error in sqlite3_prepare(): unknown database grass I don't think SQLite supports schemas. Moritz ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] SQLite Error
Thanks Moritz - good suggestion - I just need to these as deliverables out to 2 separate clients (nice to get paid work for this stuff) - then I will write up and contribute back via Wiki. The slow rate of writing out the v.net.allpair results from PostgreSQL was due to the sheer volume of line strings, as the number of pairs increased (n^2). Simple math said stop. I’ve since changed my approach and am using v.net.distance in a novel way where the to_cat is the origin, and the from_cat is a string of destinations - this is an equivalent way of generating multiple v.net.paths in a single operation. Moreover, I’m feeding each origin - destination collection into GNU Parallel as a separate job, so it rips through the data at scale! It would be great if you could put this info in a more structured form onto the GRASS wiki, possibly as a FAQ. Moritz ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] v.net parallelisation issues
I’ve encountered a bottleneck somewhere with v.net when scaling out with GNU Parallel… not sure if its an underlying issue with v.net or the way I’m calling the batch jobs? I’ve got 32 CPUs and commensurate RAM. What I’m observing is v.net CPU utilisation dropping off in accordance with number of jobs running. I’ve tried launching a single batch job with single mapset, as well as multiple batch jobs each with their own mapset (and database). I’ve tried both PG and sqlite backends. Same issue. The script at the bottom describes the approach of launching multiple batch jobs each with their own map set.Executing a single batch job, and then launching parallel within the batch script is much cleaner code - but the results are no different. I feel I’m so close, yet so far at such a critical stage of project delivery. Hope someone can help Kind regards Mark RESULTS ONE JOB TOTAL SCRIPT TIME: 70 PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 31313 root 20 0 28876 4080 1284 S 76.5 0.0 0:20.25 sqlite 31293 root 20 0 276m 134m 8320 S 68.5 0.2 0:20.22 v.net.distance — TWO JOBS TOTAL SCRIPT TIME: 96 PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 21391 root 20 0 28876 4080 1284 R 53.0 0.0 0:01.90 sqlite 21392 root 20 0 28876 4080 1284 R 52.6 0.0 0:01.86 sqlite 21380 root 20 0 276m 128m 8320 R 49.3 0.2 0:04.02 v.net.distance 21381 root 20 0 276m 128m 8320 S 48.3 0.2 0:03.97 v.net.distance — FOUR JOBS TOTAL SCRIPT TIME: 187 PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 6953 mark 20 0 180m 100m 9520 S 63.6 0.2 1:47.39 x2goagent 23025 root 20 0 28876 4080 1284 S 21.5 0.0 0:02.03 sqlite 23026 root 20 0 28876 4080 1284 R 19.9 0.0 0:02.08 sqlite 23027 root 20 0 28876 4080 1284 S 19.5 0.0 0:01.87 sqlite 23028 root 20 0 28876 4080 1284 S 19.5 0.0 0:01.84 sqlite 23014 root 20 0 276m 128m 8320 R 18.5 0.2 0:04.06 v.net.distance 23012 root 20 0 276m 128m 8320 R 17.5 0.2 0:03.91 v.net.distance 23011 root 20 0 276m 128m 8320 S 16.9 0.2 0:04.13 v.net.distance 23015 root 20 0 276m 128m 8320 R 16.9 0.2 0:03.80 v.net.distance — EIGHT JOBS TOTAL SCRIPT TIME: 373 PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 27157 root 20 0 28876 4088 1284 S 19.5 0.0 0:42.39 sqlite 27162 root 20 0 28876 4088 1284 R 16.9 0.0 0:40.60 sqlite 6953 mark 20 0 181m 101m 9520 S 16.5 0.2 2:18.86 x2goagent 27154 root 20 0 28876 4088 1284 S 16.5 0.0 0:39.38 sqlite 27153 root 20 0 28876 4088 1284 S 16.2 0.0 0:35.60 sqlite 27156 root 20 0 28876 4088 1284 R 16.2 0.0 0:38.18 sqlite 27161 root 20 0 28876 4088 1284 S 15.9 0.0 0:40.96 sqlite 27155 root 20 0 28876 4088 1284 S 15.6 0.0 0:38.41 sqlite 27104 root 20 0 284m 139m 8332 S 14.9 0.2 0:39.94 v.net.distance 27158 root 20 0 28876 4088 1284 R 14.6 0.0 0:37.49 sqlite 27095 root 20 0 284m 138m 8332 S 14.2 0.2 0:34.48 v.net.distance 27099 root 20 0 284m 138m 8332 S 14.2 0.2 0:38.27 v.net.distance 27101 root 20 0 284m 139m 8332 R 14.2 0.2 0:38.80 v.net.distance 27105 root 20 0 284m 139m 8332 R 14.2 0.2 0:37.95 v.net.distance 27093 root 20 0 284m 138m 8332 R 13.9 0.2 0:32.64 v.net.distance 27102 root 20 0 284m 140m 8332 R 13.6 0.2 0:40.90 v.net.distance 27094 root 20 0 284m 138m 8332 R 13.2 0.2 0:35.78 v.net.distance — WORKER FUNCTION#
Re: [GRASS-user] SQLite Error
Haven’t come across anything to that effect Can you set schema='' or something like that ? Moritz I’ve reverted back to using solely PG driver - notwithstanding this SQLite issue. As a prologue to the several issues I encountered with Grass over the last few days… The slow rate of writing out the v.net.allpair results from PostgreSQL was due to the sheer volume of line strings, as the number of pairs increased (n^2). Simple math said stop. I’ve since changed my approach and am using v.net.distance in a novel way where the to_cat is the origin, and the from_cat is a string of destinations - this is an equivalent way of generating multiple v.net.paths in a single operation. Moreover, I’m feeding each origin - destination collection into GNU Parallel as a separate job, so it rips through the data at scale! Still haven’t resolved the issues of v.net.allpair and v.net.distance geometry features being dropped with v.out.ogr(to postgres) and v.out.postgis. Luckily, the two step process of v.out.ogr(to shapefile), and then ogr2ogr doesn’t encounter this problem. Hence, thats what I’ve gone with. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] Help with defining DB Drivers for input and outputs
Within the same mapset, is it possible to specify SQLITE as the database for the output layer, whilst using a map as an input that has a PG driver? I’ve read the manual and tried a few approaches, but I found the outputs still get written to PG. May be I’ve missed a critical step? See messages below. I also experimented with v.add.table and v.add.columns to the empty map, to no avail. I want the outputs written to sqlite, because v.out.ogr and v.out.postgis when the attributes are stored in PostgreSQL seems unworkably slow (which may be a pg_conf issue but my DB is performing well at scale for non-GRASS parallelised workflows)? I want to keep the input attributes still in PostgreSQL because its the road table, and the fwdcost and bwdcost columns get updated by data from non-GRASS schema. I need the fastest pipeline performance possible for when I scale the solution out (which will be done using GNU Parallel). Right now I’m stuck between a rock and a hard place. Really hoping someone can help me out… :-) Thanks Mark #CREATE EMPTY MAP TO HOLD THE OUTPUTS g.remove -f type=vector pattern=temp** layername=temp_8 v.edit --overwrite tool=create map=$layername v.db.connect map=$layername table=$layername driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db’ v.info $layername #RUN THE ANALYSIS, WHERE THE INPUT LAYER ATTRIBUTES RESIDE IN POSTGRESQL v.net.allpairs --overwrite in=nsw_road_network_final_connected cats=1-100 out=$layername afcolumn=fwdcost abcolumn=bwdcost # NO GOOD - THE OUTPUT ATTRIBUTE TABLE STILL APPEARS IN PGL ## GRASS 7.0.0svn (nodeclean):~ v.edit --overwrite tool=create map=$layername Building topology for vector map temp_8@PERMANENT... Registering primitives... 0 primitives registered 0 vertices registered Building areas... 0 areas built 0 isles built Attaching islands... Attaching centroids... Number of nodes: 0 Number of primitives: 0 Number of points: 0 Number of lines: 0 Number of boundaries: 0 Number of centroids: 0 Number of areas: 0 Number of isles: 0 v.edit complete. GRASS 7.0.0svn (nodeclean):~ v.db.connect map=$layername table=$layername driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db' WARNING: Table temp_8 does not exist in database $GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db The table temp_8 is now part of vector map temp_8 and may be deleted or overwritten by GRASS modules DBMI-SQLite driver error: Unable to create index: create unique index if not exists temp_8_cat on temp_8 ( cat ) no such table: main.temp_8 DBMI-SQLite driver error: Unable to create index: create unique index if not exists temp_8_cat on temp_8 ( cat ) no such table: main.temp_8 WARNING: Cannot create index Select privileges were granted on the table GRASS 7.0.0svn (nodeclean):~ v.db.connect -p map=$layername Vector map temp_8 is connected by: layer 1/temp_8 table temp_8 in database /var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db through driver sqlite with key cat GRASS 7.0.0svn (nodeclean):~ v.info $layername ++ | Name:temp_8| | Mapset: PERMANENT | | Location:nodeclean | | Database:/var/tmp | | Title: | | Map scale: 1:1 | | Name of creator: root | | Organization: | | Source date: Wed Feb 11 22:49:13 2015 | | Timestamp (first layer): none | || | Map format: native| || | Type of map: vector (level: 2) | || | Number of points: 0 Number of centroids: 0 | | Number of lines:0 Number of boundaries: 0 | | Number of areas:0 Number of islands:0 | || | Map is 3D: No | | Number of dblinks: 1| || | Projection: Albers Equal Area
[GRASS-user] SQLite Error
Thanks Moritz. I am getting an SQLite error when trying to run v.net.allpairs. This is my latest workflow, incl. my checks and output messages of each step. #(1) CREATE A COPY OF MY ROAD NETWORK, AND DROP THE ATTRIBUTE TABLE (CURRENTLY SET WITH PG DRIVER) db.connect driver=pg database=host=localhost,dbname=osm_au db.login user=postgres [pass=password] db.connect -p db.tables -p g.copy --overwrite vect=nsw_road_network_final_connected,nsw_road_network_final_connected_copy v.db.droptable nsw_road_network_final_connected_copy v.db.droptable nsw_road_network_final_connected_copy -f # Outputs confirm my copied map no longer is connected to an attribute table Dropping table grass.nsw_road_network_final_connected_copy... Current attribute table link(s): Map nsw_road_network_final_connected_copy is not connected to a database #(2) RESET THE DATABASE DRIVER FOR THE CURRENT MAPSET TO SQLITE db.connect driver=sqlite database='$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db' schema=grass db.connect -p #Output driver: sqlite database: /var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db schema: grass group: #(3) COPY THE ATTRIBUTE TABLE OF THE ORIGINAL ROAD NETWORK IN POSTGRESQL TO THE NEW SQLITE DB db.copy from_driver=pg from_database=host=localhost,dbname=osm_au from_table=grass.nsw_road_network_final_connected to_driver=sqlite to_database=/var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db to_table=nsw_road_network_final_connected_sqlite db.copy from_driver=pg from_database=host=localhost,dbname=osm_au from_table=grass.nsw_tz_centroids_nodes to_driver=sqlite to_database=/var/tmp/nodeclean/PERMANENT/sqlite/sqlite.db to_table=nsw_tz_centroids_nodes_sqlite db.tables -p #Output confirms table successfully copied nsw_road_network_final_connected_sqlite nsw_tz_centroids_nodes_sqlite #(4) CONNECT THE COPIED ATTRIBUTE TABLES TO THE COPIED MAP IN (1) v.db.connect map=nsw_road_network_final_connected_copy table=nsw_road_network_final_connected_sqlite layer=1 v.db.connect map=nsw_road_network_final_connected_copy table=nsw_tz_centroids_nodes_sqlite layer=2 v.category nsw_road_network_final_connected_copy op=report v.category nsw_road_network_final_connected_copy op=report Layer/table: 1/nsw_road_network_final_connected_sqlite type countminmax point 0 0 0 line 467531 1 465943 boundary 0 0 0 centroid 0 0 0 area 0 0 0 face 0 0 0 kernel 0 0 0 all 467531 1 465943 Layer: 2 type countminmax point 3514 1 3514 line 0 0 0 boundary 0 0 0 centroid 0 0 0 area 0 0 0 face 0 0 0 kernel 0 0 0 all 3514 1 3514 v.info -c nsw_road_network_final_connected_copy layer=1 Displaying column types/names for database connection of layer 1: INTEGER|cat INTEGER|lid INTEGER|osm_id CHARACTER|name INTEGER|bridge INTEGER|tunnel INTEGER|oneway CHARACTER|type INTEGER|tid CHARACTER|poa_code INTEGER|reversed INTEGER|open DOUBLE PRECISION|length DOUBLE PRECISION|fwdcost DOUBLE PRECISION|bwdcost v.info -c nsw_road_network_final_connected_copy layer=2 Displaying column types/names for database connection of layer 2: INTEGER|cat INTEGER|tz_code11 CHARACTER|tz_name11 INTEGER|sa2_main11 INTEGER|sa3_code11 #(5) Run v.net.allpairs…. g.remove -f type=vector pattern=temp** v.net.allpairs --overwrite in=nsw_road_network_final_connected_copy cats=1-10 out=temp_8 afcolumn=fwdcost abcolumn=bwdcost DBMI-SQLite driver error: Error in sqlite3_prepare(): unknown database grass DBMI-SQLite driver error: Error in sqlite3_prepare(): unknown database grass ERROR: Unable to create table: 'create table grass.temp_8_1 ( cat integer, from_cat integer, to_cat integer, cost double precision)' Segmentation fault On 12/02/15 00:12, Mark Wynter wrote: Within the same mapset, is it possible to specify SQLITE as the database for the output layer, whilst using a map as an input that has a PG driver? I?ve read the manual and tried a few approaches, but I found the outputs still get written to PG. May be I?ve missed a critical step? See messages below. I also experimented with v.add.table and v.add.columns to the empty map, to no avail. You should use db.connect to set your default settings to SQLite, then run your command. By overwriting your map in the v.net.allpairs call you overwrite everything, including any db connections set before and you use the default mapset connection settings which seem to be set to PG. Moritz ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] v.out.postgis and v.out.ogr - inconsistencies in output
As context, I’m running analysis using v.net.allpairs, whereby vector geometries are often common to more than 1 origin-destination journey path, hence the line segments appear multiple times but with unique cat values, and from_cat and to_cat combinations. When exporting the output layer of v.net.allpairs back into PostGIS, only the third method returns the correct number of geometry features - and this requires first exporting to shapefile and then importing using ogr2ogr. Methods (1) and (2) discard line strings that appear more than once (despite them having unique cat values). This is not what I was expecting. The third method correctly treats each linestring feature at unique, hence assigns a unique ogc_fid to each feature. Are there flags for methods (1) and (2) that deal with this issue - or is method (3) the only feasible option? (1) v.out.postgis --o input=temp_8 output=PG:host=localhost dbname=osm_au user=postgres password=password olayer=jtw.temp_8_pg options=GEOMETRY_NAME=wkb_geometry,SRID=3577 134 Features Written (2) v.out.ogr --o input=temp_8 type=line output=PG:host=localhost dbname=osm_au user=postgres password=password output_layer=jtw.temp_8_ogr format=PostgreSQL 134 Features Written (3) v.out.ogr --o input=temp_8 output=/var/tmp/temp_8 type=line ogr2ogr -overwrite -f PostgreSQL PG:host=localhost dbname=osm_au user=postgres password=password /var/tmp/temp_8/temp_8.shp -nln jtw.temp_8_ogr -s_srs EPSG:3577 -t_srs EPSG:3577 -a_srs EPSG:3577 -nlt LINESTRING 1323 Features Written Kind regards Mark ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] v.out.postgis dropping cats
Hi list Context is I’m running v.net.allpairs over a large road network - the output is all combination origin - destination pairs, and the individual line segments that make up a journey path. Individual line segments will appear more than once where they’re common to more than 1 journey paths. I’m running PostgreSQL as the db driver. If I v.out.postgis, its reassigning cats - which fouls my results. See message below. If I v.out.ogr, I don’t experience this problem - except it is unworkably slow e.g. 25 mins for circa 1 features. The slow speed writing to file, or writing STOUT when PG Driver I haven’t got to the bottom of either? v.out.postgis -t is my preferred option, and I then join the attribute tables in PostgreSQL using the cat values - which avoids shovelling data unnecessarily between PostGIS grass PostGIS WARNING: Feature has more categories, using category 9802 (from layer 1) WARNING: Feature has more categories, using category 99 (from layer 1) WARNING: Feature has more categories, using category 9802 (from layer 1) WARNING: Feature has more categories, using category 99 (from layer 1) 100% WARNING: Feature has more categories, using category 9802 (from layer 1) 100 features without category or from different layer skipped Building topology for vector map temp_8pg@PERMANENT... Using external data format 'PostgreSQL' (feature type 'linestring') Building pseudo-topology over simple features... Registering primitives... 6613 primitives registered 37675 vertices registered Number of nodes: 4710 Number of primitives: 6613 Number of points: 0 Number of lines: 6613 Number of boundaries: 0 Number of centroids: 0 Number of areas: 0 Number of isles: 0 v.out.postgis complete. 6613 features (linestring type) written to temp_8pg. Hope someone can help - I’m at a critical point in the analysis where I need to scale out the solution. Kind regards Mark___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] SQLITE db locking problem
Thanks Marcus - am I right in understanding that read and write resulted in the db being opened twice? Was this confined to just v.patch? Anything a user needs to do differently, apart from update? ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] SQLITE db locking problem - update
Still no luck solving SQLite db locking issue. I’ve opted for PG db instead - a nicer solution insofar PG is where the data comes from and goes to in my analytics pipeline. There’s also an efficiency dividend, thanks to the v.out.postgis -t flag - which exports the grass geometries and cat values, but not the attribute table - which is much faster than v.out.ogr, as the grass attribute data is already in PG. I then use psql to do a join in PostGIS on the geom and attribute cat values to build the final table for subsequent analysis in PostGIS. :-)) v.out.postgis -t is a great feature - thanks Martin Landa!!! -- Message: 1 Date: Mon, 19 Jan 2015 00:53:58 +1030 From: Mark Wynter m...@dimensionaledge.com To: grass grass-user@lists.osgeo.org Subject: [GRASS-user] SQLITE db locking problem Message-ID: 6bd86e4f-3b17-42d1-acc3-25e3a3c29...@dimensionaledge.com Content-Type: text/plain; charset=windows-1252 Hi everyone I?m having issues with SQLITE as the db driver - I encounter this BUSY warning. Only occurs when patching a particular map, in this case ?temp_nbt_cleaned which is the largest of the datasets being patched. If I work with smaller datasets, it works fine. Pre-patching, SQLITE DB size is only 170MB. I?ve come across various threads reporting something similar with db locking. But couldn?t find a resolution. I?m running grass70 on Centos65, hosted on AWS.. SQLite is version 3.6.23.1 This is the error message. GRASS 7.0.0svn (nodeclean):/var/tmp v.patch -e input=temp_t_cleaned,temp_b_cleaned,temp_nbt_cleaned out=temp_patched Patching vector map temp_t_cleaned... Patching vector map temp_b_cleaned... Patching vector map temp_nbt_cleaned... WARNING: Busy SQLITE db, already waiting for 10 seconds? WARNING: Busy SQLITE db, already waiting for 20 seconds? etc etc I encounter no problems patching the maps, including ?temp_nbt_cleaned if I use PG db driver, or DBF driver. I?m more than comfortable using PG in the backend, but I find v.out.postgis too slow when exporting the resultant dataset back to postgis (possibly because its simultaneously trying to read and write to the same db). I haven?t tested whether v.out.ogr quicker than v.out.postgis. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] SQLITE db locking problem
Hi everyone I’m having issues with SQLITE as the db driver - I encounter this BUSY warning. Only occurs when patching a particular map, in this case “temp_nbt_cleaned which is the largest of the datasets being patched. If I work with smaller datasets, it works fine. Pre-patching, SQLITE DB size is only 170MB. I’ve come across various threads reporting something similar with db locking. But couldn’t find a resolution. I’m running grass70 on Centos65, hosted on AWS.. SQLite is version 3.6.23.1 This is the error message. GRASS 7.0.0svn (nodeclean):/var/tmp v.patch -e input=temp_t_cleaned,temp_b_cleaned,temp_nbt_cleaned out=temp_patched Patching vector map temp_t_cleaned... Patching vector map temp_b_cleaned... Patching vector map temp_nbt_cleaned... WARNING: Busy SQLITE db, already waiting for 10 seconds… WARNING: Busy SQLITE db, already waiting for 20 seconds… etc etc I encounter no problems patching the maps, including “temp_nbt_cleaned if I use PG db driver, or DBF driver. building topology for vector map temp_patched@PERMANENT... Registering primitives... 1860504 primitives registered 9027638 vertices registered Building areas... 100% 0 areas built 0 isles built Attaching islands... Attaching centroids... 100% Number of nodes: 1457611 Number of primitives: 1860504 Number of points: 0 Number of lines: 1860504 Number of boundaries: 0 Number of centroids: 0 Number of areas: 0 Number of isles: 0 Intersections at borders will have to be snapped Lines common between files will have to be edited The header information also may have to be edited v.patch complete. 3 vector maps patched I’m more than comfortable using PG in the backend, but I find v.out.postgis too slow when exporting the resultant dataset back to postgis (possibly because its simultaneously trying to read and write to the same db). I haven’t tested whether v.out.ogr quicker than v.out.postgis. Any assistance resolving the SQLite issue would be greatly appreciated. Kind regards Mark ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Checking and fixing roads for network analysis
Hi Daniel - not sure why you’re getting that error - I’ll leave that to someone more familiar with the backend workings to comment on. In the interests of you moving forward, one approach might be that you simply manually add nodes in say QGIS before running v.snap. By that, I mean: 1) In QGIS, open up the shapefile 2) Click the Edit layer icon 3) Find the line at that is closest to the point you wish to snap to another line end. Double click on the line at the point where you want to insert a new node - a new node should appear. Make sure the other line end is within your snap tolerance of your newly added node. If not, drag the line end so that it is. 4) Repeat for other “disconnected” parts of your road network 5) Save. 6) Import into GRASS and run v.clean. You still need to break the lines that cross - which are not tunnels or bridges. Even if v.clean tool=break is working correctly, in my experience you may still encounter unconnected T junctions where the lines undershoot, and there is no node on the adjacent line to which the line ending can snap to. The QGIS approach I’ve outlined will fix that - provided you know where the disjointed lines are. If you’re working with a large scale road networks, its not practical to manually detect or fix such anomalies - which is why I developed my own PostGIS method I outlined yesterday. HTH Mark On 17 Jan 2015, at 4:05 am, Daniel Victoria daniel.victo...@gmail.com wrote: Hi Mark, I tried using v.clean with break, rmdup and snap. However, when I try to run v.net.components I get an error saying that v.net.components input=rede_brk_rmdup_snap@PERMANENT node_layer=1 output=component_brk_rmdup_snap method=weak Building graph... Registering arcs... Flattening the graph... Graph was built ERROR: Cannot insert new record: insert into component_brk_rmdup_snap values (3689, 1) I'm using Grass v7RC1 on Windows and v.net.components work if I use only the snap tool in my roads network Cheers Daniel ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] SOLVED: Compilation Errors Installing GRASS 70 on Centos 6.5
Re-checked out SVN Clean install directory Installed without a hitch :-) ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] Compilation Errors Installing GRASS 70 on Centos 6.5
Upon running ‘make’, I get errors consistent with the message below. Any clues as to the source of the Error 1 messages, and what the fix is? Thanks Mark # make -C v.centroids || echo /home/grass/grass70/releasebranch_7_0/scripts/v.centroids /home/grass/grass70/releasebranch_7_0/error.log make[3]: Entering directory `/home/grass/grass70/releasebranch_7_0/scripts/v.centroids' if [ /home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/scripts/v.centroids != ] ; then GISRC=/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/demolocation/.grassrc70 GISBASE=/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu PATH=/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/bin:/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/bin:/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/scripts:$PATH PYTHONPATH=/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/etc/python:/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/gui/wxpython:$PYTHONPATH LD_LIBRARY_PATH=/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/bin:/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/scripts:/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/lib:/home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/lib: LC_ALL=C /home/grass/grass70/releasebranch_7_0/dist.x86_64-unknown-linux-gnu/scripts/v.centroids --html-description /dev/null | grep -v '/body\|/html' v.centroids.tmp.html ; fi make[3]: *** [v.centroids.tmp.html] Error 1 rm v.centroids.tmp.html make[3]: Leaving directory `/home/grass/grass70/releasebranch_7_0/scripts/v.centroids’ #Essentially the same errors for…….. Errors in: /home/grass/grass70/releasebranch_7_0/scripts/d.correlate /home/grass/grass70/releasebranch_7_0/scripts/d.out.file /home/grass/grass70/releasebranch_7_0/scripts/d.polar /home/grass/grass70/releasebranch_7_0/scripts/d.rast.edit /home/grass/grass70/releasebranch_7_0/scripts/d.rast.leg /home/grass/grass70/releasebranch_7_0/scripts/d.redraw /home/grass/grass70/releasebranch_7_0/scripts/d.shadedmap /home/grass/grass70/releasebranch_7_0/scripts/d.vect.thematic /home/grass/grass70/releasebranch_7_0/scripts/db.dropcolumn /home/grass/grass70/releasebranch_7_0/scripts/db.droptable /home/grass/grass70/releasebranch_7_0/scripts/db.in.ogr /home/grass/grass70/releasebranch_7_0/scripts/db.out.ogr /home/grass/grass70/releasebranch_7_0/scripts/db.test /home/grass/grass70/releasebranch_7_0/scripts/g.extension.all /home/grass/grass70/releasebranch_7_0/scripts/g.manual /home/grass/grass70/releasebranch_7_0/scripts/i.colors.enhance /home/grass/grass70/releasebranch_7_0/scripts/i.image.mosaic /home/grass/grass70/releasebranch_7_0/scripts/i.in.spotvgt /home/grass/grass70/releasebranch_7_0/scripts/i.oif /home/grass/grass70/releasebranch_7_0/scripts/i.pansharpen /home/grass/grass70/releasebranch_7_0/scripts/i.spectral /home/grass/grass70/releasebranch_7_0/scripts/i.tasscap /home/grass/grass70/releasebranch_7_0/scripts/m.proj /home/grass/grass70/releasebranch_7_0/scripts/r.blend /home/grass/grass70/releasebranch_7_0/scripts/r.buffer.lowmem /home/grass/grass70/releasebranch_7_0/scripts/r.colors.stddev /home/grass/grass70/releasebranch_7_0/scripts/r.fillnulls /home/grass/grass70/releasebranch_7_0/scripts/r.grow /home/grass/grass70/releasebranch_7_0/scripts/r.in.aster /home/grass/grass70/releasebranch_7_0/scripts/r.in.srtm /home/grass/grass70/releasebranch_7_0/scripts/r.in.wms /home/grass/grass70/releasebranch_7_0/scripts/r.mask /home/grass/grass70/releasebranch_7_0/scripts/r.out.xyz /home/grass/grass70/releasebranch_7_0/scripts/r.pack /home/grass/grass70/releasebranch_7_0/scripts/r.plane /home/grass/grass70/releasebranch_7_0/scripts/r.reclass.area /home/grass/grass70/releasebranch_7_0/scripts/r.rgb /home/grass/grass70/releasebranch_7_0/scripts/r.tileset /home/grass/grass70/releasebranch_7_0/scripts/r.unpack /home/grass/grass70/releasebranch_7_0/scripts/r3.in.xyz /home/grass/grass70/releasebranch_7_0/scripts/v.build.all /home/grass/grass70/releasebranch_7_0/scripts/v.centroids /home/grass/grass70/releasebranch_7_0/scripts/v.convert.all /home/grass/grass70/releasebranch_7_0/scripts/v.db.addcolumn /home/grass/grass70/releasebranch_7_0/scripts/v.db.addtable #The steps I followed to this point: #Instructions for installing the dependencies with a couple of additional packages needed #http://grasswiki.osgeo.org/wiki/Compile_and_Install#CentOS #yum -y install readline readline-devel cd /home/grass/grass70/releasebranch_7_0 CFLAGS=-g ./configure \ --enable-debug \ --with-cxx \ --without-ffmpeg \ --with-gdal=/usr/bin/gdal-config \ --with-geos \ --with-readline \ --with-freetype=yes \ --with-freetype-includes=/usr/include/freetype2/ \ --enable-largefile=yes \ --with-proj-share=/usr/share/proj/ \ --with-geos=/usr/bin/geos-config \ --with-cairo \ --with-tcltk-includes=/usr/lib64/tcl8.5/ \
[GRASS-user] v.net multi-threaded parallel processing
Curious to hear if anyone has considered or is considering v.net in a high performance computing environment? - Mark ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] speeding up v.clean for large datasets
Thanks Markus for the explanation. I've set PostGIS as my backend. Will revert as I get more into v.net On 22/04/2013, at 8:20 PM, Markus Metz wrote: On Mon, Apr 22, 2013 at 11:03 AM, Mark Wynter m...@dimensionaledge.com wrote: Thanks Marcus. Tried sqlite backend suggestion - no improvement - then read that that sqlite is the default backend for grass7. I suspect the complexity of the input dataset may be the contributing factor. For example, I ran v.clean over the already cleaned OSM dataset (2.6M lines), and it took only a few minutes since there were no intersections and no duplicates to remove. I tested with a OSM road vector with 2.6M lines, the output has 5.3M lines: lots of intersections and duplicates which were cleaned in less than 15 minutes. I am surprised that you experience slow removal of duplicates, breaking lines should take much longer. About why removing duplicates takes longer at the end: when you have 5 lines that could be duplicates you could check 1 with 2, 3, 4, 5 2 with 1, 3, 4, 5 3 with 1, 2, 4, 5 4 with 1, 2, 3, 5 5 with 1, 2, 3, 4 or checking each combination only once: 1 with 2, 3, 4, 5 2 with 3, 4, 5 3 with 4, 5 4 with 5 alternatively 2 with 1 3 with 1, 2 4 with 1, 2, 3 5 with 1, 2, 3, 4 The current implementation uses the latter. Markus M Something is wrong there. Your dataset has 971074 roads, I tested with an OSM dataset with 2645287 roads, 2.7 times as many as in your dataset. Cleaning these 2645287 lines took me less than 15 minutes. I suspect a slow database backend (dbf). Try to use sqlite as database backend: db.connect driver=sqlite database=$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db Do not substitute the variables. HTH, Markus M ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Grass7 - Inconsistencies with v.out.ogr and v.out.postgis
Do you have fresh installation from svn? Thanks Martin. Good advice - I just pulled the latest from svn! v.out.postgis now works, generating correct feature count and SRID=2193 yet... v.out.ogr when writing directly to PostGIS assigns SRID 900915 to the geometry column - when it should be 2193. SRID 2193 is in the PostGIS spatial_ref_sys table. v.out.ogr when writing to shapefile correctly assigns 2193 - see below. Any idea as to why this is happening with v.out.ogr and PostGIS specifically? I suspect it has something to do with SRID 2193 not being recognised (somewhere in the process), hence a new SRID is created? Yet it works with v.out.postgis. SELECT postgis_full_version(); postgis_full_version --- POSTGIS=2.1.0SVN r11004 GEOS=3.3.8-CAPI-1.7.8 PROJ=Rel. 4.8.0, 6 March 2012 GDAL=GDAL 1.9.2, released 2012/10/08 LIBXML=2.7.8 LIBJSON=UNKNOWN TOPOLOGY RASTER (1 row) v.out.ogr input=osm_nz_roads_2193_cleaned type=line dsn=/var/tmp/osm_nz_roads_export.shp ogrinfo -so osm_nz_roads_export.shp osm_nz_roads_export INFO: Open of `osm_nz_roads_export.shp' using driver `ESRI Shapefile' successful. Layer name: osm_nz_roads_export Geometry: Line String Feature Count: 176783 Extent: (1148817.072845, 4793789.625743) - (2089039.165212, 6193819.546965) Layer SRS WKT: PROJCS[Transverse_Mercator, GEOGCS[GCS_grs80, DATUM[New_Zealand_Geodetic_Datum_2000, SPHEROID[Geodetic_Reference_System_1980,6378137,298.257222101]], PRIMEM[Greenwich,0], UNIT[Degree,0.017453292519943295]], PROJECTION[Transverse_Mercator], PARAMETER[latitude_of_origin,0], PARAMETER[central_meridian,173], PARAMETER[scale_factor,0.9996], PARAMETER[false_easting,160], PARAMETER[false_northing,1000], UNIT[Meter,1]] cat: Integer (10.0) osm_id: Integer (10.0) name: String (255.0) highway: String (255.0) bridge: Integer (10.0) oneway: Integer (10.0) driveable: Integer (10.0) trainable: Integer (10.0) walkable: Integer (10.0) cycleable: Integer (10.0) ignore: Integer (10.0) ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Grass7 - Inconsistencies with v.out.ogr and v.out.postgis
v.out.ogr in=roadsmajor dsn=PG:dbname=pgis_grass format=PostgreSQL - $ db.select data=pgis_grass dri=pg sql=select count(*) from roadsmajor -c 355 db.select data=pgis_grass dri=pg sql=select srid from geometry_columns where f_table_name = 'roadsmajor' -c 900914 (new srid added to the table `spatial_ref_sys`) Hi Martin, just noticed PostGIS assigned 900914 to your table above, yet below its 3358. This is the same as what I'm experiencing. When I inspect the details of 900914, it mirrors the correct SRID - its just labelled 900914 instead? Yet v.out.postgis handles everything the way you would expect $ v.out.postgis in=roadsmajor dsn=PG:dbname=pgis_grass options=GEOMETRY_NAME=wkb_geometry,SRID=3358 --o $ db.select data=pgis_grass dri=pg sql=select count(*) from roadsmajor -c 355 $ db.select data=pgis_grass dri=pg sql=select srid from geometry_columns where f_table_name = 'roadsmajor' -c 3358 Everything seems to be OK. Do you have fresh installation from svn? Martin ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] speeding up v.clean for large datasets
Thanks Marcus. Tried sqlite backend suggestion - no improvement - then read that that sqlite is the default backend for grass7. I suspect the complexity of the input dataset may be the contributing factor. For example, I ran v.clean over the already cleaned OSM dataset (2.6M lines), and it took only a few minutes since there were no intersections and no duplicates to remove. Something is wrong there. Your dataset has 971074 roads, I tested with an OSM dataset with 2645287 roads, 2.7 times as many as in your dataset. Cleaning these 2645287 lines took me less than 15 minutes. I suspect a slow database backend (dbf). Try to use sqlite as database backend: db.connect driver=sqlite database=$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite/sqlite.db Do not substitute the variables. HTH, Markus M ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] speeding up v.clean for large datasets
Hi All, we're looking for ways to speed up the cleaning of a large OSM road network (relating to Australia). We're running on a large Amazon AWS EC2 instance. What we've observed is exponential growth in time taken as number of linestrings increases. This means it's taking about 3 days to clean entire network. We were wondering if we were to split the dataset into say 4 subregions, and clean each separately, is it then possible to patch them back together at the end without having to run v.clean afterwards? We want to be able to run v.net over the entire network spanning the subregions. Alternatively, has anyone found a way to speed up v.clean for large network datasets? GRASS 6.4.3svn (road_network):/data/grassdata v.clean input=osm_roads output=osm_roads_cleaned tool=break,rmdupl -- Tool: Threshold Break: 0.00e+00 Remove duplicates: 0.00e+00 -- Copying vector lines... Rebuilding parts of topology... Building topology for vector map osm_roads_cleaned... Registering primitives... 971074 primitives registered 13142529 vertices registered Number of nodes: 1458192 Number of primitives: 971074 Number of points: 0 Number of lines: 971074 Number of boundaries: 0 Number of centroids: 0 Number of areas: - Number of isles: - -- Tool: Break lines at intersections ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] speeding up v.clean for large datasets
Thanks Markus. Upgraded to GRASS 7, and re-ran v.clean on same OSM Australia dataset. Substantially faster. The bulk of the time related to removal of duplicates, and it got exponentially slower as the process approached 100%. Overall it took 12 hours but I'm wondering how it would perform if we were to repeat v.clean for even larger road networks e.g. USA or Europe? I'm tempted to try dividing the input dataset into say 4 smaller subregions (i.e. vector tiles), and then try patching them back. I read that we will still need to run v.clean over the patched datasets to remove duplicates. Since the only duplicates should be nodes along the common tile edges, is there a way to in effect constrain the v.clean process to slithers containing the common edges? I've had a quick go at g.region but to no avail. Thanks GRASS 7.0.svn (PERMANENT):/data/grassdata v.clean input=osm_roads_split output=osm_roads_split_cleaned tool=break type=line -c -- Tool: Threshold Break: 0 -- Copying vector features... Copying features... 100% Rebuilding parts of topology... Building topology for vector map osm_roads_split_cleaned@PERMANENT... Registering primitives... 971074 primitives registered 13142529 vertices registered Number of nodes: 1458192 Number of primitives: 971074 Number of points: 0 Number of lines: 971074 Number of boundaries: 0 Number of centroids: 0 Number of areas: - Number of isles: - -- Tool: Break lines at intersections 100% Tool: Remove duplicates 100% -- Rebuilding topology for output vector map... Building topology for vector map osm_roads_split_cleaned@PERMANENT... Registering primitives... 2462829 primitives registered 13322052 vertices registered Building areas... 100% 0 areas built 0 isles built Attaching islands... Attaching centroids... 100% Number of nodes: 1819237 Number of primitives: 2462829 Number of points: 0 Number of lines: 2462829 Number of boundaries: 0 Number of centroids: 0 Number of areas: 0 Number of isles: 0 On 19/04/2013, at 6:07 PM, Markus Metz wrote: On Fri, Apr 19, 2013 at 9:06 AM, Mark Wynter m...@dimensionaledge.com wrote: Hi All, we're looking for ways to speed up the cleaning of a large OSM road network (relating to Australia). We're running on a large Amazon AWS EC2 instance. What we've observed is exponential growth in time taken as number of linestrings increases. This means it's taking about 3 days to clean entire network. We were wondering if we were to split the dataset into say 4 subregions, and clean each separately, is it then possible to patch them back together at the end without having to run v.clean afterwards? We want to be able to run v.net over the entire network spanning the subregions. Alternatively, has anyone found a way to speed up v.clean for large network datasets? Yes, implemented in GRASS 7 ;-) Also, when breaking lines it is recommended to split the lines first in smaller segments with v.split using the vertices option. Then run v.clean tool=break. After that, use v.build.polylines to merge lines again. Or use in GRASS 7 the -c flag with v.clean tool=break type=line. The rmdupl tool is then automatically added, and the splitting and merging is done internally. Markus M ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Problem accessing GRASS from GRASS from PostgreSQL9.1/PostGIS2.0 via pg-python or plpythonu
Thanks Andrew I suggest running a test script that reports sys.path to confirm that the actual Python module search path seen by the Postgres server's environment is as expected. From the python console, logged in as posgres user postgres@ip-10-252-74-140:/home/ubuntu$ python -c 'import sys; print sys.version, sys.path' 2.7.3 (default, Apr 20 2012, 22:39:59) [GCC 4.6.3] ['', '/home/ubuntu', '/usr/local/grass-6.4.3svn/etc/python', '/usr/local/grass-6.4.3svn/etc/python/grass', '/usr/local/grass-6.4.3svn/etc/python/grass/script', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode'] From postgresql test=# CREATE FUNCTION pyversion() RETURNS text AS $$ test$# import sys test$# return sys.version + '\n' + '\n'.join(sys.path) test$# $$ LANGUAGE plpythonu; CREATE FUNCTION test=# select pyversion(); pyversion -- 2.7.3 (default, Apr 20 2012, 23:04:22) + [GCC 4.6.3] + /usr/lib/python2.7 + /usr/lib/python2.7/plat-linux2 + /usr/lib/python2.7/lib-tk + /usr/lib/python2.7/lib-old + /usr/lib/python2.7/lib-dynload + /usr/local/lib/python2.7/dist-packages + /usr/lib/python2.7/dist-packages+ /usr/lib/pymodules/python2.7+ /usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode (1 row) test=# CREATE OR REPLACE FUNCTION pgpython_test() test-# RETURNS text LANGUAGE python AS test-# $python$ test$# import sys test$# def main(): test$# return sys.version + '\n' + '\n'.join(sys.path) test$# $python$; CREATE FUNCTION test=# select pgpython_test(); pgpython_test - 3.2.3 (default, May 3 2012, 15:48:36) + [GCC 4.6.3]+ /usr/lib/python3.2 + /usr/lib/python3.2/plat-linux2 + /usr/lib/python3.2/lib-dynload + /usr/local/lib/python3.2/dist-packages + /usr/lib/python3/dist-packages (1 row) OK, I've got two versions of python: 2.7 and 3.2... Both python console and postgresql can see python 2.7 but plus it appears the grass environmental variables are not finding their way into the $pythonpath as seen by the postgresql server?? Any suggestions? ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] Problem accessing GRASS from PostgreSQL9.1/PostGIS2.0 via pg-python or plpythonu
My requirement is to connect to GRASS from PostgreSQL9.1/PostGIS2.0 via the
pg-python or plpythonu procedural language. I'm wondering whether I have a
permissions issue?
I'm running my stack on Ubuntu 12.04 LTS
I've added the following environmental variables to /etc/bash.bashrc
export GISBASE=/usr/local/grass-6.4.3svn
export PATH=$PATH:$GISBASE/bin:$GISBASE/scripts:$GISBASE/lib
export PYTHONPATH=${PYTHONPATH}:$GISBASE/etc/python/
export PYTHONPATH=${PYTHONPATH}:$GISBASE/etc/python/grass
export PYTHONPATH=${PYTHONPATH}:$GISBASE/etc/python/grass/script
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GISBASE/lib
export GIS_LOCK=$$
export GISRC=$HOME/.grassrc6
If I log out and log in again, the following environment variables are now
available to the postgres user:
ubuntu@ip-10-252-74-140:~$ sudo su postgres
postgres@ip-10-252-74-140:/home/ubuntu$ echo $GISBASE
/usr/local/grass-6.4.3svn
postgres@ip-10-252-74-140:/home/ubuntu$ echo $PYTHONPATH
:/usr/local/grass-6.4.3svn/etc/python/:/usr/local/grass-6.4.3svn/etc/python/grass:/usr/local/grass-6.4.3svn/etc/python/grass/script
postgres@ip-10-252-74-140:/home/ubuntu$ echo $HOME
/var/lib/postgresql
if I call the function import grass.script as grass from the python console,
there is no error. All seems ok.
postgres@ip-10-252-74-140:/$ python
Python 2.7.3 (default, Apr 20 2012, 22:39:59)
[GCC 4.6.3] on linux2
Type help, copyright, credits or license for more information.
import grass.script as grass
If I now log into PostgreSQL/PostGIS via psql, and try to run the same command
within in a postgresql plpythonu or pg-python function, I get the following
error messages.
psql -d test
CREATE OR REPLACE FUNCTION pythontest() RETURNS text AS
$$
import grass.script as grass
$$
LANGUAGE 'plpythonu' VOLATILE;
test=# select pythontest();
ERROR: ImportError: No module named grass.script
CONTEXT: Traceback (most recent call last):
PL/Python function pythontest, line 3, in module
import grass.script as grass
PL/Python function pythontest
test=#
I've also tried the pg-python language - in case it had something to do with
plpythonu:
CREATE OR REPLACE FUNCTION pgpython_test()
RETURNS text LANGUAGE python AS
$python$
import Postgres
import grass.script as grass
def main():
return (Hello World)
$python$;
test=# select pgpython_test();
ERROR: could not load Python function's module object
CONTEXT: [exception from Python]
Traceback (most recent call last):
File public.pgpython_test(), line 3, in module
ImportError: No module named grass.script
[public.pgpython_test() while loading function module]
test=#
I can however access GRASS from PostgreSQL9.1/PostGIS2.0 via pl/r and R. For
example, this function works...
CREATE OR REPLACE FUNCTION generate_viewshed() RETURNS text as
$$
library(spgrass6)
#initialise GRASS.
initGRASS(gisBase = “/usr/local/grass-6.4.3svn/”, home = tempdir(), gisDbase =
“/usr/local/share/grassdata/”, location = “aust_dem_1sec_wgs84″, mapset =
“postgres”, SG=”aust_dem_1sec”, override = TRUE)
#define the computational region for the analysis – limiting the region to an
area slightly greater than the max_distance from the observation point.
execGRASS(“g.region”, parameters = list(n=”-34.9759927736275″,
s=”-34.9985728860995″, e=”138.641896696527″, w=”138.619947232807″,
align=”aust_dem_1sec”))
#run the viewshed module – all of these parameters can be set dynamically and
parsed as plr function arguments – the output is a raster file.
execGRASS(“r.viewshed”, parameters = list(input = “aust_dem_1sec”, output =
“viewshed_raster”, coordinate = “138.630922,-34.987283″, obs_elev = 25,
max_dist = 1000), flags = c(“b”, “overwrite”))
#convert the raster output to a vector.
execGRASS(“r.to.vect”, parameters = list(input = “viewshed_raster”, output =
“viewshed_vector”, feature = “area”), flags = c(“overwrite”))
#write the vector data back to PostGIS.
execGRASS(“v.out.ogr”, parameters = list(input = “viewshed_vector”, dsn =
“PG:host=localhost dbname=test user=postgres password=password”, olayer =
“viewshed_vector”, format = “PostgreSQL”, type = “area”), flags = c(“c”))
$$
LANGUAGE ‘plr’;
In terms of GRASS permissions...
postgres@ip-10-252-74-140:/home/ubuntu$ vi $HOME/.grassrc6
GISDBASE: /usr/local/share/grassdata
LOCATION_NAME: aust_dem_1sec_wgs84
MAPSET: postgres
GRASS_GUI: text
~
This is as far as I've got and need some help to connect to GRASS from
PostgeSQL/PostGIS via python (as an alternative to pl/r).
Is there something obvious I have missed? Could someone please guide me
through to a solution.
With thanks and regards
Mark
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
