[GRASS-user] Ideas for a rush project (Thomas Adams): 400 parallel GRASS jobs on a cluster
Tom, you will be able to speed up your work (or compute more alternatives) if you can run your project on a computing cluster, maybe at a university or research institute ? Some info here: https://grasswiki.osgeo.org/wiki/Parallel_GRASS_jobs https://courses.neteler.org/building-a-cluster-for-grass-gis-and-other-software-from-the-osgeo-stack/ http://gfzpublic.gfz-potsdam.de/pubman/item/escidoc:100071:1/component/escidoc:100070/5_GISDAY-2012_loewe_thaler_State_of_the_Cluster_bib.pdf%3Bjsessionid=87D8757B6257885C605BFDC531F067A3 (-> slide 20 gives an overview over the time saved by going parallel). Good luck with your project, peter ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] Global overview of GRASS HPC resources/installations ?
Hi all, is there already some kind of up to date list / global map of GRASS installation in HPC-cluster environments ? This would be handy to get a feeling about how much "big geospatial number crunching" is currently done with GRASS, and where to visit / whom to talk to to learn from others. Best, Peter ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?
On 14/05/18 11:38, "Peter Löwe" wrote: Hi all, is there already some kind of up to date list / global map of GRASS installation in HPC-cluster environments ? I don't know of any. This would be handy to get a feeling about how much "big geospatial number crunching" is currently done with GRASS, and where to visit / whom to talk to to learn from others. +1 Moritz ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] r.viewshed speed improvment
Hello Grass users, I'm using r.viewshed to calculate the vertical height visible of object (windturbine) from any point on my DEM. So, I iterate upto the maximum height (from 0 to 250m) every meter. At the end, I made a r.series to sum the maps. As I have sometimes about 300 wtg on the territory, it takes a while (about two days) to calculate all the visibile height for each wtg on a 80kmx80km territory with 75m resolution (1066x1066). But it works. Today, I need to calculate one wtg with a topo raster about 22000 rows x 22000 colums (5m resolution on about 50km radius = near 500 milions cells). I wrote a new python script that parallelized jobs (inspired by https://grasswiki.osgeo.org/wiki/Parallelizing_Scripts). But I notice that the proc are running at 100% a small part of the time and the memory not so much used. On the other hand, the hdd is very busy. I'm using Grass 7.4 on Debian 9 with Intel i7 (8 cores) and 32Go RAM. And I have setup 4096MB per r.viewshed and directory=/tmp I wonder if it's not better to use only one core and put 32GB memory on the one calculation ? What are the best settings to improve calculation speed ?. Or is there any other faster approach to calculate the visibility ? (I had been thinking with r.horizon to calculate the minimum visible height. But unfortunatly the raster mode is not able to give horizon to a specific target coordinates, unless to do it for each cell independently...) See below a part of output (the script is still running) https://grasswiki.osgeo.org/wiki/Parallelizing_Scripts GRASS_INFO_END(13720,1) Estimated size active structure: (key=64, ptr=8, total node=96 B) Total= 2103840 B Start sweeping. Computing events... Nodata value set to -nan rows=21915, cols=21915, total = 480267225 In-memory memory usage is 48026897820 B (45802 MB), max mem allowed=4294967296 B(4096MB) * EXTERNAL_MEMORY MODE * Intermediate files will not be deletedin case of abnormal termination. Intermediate location: /media/hdd1/temp/ To save space delete these files manually! Estimated size active structure: (key=64, ptr=8, total node=96 B) Total= 2103840 B Start sweeping. Computing events... Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. Sorting events... Sorting events... Sorting events... Sorting events... Sorting events... Sorting events... Sorting events... Sorting events... Initialize sweepline... Determine visibility... Initialize sweepline... Determine visibility... Initialize sweepline... Determine visibility... Initialize sweepline... Determine visibility... Initialize sweepline... Determine visibility... Initialize sweepline... Determine visibility... Initialize sweepline... Determine visibility... Initialize sweepline... Determine visibility... the /tmp directory content is : 16 files, for 369 Go The maps are created by series of 8 (in 4 hours), but Grass needs between 24h upto 36h between two sets of 8 maps... Thank you for your help. Frank ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Ideas for a rush project (Thomas Adams): 400 parallel GRASS jobs on a cluster
Peter, A good idea, but not an option for me at this point. I hope to have access to such resources in the near future. Best, Tom On Mon, May 14, 2018 at 5:09 AM, "Peter Löwe" wrote: > Tom, > > you will be able to speed up your work (or compute more alternatives) if > you can run your project on a computing cluster, maybe at a university or > research institute ? > > Some info here: > https://grasswiki.osgeo.org/wiki/Parallel_GRASS_jobs > https://courses.neteler.org/building-a-cluster-for-grass- > gis-and-other-software-from-the-osgeo-stack/ > http://gfzpublic.gfz-potsdam.de/pubman/item/escidoc:100071: > 1/component/escidoc:100070/5_GISDAY-2012_loewe_thaler_ > State_of_the_Cluster_bib.pdf%3Bjsessionid=87D8757B6257885C605BFDC531F067A3 > (-> slide 20 gives an overview over the time saved by going parallel). > > Good luck with your project, > peter > > > > > > -- Thomas E Adams, III 1724 Sage Lane Blacksburg, VA 24060 tea...@gmail.com (personal) t...@terrapredictions.org (work) 1 (513) 739-9512 (cell) ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?
On Mon, May 14, 2018 at 11:55 AM, Moritz Lennert wrote: > On 14/05/18 11:38, "Peter Löwe" wrote: >> >> Hi all, >> >> is there already some kind of up to date list / global map of GRASS >> installation in HPC-cluster environments ? > > I don't know of any. Perhaps best collected in a Wiki page. >> This would be handy to get a feeling about how much "big geospatial number >> crunching" is currently done with GRASS, and where to visit / whom to talk >> to to learn from others. > > > +1 Yes, nice idea. Markus ___ grass-user mailing list grass-user@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-user
[GRASS-user] Fwd: [OSGeo-Announce] Impressions from OSGeo Code Sprint 2018 in Bonn
-- Forwarded message -- From: Astrid Emde (OSGeo) Date: Tue, May 15, 2018 at 7:33 AM Subject: [OSGeo-Announce] Impressions from OSGeo Code Sprint 2018 in Bonn To: Announce * see OSGeo Foundation News: https://www.osgeo.org/foundation-news/impressions-from-osgeo-code-sprint-2018-in-bonn/ The annual OSGeo Code sprint was held in Bonn March 18 to 25, bringing together nearly hundred members of the OSGeo community together. A code sprint is a chance for the community to meet in person for direct collaboration, projects use this as an opportunity to work together or tackle larger issues as a team. We made use of BaseCamp Bonn (http://www.basecamp-bonn.de/) to host the code sprint, providing a friendly indoor "campground' with picnic tables for working, surrounded by caravans of all shapes and sizes making up the accommodation. It can be quite a shock to roll out of bed, and find python developers in a cluster before your morning coffee. Participation was open to all, not only OSGeo projects, and we were pleased to welcome over eighty developers from all over the world! This was a chance for developers to meet (often for the fist time) and collaborate with core developers, gaining a insight into the software, and participate in open source development, An impressive list of projects was represented: uDig, GeoTools, LocationTech, JTS Topology Suite, JAI Replacement, TilelessMap, PostGIS, SHOGun, react-geo, Saga, OSGeoLive, owslib, debian/ubuntu, UbuntuGIS, OpenLayers, MapServer, QGIS, GeoExt 3, Nominatim, GDAL, t-rex, PyWPS, GRASS GIS, OWSLib, pygeometa, MapServer, geocatalogo, GeoNode, OTB, ZOO-Project, Proj.4, Point clouds, meshes, Potree, PDAL, GeoTrellis, snappy, OSM-Vector-Tiles, Geonetwork, Geoserver, OpenLayers Editor, Noise, CouchDB, MS4W, EOxServer, maposmatic, mapnik, Mapbender, gvSIG, gvSIG Mobile, Geopaparazzi, pyModis, GeoMajas, GeoHealthCheck, Stetl, Entwine, Greyhound, deegree, TEAM Engine, actinia, 3DCityDB and testing GMLAS tools! Entwine * EPT (https://github.com/connormanning/ept) implementation for Potree (http://potree.org). GeoHealthCheck (GHC) * Per-Resource Scheduling (https://github.com/geopython/GeoHealthCheck/tree/per_resource_scheduling) * Re-architecture to run in all-in-one daemon mode * Docker deployment improvements * Presented GHC at FOSSGIS(https://media.ccc.de/v/2018-5294-geohealthcheck) * OSGeo Community Project start Geopaparazzi * Added Cloud Profiles, allowing users to easily download complex data sets from the web Ticket #441 https://github.com/geopaparazzi/geopaparazzi/issues/441. * Added view-only of geographic feature attributes * Enhanced the viewing of files (images, PDFs, etc) linked to geographic features ticket #454 (https://github.com/geopaparazzi/geopaparazzi/issues/454). * Various bug fixes ticket #456 (https://github.com/geopaparazzi/geopaparazzi/issues/456), ticket #460 (https://github.com/geopaparazzi/geopaparazzi/issues/460), ticket #461 (https://github.com/geopaparazzi/geopaparazzi/issues/461), ticket #466 (https://github.com/geopaparazzi/geopaparazzi/issues/466). GRASS GIS * A lot of work, please look at the dedicated GRASS GIS wiki page (https://grasswiki.osgeo.org/wiki/Talk:GRASS_Community_Sprint_Bonn_2018) * Press release with photo collage: https://grass.osgeo.org/news/75/15/Recap-from-the-OSGeo-Code-Sprint-in-Bonn-2018/ TEAM Engine * Discussed test strategies for test suites and enhanced the existing tests for ETS WFS 2.0 Ticket #318 (https://github.com/opengeospatial/teamengine/issues/318). * Enhanced the transformation from the CTL log to the EARL report by mapping hierarchies. In addition some bugs was fixes in the transformation ticket #319 (https://github.com/opengeospatial/teamengine/issues/319). Mapbender * Working on the documentation and the translations as on the restructuring and small designs of the new theme (https://github.com/mapbender/mapbender/projects/5) * Working on the OpenLayers4/5 implementation (https://github.com/mapbender/mapbender/projects/3) * Working on some bugs that occured. PyWPS * Code sprint summary at the GitHUb Wiki (https://github.com/geopython/pywps/wiki/Code-Sprint-2018-Bonn) * PyWPS was proposed for OSGeo graduation (http://osgeo-org.1560.x6.nabble.com/recommend-PyWPS-for-graduation-td5359073.html) OSGeoLive * Tasks for incubation were done ** Provenance review ** Final touches of the new Documentation framework OpenLayers Mapbox styles generation library from OpenLayers (ol-mapbox-style) and upgrading the Boundless SDK (not an OSGeo project, but open source: https://github.com/boundlessgeo/sdk/) to OpenLayers 5. OWSLib * Merged Pull-Requests * Fixed test-suite PDAL * Refactored documentation and website (https://medium.com/@chambbj/pdal-at-the-osgeo-code-sprint-cf0a56eee59d) * Conda builds * Numpy reader support * 1.7
