Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
It is entirely unacceptable that data produced with European public funding is available in one random, proprietary format only. Maybe we should all email them individually and ask for an open standard format, so they see that there is some wider interest in this. Ben - Original Message - From: Felix Schalck felix.scha...@gmail.com To: Markus Neteler nete...@osgeo.org, benjamin ducke benjamin.du...@oxfordarch.co.uk Cc: grass-user grass-user@lists.osgeo.org Sent: Saturday, September 19, 2009 12:05:10 AM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna Subject: Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE Hi, I got news from the GDAL mailing list, where I posted a similar question about that strange format I downloaded on the CCM JRC web page. It seems to be the new ArcGIS file geodatabase format introduced by ESRI in ArcGIS 9.2, according to Jason Roberts and Frank Warmerdam. The important thing here is that it is a different database format (from .mdb or argis personal database), entirely proprietary which cannot, as of september 2009, be read by any OGR driver. So we took steps to contact the author/distributor of CCM data in order to have the set distributed in another format. More on this to follow. Regards, Felix -- Files attached to this email may be in ISO 26300 format (OASIS Open Document Format). If you have difficulty opening them, please visit http://iso26300.info for more information. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
I agree that this is unacceptable and contradicts/undermines the efforts of organizations and standards body who work on data-formats and web services that help to support vendor-independent standards, such as GML, WFS, Interlis, etc. Doesn't Inspire, and other european projects, mandate open formats and vendor independent access to data? Andreas Benjamin Ducke wrote: It is entirely unacceptable that data produced with European public funding is available in one random, proprietary format only. Maybe we should all email them individually and ask for an open standard format, so they see that there is some wider interest in this. Ben - Original Message - From: Felix Schalck felix.scha...@gmail.com To: Markus Neteler nete...@osgeo.org, benjamin ducke benjamin.du...@oxfordarch.co.uk Cc: grass-user grass-user@lists.osgeo.org Sent: Saturday, September 19, 2009 12:05:10 AM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna Subject: Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE Hi, I got news from the GDAL mailing list, where I posted a similar question about that strange format I downloaded on the CCM JRC web page. It seems to be the new ArcGIS file geodatabase format introduced by ESRI in ArcGIS 9.2, according to Jason Roberts and Frank Warmerdam. The important thing here is that it is a different database format (from .mdb or argis personal database), entirely proprietary which cannot, as of september 2009, be read by any OGR driver. So we took steps to contact the author/distributor of CCM data in order to have the set distributed in another format. More on this to follow. Regards, Felix -- Files attached to this email may be in ISO 26300 format (OASIS Open Document Format). If you have difficulty opening them, please visit http://iso26300.info for more information. ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
On Sat, 2009-09-19 at 12:38 +, Benjamin Ducke wrote: It is entirely unacceptable that data produced with European public funding is available in one random, proprietary format only. Maybe we should all email them individually and ask for an open standard format, so they see that there is some wider interest in this. Ben +1 Nikos - Original Message - From: Felix Schalck felix.scha...@gmail.com To: Markus Neteler nete...@osgeo.org, benjamin ducke benjamin.du...@oxfordarch.co.uk Cc: grass-user grass-user@lists.osgeo.org Sent: Saturday, September 19, 2009 12:05:10 AM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna Subject: Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE Hi, I got news from the GDAL mailing list, where I posted a similar question about that strange format I downloaded on the CCM JRC web page. It seems to be the new ArcGIS file geodatabase format introduced by ESRI in ArcGIS 9.2, according to Jason Roberts and Frank Warmerdam. The important thing here is that it is a different database format (from .mdb or argis personal database), entirely proprietary which cannot, as of september 2009, be read by any OGR driver. So we took steps to contact the author/distributor of CCM data in order to have the set distributed in another format. More on this to follow. Regards, Felix ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
Looks like we all agree on this. I'm sure there was a good reason behind adopting ESRI file geodatabase, like the 2Gb file limit of previously used ms access dbs, but adopting ONLY ESRI formats is definitely not very public-friendly. Now, I'm sure that each new short message left int Mr Vogt Message box (juergen.vogt(-at/arobase-)jrc.ec.europa.eu), contributes to increase our chances to get the data published in a another format. Hermann Pfeffer, from the eea (european environment agency) just wrote me that an existing eu law obliging all eu-institutions to respond to public inquiries within 2 weeks, further increasing our chances to get an answer. Thank you all for your interest, Felix 2009/9/19 Nikos Alexandris nikos.alexand...@felis.uni-freiburg.de: On Sat, 2009-09-19 at 12:38 +, Benjamin Ducke wrote: It is entirely unacceptable that data produced with European public funding is available in one random, proprietary format only. Maybe we should all email them individually and ask for an open standard format, so they see that there is some wider interest in this. Ben +1 Nikos - Original Message - From: Felix Schalck felix.scha...@gmail.com To: Markus Neteler nete...@osgeo.org, benjamin ducke benjamin.du...@oxfordarch.co.uk Cc: grass-user grass-user@lists.osgeo.org Sent: Saturday, September 19, 2009 12:05:10 AM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna Subject: Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE Hi, I got news from the GDAL mailing list, where I posted a similar question about that strange format I downloaded on the CCM JRC web page. It seems to be the new ArcGIS file geodatabase format introduced by ESRI in ArcGIS 9.2, according to Jason Roberts and Frank Warmerdam. The important thing here is that it is a different database format (from .mdb or argis personal database), entirely proprietary which cannot, as of september 2009, be read by any OGR driver. So we took steps to contact the author/distributor of CCM data in order to have the set distributed in another format. More on this to follow. Regards, Felix ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
Felix Schalck wrote: Looks like we all agree on this. I'm sure there was a good reason behind adopting ESRI file geodatabase, like the 2Gb file limit of previously used ms access dbs, but adopting ONLY ESRI formats is definitely not very public-friendly. Now, I'm sure that each new short message left int Mr Vogt Message box (juergen.vogt(-at/arobase-)jrc.ec.europa.eu), contributes to increase our chances to get the data published in a another format. Hermann Pfeffer, from the eea (european environment agency) just wrote me that an existing eu law obliging all eu-institutions to respond to public inquiries within 2 weeks, further increasing our chances to get an answer. Felix, I am convinced that the EEA will respond. FYI, you can have a look at a similar discussion concerning the CORINE land cover which took place some months ago [1][2][3][4][5]. As an extra-sidenote: I never got a reply from the official authority in Greece (although I've contacted 2-3 times by e-mail) who sells the (public domain) CORINE data at *high* prices. Best regards, Nikos --- [1] http://lists.osgeo.org/pipermail/geodata/2009-January/000801.html [2] http://lists.osgeo.org/pipermail/geodata/2009-January/000807.html [3] http://lists.osgeo.org/pipermail/geodata/2009-February/000811.html [4] http://lists.osgeo.org/pipermail/geodata/2009-February/000812.html [5] http://lists.osgeo.org/pipermail/geodata/2009-February/000813.html ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
Hi, I got news from the GDAL mailing list, where I posted a similar question about that strange format I downloaded on the CCM JRC web page. It seems to be the new ArcGIS file geodatabase format introduced by ESRI in ArcGIS 9.2, according to Jason Roberts and Frank Warmerdam. The important thing here is that it is a different database format (from .mdb or argis personal database), entirely proprietary which cannot, as of september 2009, be read by any OGR driver. So we took steps to contact the author/distributor of CCM data in order to have the set distributed in another format. More on this to follow. Regards, Felix ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
I frankly don't get it: the data I downloaded is definitely sort of a database, with each entry split among 4 differents files: entry.freelist,.gdbindexes,.gdbtable and .gdbtablx. ogrinfo deosn't seem to recognise any of the different files (but maybe I don't have the right driver compiled into (although PGEO is shown supported by ogrinfo -f). Anyway, I read somewhere about GRASS database support, and now I'm wondering if I don't have to import the whole fileset as a db first. Unfortunately, I don't have a clue on how to do this, so help would be greatly appreciated. Besides: I already followed your advice Markus, and wrote Mr. Vogt (whose mail is given on the CCM download page) and asked him if he would consider publishing the data in shapefile format (or anything else supported by GRASS). Thanks for your time and your patience, Felix 2009/9/14 Markus Neteler nete...@osgeo.org: On Mon, Sep 14, 2009 at 6:19 PM, Martin Landa landa.mar...@gmail.com wrote: 2009/9/14 Felix Schalck felix.scha...@gmail.com: [...] How do you import ArcGIS Geodatabase tiles into GRASS ? probably using v.in.ogr (PGeo driver [1]). Check available formats using `v.in.ogr -f`. If that fails I would also consider to ask the data provider to additionally publish the data in a documented data format. Markus [1] http://gdal.osgeo.org/ogr/drv_pgeo.html ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
Hi, 2009/9/14 Felix Schalck felix.scha...@gmail.com: [...] How do you import ArcGIS Geodatabase tiles into GRASS ? probably using v.in.ogr (PGeo driver [1]). Check available formats using `v.in.ogr -f`. Martin [1] http://gdal.osgeo.org/ogr/drv_pgeo.html -- Martin Landa landa.martin gmail.com * http://gama.fsv.cvut.cz/~landa ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
On Mon, Sep 14, 2009 at 6:19 PM, Martin Landa landa.mar...@gmail.com wrote: 2009/9/14 Felix Schalck felix.scha...@gmail.com: [...] How do you import ArcGIS Geodatabase tiles into GRASS ? probably using v.in.ogr (PGeo driver [1]). Check available formats using `v.in.ogr -f`. If that fails I would also consider to ask the data provider to additionally publish the data in a documented data format. Markus [1] http://gdal.osgeo.org/ogr/drv_pgeo.html ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
There is some very good hydrodata here: http://ccm.jrc.ec.europa.eu/php/index.php?action=viewid=24# Two drawbacks: - non-commercial use only (see license text) - comes as ArcGIS Geodatabase file (yuck!) Version 1 used to be shapefiles, maybe it's still available. Ben Felix Schalck wrote: Dear All-who-may-be-intersted, First, I'd like to apologize for the delay: lots of stuff prevented me from working on the map. I've got some spared time now, and the map isn't finished yet - so I'm back into buisiness! Secondly, I'd like to thank you Markus, for the great script you sent me: it worked withouth a hitch, and I have now a single big shapefile to work on. And sorry for the last message: it was a misclick. Then comes the map: basically, even though the last replies provided my with some valuable advices, I'm still stuck with bathymetry - rivers and coastlines. 1. For the bathymetry, I finally took ETOPO1 data, which gives me another nice raster, although of much lower resolution (1' compared to the 3 topographic raster from SRTM DEM). So the plan is to cut off all the land data from the ETOPO1 raster along the coastlines, to keep only the sea aeras and than paste this reduced raster onto the main SRTM topographic raster of greater resolution. Can this step be automated ? (eg: is there a tool to do the job ?) Of course, should raster-merge be to complicated, there is always the other option, which is to export pngs first, and paste the pngs together; but in both cases, the bathymetric raster has to be cut. 2. Rivers and coastlines are a real pain. a. I could import a big pasted SWBD shapefile thanks to Markus script, but the cleaning process of the import command takes days, and gets somehow stuck within the brigdge removing phase. Result is an uncomplete vector map, lacking centroids, which can't be further processed by v.generalize for smoothing (the command dies saying there is an error in input data), but can be displayed with v.display. Another problem are all the square borders left around former SWBD tiles in water aeras; can they be removed ? Or do I have to manually edit the (huge) vector map ? b. Then come the rivers: the (somehow corrupted) SWBD vector map shows only coastlines, smaller closed waterbodies and - although only partially - larger rivers. I somehow have to complete the river data, and try following methods: -r.watershed command in grass, to compute the rivers from DEM data. This command just dies during the memory allocation process (KILL signal) because the map is too huge for my system. I tried with differend -m parameters, but the command always askes for up to 10gb virtual memory the kernel won't be able to provide. I guess I would have to work on smaller pieces of the map, set with g.region, and paste the results together, but this is another time consuming option. -VMAP0 data import works, but the result is quite disappointing. The secondary rivers network seems quite good, but this datased is unable to fix the uncomplete major rivers from the SWBD dataset. For an instance, by showing both layers (SWBD + VMAP0) on the monitor, I get all smaller rivers flowing into the rhine, while the rhine itself remains divided into several un-joined segments. -OSM(openstreetmap) data is a bit confusing. First it is divided along countries, so you have to dowload and paste lots of different files. Than the Waterway shapefile is quite poor. And finally the natural shapefile comes with nice river data, but also a lot of confusing data, like forest aeras. This need some sort of filtering, but I don't know at all how to do this. c. Of course, if someone knows better river datasets (scale approching 3, complete, easy to use), I would be happy to try them. Voila. Even though a lot more problems arise during each step (resolution has become another one, for an instance), I hope this project will see an end soon. Thanks for your help and your patience, Felix 2009/8/17 Markus Neteler nete...@osgeo.org: Hi Felix, On Sun, Aug 16, 2009 at 7:37 PM, Felix Schalckfelix.scha...@gmail.com wrote: ... a - Importing the vectors from SWBD is no problem, tough It would be nice to have the 200+ NASA shapefiles merged BEFORE importing a new layer in GRASS. Is this possible with ogr2ogr ? Merge of two SHAPE files 'file1.shp' and 'file2.shp' into a new file 'file_merged.shp' is performed like this: # note order out, then in: ogr2ogr file_merged.shp file1.shp ogr2ogr -update -append file_merged.shp file2.shp -nln file_merged file2 The second command is opening file_merged.shp in update mode, and trying to find existing layers and append the features being copied. The -nln option sets the name of the layer to be copied to. Attached a script to do as many as you want. b - The big problem are coastlines and waterbodies (+main rivers): somehow I have to show them on the topographic map, which gdalwarp has filled out
Re: re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
Dear All-who-may-be-intersted, First, I'd like to apologize for the delay: lots of stuff prevented me from working on the map. I've got some spared time now, and the map isn't finished yet - so I'm back into buisiness! Secondly, I'd like to thank you Markus, for the great script you sent me: it worked withouth a hitch, and I have now a single big shapefile to work on. And sorry for the last message: it was a misclick. Then comes the map: basically, even though the last replies provided my with some valuable advices, I'm still stuck with bathymetry - rivers and coastlines. 1. For the bathymetry, I finally took ETOPO1 data, which gives me another nice raster, although of much lower resolution (1' compared to the 3 topographic raster from SRTM DEM). So the plan is to cut off all the land data from the ETOPO1 raster along the coastlines, to keep only the sea aeras and than paste this reduced raster onto the main SRTM topographic raster of greater resolution. Can this step be automated ? (eg: is there a tool to do the job ?) Of course, should raster-merge be to complicated, there is always the other option, which is to export pngs first, and paste the pngs together; but in both cases, the bathymetric raster has to be cut. 2. Rivers and coastlines are a real pain. a. I could import a big pasted SWBD shapefile thanks to Markus script, but the cleaning process of the import command takes days, and gets somehow stuck within the brigdge removing phase. Result is an uncomplete vector map, lacking centroids, which can't be further processed by v.generalize for smoothing (the command dies saying there is an error in input data), but can be displayed with v.display. Another problem are all the square borders left around former SWBD tiles in water aeras; can they be removed ? Or do I have to manually edit the (huge) vector map ? b. Then come the rivers: the (somehow corrupted) SWBD vector map shows only coastlines, smaller closed waterbodies and - although only partially - larger rivers. I somehow have to complete the river data, and try following methods: -r.watershed command in grass, to compute the rivers from DEM data. This command just dies during the memory allocation process (KILL signal) because the map is too huge for my system. I tried with differend -m parameters, but the command always askes for up to 10gb virtual memory the kernel won't be able to provide. I guess I would have to work on smaller pieces of the map, set with g.region, and paste the results together, but this is another time consuming option. -VMAP0 data import works, but the result is quite disappointing. The secondary rivers network seems quite good, but this datased is unable to fix the uncomplete major rivers from the SWBD dataset. For an instance, by showing both layers (SWBD + VMAP0) on the monitor, I get all smaller rivers flowing into the rhine, while the rhine itself remains divided into several un-joined segments. -OSM(openstreetmap) data is a bit confusing. First it is divided along countries, so you have to dowload and paste lots of different files. Than the Waterway shapefile is quite poor. And finally the natural shapefile comes with nice river data, but also a lot of confusing data, like forest aeras. This need some sort of filtering, but I don't know at all how to do this. c. Of course, if someone knows better river datasets (scale approching 3, complete, easy to use), I would be happy to try them. Voila. Even though a lot more problems arise during each step (resolution has become another one, for an instance), I hope this project will see an end soon. Thanks for your help and your patience, Felix 2009/8/17 Markus Neteler nete...@osgeo.org: Hi Felix, On Sun, Aug 16, 2009 at 7:37 PM, Felix Schalckfelix.scha...@gmail.com wrote: ... a - Importing the vectors from SWBD is no problem, tough It would be nice to have the 200+ NASA shapefiles merged BEFORE importing a new layer in GRASS. Is this possible with ogr2ogr ? Merge of two SHAPE files 'file1.shp' and 'file2.shp' into a new file 'file_merged.shp' is performed like this: # note order out, then in: ogr2ogr file_merged.shp file1.shp ogr2ogr -update -append file_merged.shp file2.shp -nln file_merged file2 The second command is opening file_merged.shp in update mode, and trying to find existing layers and append the features being copied. The -nln option sets the name of the layer to be copied to. Attached a script to do as many as you want. b - The big problem are coastlines and waterbodies (+main rivers): somehow I have to show them on the topographic map, which gdalwarp has filled out with -32768 values in nodata-waterzones. So either I cut waterbodies out of the topographic raster along the vectors, or I somehow have GRASS compute me all waterbodies from the vector layer, fill them with a nice blue and create a raster which can be pasted over the topographic raster to get the final map. It seems doable with mapcalc, but frankly, I do not
Re: re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
Dear Markus and other Grass users,
Here comes the latest update about my high resolution topographic map
of Europe - as usual - with some new questions.
Part1 (pasting cgiar-srtm tiles together) was a real pain, since I
needed to recompile GDAL with BigTIFF support in GTiFF driver to be
able to create such large geotiff files. Of course, once new gdal
successfully installed, I had to recompile GRASS against the new
gdal-bigtiff...
As posted on the gdal-dev mailing list, strange things occured during
my various gdalwarp attemps: while pasting all tiles bewteen
{34N;60N;-11W;35E} together in one command takes about 20h, dividing
the job in:
a - first merge slices of saying all tiles located in one long. column
b - finally paste all slices together
reduces the time needed to complete the command to around 4-5h. (I
posted 1 hour completion-time in the gdal list, but it was on a
smaller region)
Part2 has been updated to import the geotiff file in a lat/long
projection location native to srtm data, and than reproject the
obtained raster into my selected lcc (lambert conical conformal)
projection. This was probably the longest part, since it took r.proj
about 8h to complete this task. It tooks me actually twice as long,
since my region bounds were not properly set to include the distortion
of the lcc projection, so that the first re-projected raster had been
cut off just over Denmark; and I had to launch the reprojection
again...
Part3 is about creating the shadings, still ongoing, and works
surprinsingly well with r.shaded.relief, tough I'm still messing
around with zmult and azimut options to get the desired shadings.
Should somebody have advices here, to get nice shadings for large
srtm-based topographic maps, feel free to post them.
@Markus: I did not try the gdaldem tool to create the shadings, since
I re-projected the DEM in GRASS raster format.
Part4 is were I'm stuck currently, and is about creating coastline
vectors, bathymetry and exporting the results to be re-worked/merged
in GIMP.
a - Importing the vectors from SWBD is no problem, tough It would be
nice to have the 200+ NASA shapefiles merged BEFORE importing a new
layer in GRASS. Is this possible with ogr2ogr ?
b - The big problem are coastlines and waterbodies (+main rivers):
somehow I have to show them on the topographic map, which gdalwarp has
filled out with -32768 values in nodata-waterzones. So either I cut
waterbodies out of the topographic raster along the vectors, or I
somehow have GRASS compute me all waterbodies from the vector layer,
fill them with a nice blue and create a raster which can be pasted
over the topographic raster to get the final map. It seems doable
with mapcalc, but frankly, I do not know at all how to proceed. Any
help here would be greatly appreciated.
Part5 will be about secondary rivers, which will either be taken
directly from VMAP0 hydro layer or extracted with r.watershed (thank
you Markus for pointing this possibility) following the nice little
tutorial provided in the latest r.watershed manpage (bottom of the
page).
I will probably tryout both on a small scale, and keep the one
offering the best output (at my scale). The plan is to create another
vector layer showing only secondary rivers. More on this to follow.
Part6/final: rescaling (optional), exporting and merging in GIMP.
Notice that - at least until now, very little rework is required so
that I might consider finishing the job with GRASS and export one
finished map.
Voila. I sincerely hope this progress report is of some interest,
Thank you for your time and patience when it comes to answer my
numerous questions,
regards,
Felix
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
Very nice! b - The big problem are coastlines and waterbodies (+main rivers): somehow I have to show them on the topographic map, which gdalwarp has filled out with -32768 values in nodata-waterzones. So either I cut waterbodies out of the topographic raster along the vectors, or I somehow have GRASS compute me all waterbodies from the vector layer, When I understand right, its easy. In case of having all vectors as polygons, try v.to.rast. Then you have a raster layer of desired resolution (set with g.region). fill them with a nice blue and create a raster which can be pasted over the topographic raster to get the final map. It seems doable with mapcalc, but frankly, I do not know at all how to proceed. Any help here would be greatly appreciated. When you want to overlay the topographic-map with water-information, be aware what you want to do: when you just want the information, that there is water, you are done with telling the new raster-layer to be blue and transparent. If you want to change the topology for calculation with r.watershed (e.g. river-burning) or flatten lake-regions or set topography on coast-lines to zero, it would be a bit more to do. Cheers, Achim ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices: UPDATE
Hi Felix, On Sun, Aug 16, 2009 at 7:37 PM, Felix Schalckfelix.scha...@gmail.com wrote: ... a - Importing the vectors from SWBD is no problem, tough It would be nice to have the 200+ NASA shapefiles merged BEFORE importing a new layer in GRASS. Is this possible with ogr2ogr ? Merge of two SHAPE files 'file1.shp' and 'file2.shp' into a new file 'file_merged.shp' is performed like this: # note order out, then in: ogr2ogr file_merged.shp file1.shp ogr2ogr -update -append file_merged.shp file2.shp -nln file_merged file2 The second command is opening file_merged.shp in update mode, and trying to find existing layers and append the features being copied. The -nln option sets the name of the layer to be copied to. Attached a script to do as many as you want. b - The big problem are coastlines and waterbodies (+main rivers): somehow I have to show them on the topographic map, which gdalwarp has filled out with -32768 values in nodata-waterzones. So either I cut waterbodies out of the topographic raster along the vectors, or I somehow have GRASS compute me all waterbodies from the vector layer, fill them with a nice blue and create a raster which can be pasted over the topographic raster to get the final map. It seems doable with mapcalc, but frankly, I do not know at all how to proceed. Any help here would be greatly appreciated. Not sure, but set -32768 to NULL (no data) with r.null? Then use r.colors (nv to set color for NULL): http://grass.osgeo.org/grass64/manuals/html64_user/r.colors.html cheers Markus ogr_shape_merge.sh Description: Bourne shell script ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices
On Wed, Aug 12, 2009 at 4:06 PM, Felix Schalckfelix.scha...@gmail.com wrote: Dear Markus, Thank you s much for your response; it provided me with valuables hints - but also with new questions. Sure: things at that level aren't naturally obvious. First, following your avice, I finally decided to compile the new GRASS6.4rc5. Having an amd64 cpu, there war no binary available; so I had to do the job myself. Let's face it: manual configuration is a pain for a newcommer like me, You mean to find all dependences? We have collection some help in the Wiki: http://grass.osgeo.org/wiki/Compile_and_Install but once the ./configure script shows the long awaited final recap, the make command - although quite long - If you have a dual/multi-core CPU, you can use make -j2 make -j4 or so to be much faster (I always use make -j4). runs withouth a hitch. I reminds me of good ol' freebsd days, although we had a lot more troubles finishing compilation without errors: great job guys ! But the resulat was worth it; I don't know if it is the work you've done for the last few years, or the gcc optiomization flags (or perhaps both of it), but the resulting grass64 runs like a fireball compared to my old 6.2 bin provided by the Ubuntu repos. Again: great job ! Fantastic! Only now I fully understand what you meant by it is so far the only convincing software for GIS number crunching. But let's go back to the topic: my high-resolution topographic map of Europe. Thanks to your advices, the production outline has changed to this: 1. Merge the cgiar TIFs (AND NOT THE GRASS RASTERS IF I GOT THIS CORRECTLY) thanks to gdalwrap command. In what projection does this command work ? Is it possible to wrap the TIF directlly in my lcc projection ? I tried that yesterday and did NOT have luck. I would do it two-pass, even if it consumes twice as much space temporaneously: a) gdalwarp: all into one file, keeping the projection (mosaicking) b) gdalwarp: reproject to LCC (note that there are EU LCC and others). Use your preferred resampling method (gdalwarp offers several). Perhaps I got something wrong and you can do it in one step as well. 2. Add image pyramids with gdaladdo ( I frankly do not underdand this one at all; what does it mean ?) OK, it means that map copies at lower to much lower resolution are stored in the GeoTIFF (or different format which supports pyramids) file. When then opening with QGIS, Mapserver etc, it takes advantage of that and speeds up in an impressive way the visualization. Of course the file size increases. Example: I take world natural earth 250m which is huge; to cover Europe you need to download 4 tiles = 8 GB. I added pyramids with gdaladdo and now I am able to open these huge 4 tiles in no time in one step with QGIS. It's a convenience. 3. Shaded relief: I don't know your gdalhillshade command at all. Does it produce the same results as the r.shaded.relief I was planning to use ? Can you set the illumination angle with gdalhilshade ? Yes. I realise that it is called gdaldem now. http://trac.osgeo.org/gdal/browser/trunk/gdal/apps/gdaldem.cpp http://gdal.org/gdaldem.html 4. Re-gdaladdo for the shaded tif. yes. 5. Import in GRASS and checkout results. If I'm right, I will have two layers at this point, one with the relief colors, one with the shadings. Right (say, one is the relief [colors are optional], the other the shadings). You can use d.his or r.his to make a nice shaded colorized terrain map, something like this: http://grass.osgeo.org/grass60/screenshots/images/etopo2_grass_laea_9_48N_0E.jpg 6. Coastlines and Rivers. I was planning to use SWBD(coastlines and main rivers) and VLMAP0 Data (for the secondary rivers). Here again, you provide me with a complete new set of tools: r.external/r.terraflow/r.mapcalc/. What is the general idea behind ? r.external you would use in step 5. Instead of r.in.gdal. r.terraflow/r.mapcalc you may forget since I didn't understand that you would take the rivers as vector maps (I thought you wanted to extract them from the DEM). I checked the man pages, but I don't really understand how to use them for my purpose. My plan was to import the shapefile into the right projection with rvin.ogr, v.in.ogr (or v.external). and than export svg files for rework BEFORE joining the river layer and the topographic layers; but perhaps your way, once I understand it, is more efficient. Not sure (since I was confused :). 7. Export the shaded topography with r.out.png in one big png. Do I need two files (one containing the shadings, one with the relief colors) ? If you do the color shading in GRASS, then you need to export only one file. 8. Merge topography and hydrography layers in GIMP. Yes, sounds reasonable. Please consider to document your steps in the GRASS Wiki, it could be useful in future for others. Good luck Markus ___ grass-user mailing list
Re: re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices
2009/8/12 Markus Neteler nete...@osgeo.org: On Wed, Aug 12, 2009 at 4:06 PM, Felix Schalckfelix.scha...@gmail.com wrote: Dear Markus, Thanks to your advices, the production outline has changed to this: 1. Merge the cgiar TIFs (AND NOT THE GRASS RASTERS IF I GOT THIS CORRECTLY) thanks to gdalwrap command. In what projection does this command work ? Is it possible to wrap the TIF directlly in my lcc projection ? I tried that yesterday and did NOT have luck. I would do it two-pass, even if it consumes twice as much space temporaneously: a) gdalwarp: all into one file, keeping the projection (mosaicking) b) gdalwarp: reproject to LCC (note that there are EU LCC and others). Use your preferred resampling method (gdalwarp offers several). Perhaps I got something wrong and you can do it in one step as well. I immediately tried this, and ran into following problem: $gdalwarp srtm_*.tif europe_all_srtmV4_cgiar_default.tif Creating output file that is 6P x 36000L. ERROR 6: A 6 pixels x 36000 lines x 1 bands Int16 image would be larger than 4GB but this is the largest size a TIFF can be. Creation failed. If I understand this correctly, I don't have a choice here, and must reproject the whole thing while pasting it. So I computed this command: $gdalwarp -s_srs epsg:4326 -t_srs +proj=lcc +lat_1=47 +lat_2=41 +lat_0=53 +lon_0=12 +x_0=22.0 +y_0=15.0 +ellps=WSG84 +units=m +no_defs -tr 93 93 -r bilinear srtm_*.tif europe_all_srtmV4_cgiar_93m_LCC.tif It seems to work (at least a 3.6Gb TIF file is created in the right directory), but takes FOREVER (eg. the 2 first tiles took about 50min, and I have 56 tiles to be merged...). Strange thing is that neither the cpu nor the RAM is being used at full extend. The r.patch tool provided by GRASS went much faster. Any idea here to speed up things ? Would lowering the resolution (186m would be an option) help ? 2. Add image pyramids with gdaladdo ( I frankly do not underdand this one at all; what does it mean ?) OK, it means that map copies at lower to much lower resolution are stored in the GeoTIFF (or different format which supports pyramids) file. When then opening with QGIS, Mapserver etc, it takes advantage of that and speeds up in an impressive way the visualization. Of course the file size increases. Example: I take world natural earth 250m which is huge; to cover Europe you need to download 4 tiles = 8 GB. I added pyramids with gdaladdo and now I am able to open these huge 4 tiles in no time in one step with QGIS. It's a convenience. Thank you very much for the clear explanation. Indeed, this seems a very nice option. 3. Shaded relief: I don't know your gdalhillshade command at all. Does it produce the same results as the r.shaded.relief I was planning to use ? Can you set the illumination angle with gdalhilshade ? Yes. I realise that it is called gdaldem now. http://trac.osgeo.org/gdal/browser/trunk/gdal/apps/gdaldem.cpp http://gdal.org/gdaldem.html 4. Re-gdaladdo for the shaded tif. yes. 5. Import in GRASS and checkout results. If I'm right, I will have two layers at this point, one with the relief colors, one with the shadings. Right (say, one is the relief [colors are optional], the other the shadings). You can use d.his or r.his to make a nice shaded colorized terrain map, something like this: http://grass.osgeo.org/grass60/screenshots/images/etopo2_grass_laea_9_48N_0E.jpg Nice one: leads to a lot of graphical ideas concerning the final map... 6. Coastlines and Rivers. I was planning to use SWBD(coastlines and main rivers) and VLMAP0 Data (for the secondary rivers). Here again, you provide me with a complete new set of tools: r.external/r.terraflow/r.mapcalc/. What is the general idea behind ? r.external you would use in step 5. Instead of r.in.gdal. r.terraflow/r.mapcalc you may forget since I didn't understand that you would take the rivers as vector maps (I thought you wanted to extract them from the DEM). Didn't even think such things were possible, but now that you mention it, what would be your advice ? Using wmap0 set - with its errors - to get the rivers, or try to extract them from the DEM ? Which one would produce the best results (=at 93 or 186m resolution) ? I checked the man pages, but I don't really understand how to use them for my purpose. My plan was to import the shapefile into the right projection with rvin.ogr, v.in.ogr (or v.external). and than export svg files for rework BEFORE joining the river layer and the topographic layers; but perhaps your way, once I understand it, is more efficient. Not sure (since I was confused :). 7. Export the shaded topography with r.out.png in one big png. Do I need two files (one containing the shadings, one with the relief colors) ? If you do the color shading in GRASS, then you need to export only one file. 8. Merge topography and hydrography layers in GIMP. Yes, sounds reasonable. Please consider
Re: re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices
On Wed, Aug 12, 2009 at 6:21 PM, Felix Schalckfelix.scha...@gmail.com wrote: 2009/8/12 Markus Neteler nete...@osgeo.org: On Wed, Aug 12, 2009 at 4:06 PM, Felix Schalckfelix.scha...@gmail.com wrote: Dear Markus, Thanks to your advices, the production outline has changed to this: 1. Merge the cgiar TIFs (AND NOT THE GRASS RASTERS IF I GOT THIS CORRECTLY) thanks to gdalwrap command. In what projection does this command work ? Is it possible to wrap the TIF directlly in my lcc projection ? I tried that yesterday and did NOT have luck. I would do it two-pass, even if it consumes twice as much space temporaneously: a) gdalwarp: all into one file, keeping the projection (mosaicking) b) gdalwarp: reproject to LCC (note that there are EU LCC and others). Use your preferred resampling method (gdalwarp offers several). Perhaps I got something wrong and you can do it in one step as well. I immediately tried this, and ran into following problem: $gdalwarp srtm_*.tif europe_all_srtmV4_cgiar_default.tif Creating output file that is 6P x 36000L. ERROR 6: A 6 pixels x 36000 lines x 1 bands Int16 image would be larger than 4GB but this is the largest size a TIFF can be. Creation failed. If I understand this correctly, I don't have a choice here, We are lucky, you have a choice :) http://www.gdal.org/frmt_gtiff.html - BIGTIFF=YES/NO/IF_NEEDED/IF_SAFER: Control whether the created file is a BigTIFF or a classic TIFF. and must reproject the whole thing while pasting it. So I computed this command: $gdalwarp -s_srs epsg:4326 -t_srs +proj=lcc +lat_1=47 +lat_2=41 +lat_0=53 +lon_0=12 +x_0=22.0 +y_0=15.0 +ellps=WSG84 +units=m +no_defs -tr 93 93 -r bilinear srtm_*.tif europe_all_srtmV4_cgiar_93m_LCC.tif It seems to work (at least a 3.6Gb TIF file is created in the right directory), but takes FOREVER (eg. the 2 first tiles took about 50min, and I have 56 tiles to be merged...). Strange thing is that neither the cpu nor the RAM is being used at full extend. This is best asked on the gdal-dev list (I am currently running a mosaicking of 1700 DEM tiles, it runs for 10 days already...). The r.patch tool provided by GRASS went much faster. Cool! Any idea here to speed up things ? Would lowering the resolution (186m would be an option) help ? Perhaps but still all input data have to be read. Still best asked on the GDAL mailing list. Maybe someone else here knows more. ... 6. Coastlines and Rivers. I was planning to use SWBD(coastlines and main rivers) and VLMAP0 Data (for the secondary rivers). Here again, you provide me with a complete new set of tools: r.external/r.terraflow/r.mapcalc/. What is the general idea behind ? r.external you would use in step 5. Instead of r.in.gdal. r.terraflow/r.mapcalc you may forget since I didn't understand that you would take the rivers as vector maps (I thought you wanted to extract them from the DEM). Didn't even think such things were possible, but now that you mention it, what would be your advice ? Using wmap0 set - with its errors - to get the rivers, or try to extract them from the DEM ? Which one would produce the best results (=at 93 or 186m resolution) ? Good question. I am afraid that you need to experiment (but you can do that in a small map portion). Please keep us informed about your results. I recently used the rivers from OpenStreetMap for making a map, in my current study area they are quite complete. You can grab OSM as SHAPE file from http://download.geofabrik.de/osm/europe/ ... Please consider to document your steps in the GRASS Wiki, it could be useful in future for others. Do you mean writing a tutorial about creating my map ? Never though it would be able catch readers... But if you seriously think Its worth it, why not. It is moreover to document you experience - since it is a Wiki, more ideas may come in over the time. cheers Markus ___ grass-user mailing list grass-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/grass-user
Re: [GRASS-user] Very high resolution topographic map of Europe: need help and advices
Hi Felix, On Mon, Aug 10, 2009 at 8:28 PM, Felix Schalckfelix.scha...@gmail.com wrote: Hello, For some time now, I'm following sort of a personal objective to create the most precise (=high resolution) topographic map of Europe allowed by my comp (Xubuntu 9.04, AMD64, 3Gb RAM, 300+ HD space). Starting from the very scratch, I had to learn about DEMs and GIS/maptools - and I'm still not confortable with all the technicy behind. Fact is that the best data set available to serve my purpose seems to be the cgiar interpolated srtm3 geodata (no license problem here, since it is aimed for pure personal use) which has to somehow be translated into a big jpeg or png file. Here you can use gdalwarp to merge all files into a big one. Enjoy wildcards to do that in one line: gdalwarp srtm_*.tif cgiar_srtm3_LL.tif A big file is created. Then don't miss to add image pyramids: gdaladdo srtm3_LL.tif 2 4 8 16 32 64 and you can open a file of several GB in no time with QGIS for example. I posted some GDAL tricks here: http://gfoss.blogspot.com/2008/06/gdal-raster-data-tips-and-tricks.html (if you want OGR for vectors, check: http://gfoss.blogspot.com/2008/06/ogr-vector-data-tips-and-tricks.html ) ... The big problem happened to be the river data, since GSHHS provides only limited amount and precision of side-rivers, which resulted in chains of straight lines scattered all over a giant map, even after vectorization: it was a no-go. No idea, I didn't try GMT so much. And then, a few days ago, I discovered nasa SWBD data and WMAP0, which seem to be of much higher resolution, linked to a nice GRASS GIS tutorial on the french wikipedia. I immedialely dug into this new software, quite complicated I must admit, but very powerful. We hope you took the GRASS 6.4 version - even if still called release candidate it is used by many in production. Myself, I am doing heavy computations with it and it is so far the only convincing software for GIS number crunching :-) I figured out how to import GeoTiff data into GRASS Raster files, Hint: from GRASS 6.4 onwards you can use r.external to avoid import but to just link an external file into GRASS. Nice when registering 30GB of orthophotos in a few minutes... For newcomers QGIS is a nice interface to GRASS, too, since it comes with a GRASS toolbox. and how to display/export each one of them, but soon had to face new problems, especially when tried to reproject the data into LCC projection. So I decided to ask for help on this mailing list. My Current plan is to: 1. reproject the cgiar raster tiles OR one big merged raster into LCC projection (native srtm data seems to be a strange Mercator) They are in LatLong but they (those I downloaded) don't/didn't have the projection info in the metadata any more. Sad story. Maybe they fixed it later... Note that recent GDAL now provides the gdalhillshade tool for easy hillshading of GeoTIFF/other format files. Here my script (I did the same): -- snip -- #!/bin/sh # mosaic European CGIAR SRTM to 250m LAEA DEM map, bilinear gdalwarp -s_srs epsg:4326 -t_srs epsg:3035 -tr 250 250 -r bilinear \ srtm_*.tif europe_all_srtmV4_cgiar_250m_LAEA_EU.tif gdaladdo europe_all_srtmV4_cgiar_250m_LAEA_EU.tif 2 4 8 16 32 64 # generate shaded relief gdalhillshade europe_all_srtmV4_cgiar_250m_LAEA_EU.tif \ europe_all_srtmV4_cgiar_250m_LAEA_EU_shaded.tif gdaladdo europe_all_srtmV4_cgiar_250m_LAEA_EU_shaded.tif 2 4 8 16 32 64 -- snap -- 3. export a nice shaded topographic png I would still use GDAL here and gdal_translate to a PNG (do you really need this?) 4. extract rivers/coast vectors from SWBD files At this point use r.external on the huge SRTM DEM GeoTIFF. Then use r.terraflow/r.mapcalc/... (several options). 5. workout in Inkscape Oh, at this point you'll like r.out.png, you can set previously the resolution to something less (g.region) unless you don't plan to print a huge poster. 6. paste the whole thing together in GIMP. Why not ... The main problem right now seems to be the tiling of the topographic data. Each cgiar-cell (5°x5°) can be shown into a separate layer, but I'm unable to work them together. See above: gdalwarp is your friend. And it looks like most of the tools provided by GRASS assume the raster map is the size of the location (r.proj for an example). No, that's no the case. It's flexible. I tried r.patch but it produce wired results on top of giant files (I stopped when I hit the 2Gb limit), Ah, so you need to compile with Large File Support (LFS): ./configure ... --enable-largefile Then you can have huge files 2GB (if it fails somewhere please tell us but most common commands should be fine) ... I took the time to explain the whole thing with the hope to not only get help about my immediate raster division problem, but also about the big-picture, eg: is GRASS the right tools to do this My take is:
