Re: [gdal-dev] setting MAX_Z_ERROR on overviews compressed with LERC_DEFLATE
So I tried LERC_DEFLATE and MAX_Z_ERROR=0.5 and that worked amazingly! From 350MB DEFLATE PREDICTOR 3, to 45MB... There are free 32bit elevation data here if anyone's interested: https://land.copernicus.eu/imagery-in-situ/eu-dem/eu-dem-v1.1?tab=download. Rahkonen Jukka (MML) escreveu no dia quinta, 18/11/2021 à(s) 17:34: > Hi, > > > > I have no experience on 32 bit images but play with gdaladdo and > compression options and report what you find. External overviews (-ro) are > handy for testing because you can simply rename or delete the .ovr file and > make new run with other options. External overviews take some more space > than it takes to add internal overviews into TIFF but best option as > external should be best as internal too. > > > > -Jukka Rahkonen- > > > > *Lähettäjä:* Duarte Carreira > *Lähetetty:* torstai 18. marraskuuta 2021 18.30 > *Vastaanottaja:* Rahkonen Jukka (MML) > > *Kopio:* gdal-dev@lists.osgeo.org > *Aihe:* Re: [gdal-dev] setting MAX_Z_ERROR on overviews compressed with > LERC_DEFLATE > > > > Well, it didn't even occur to me to compress the overviews with > translate... > > I was expecting a bit more compressing from lerc... > > > > About tif 32bit, what compression do you think would yield better ratios? > > > > Thanks. > > > > Rahkonen Jukka (MML) escreveu no > dia quinta, 18/11/2021 à(s) 12:41: > > Hi, > > > > I made some tests with an 8 bit RGB image. > > First observation was that gdaladdo supports lerc_deflate (even it is not > documented), but it does not support “MAX_Z_ERROR”. This yields same sized > ovr file with or without max_z_error. > > gdaladdo -ro lerc_def2.tif --config compress_overview lerc_deflate > --config MAX_Z_ERROR 10 > > > > Another observation was that this option would really save some disk > space. I compressed the lossless ovr file with gdal_translate by using -co > max_z_error=10 and file sizes were: > > > > source tiff: 432 072 372 bytes > > lecr_deflate cog without overviews: 311 258 085 bytes > > > > lossless lerc_deflate overviews: 118 954 009 bytes > > lerc_deflare overviews with z_error=10: 52 400 583 bytes > > > > For comparison even not relevant to original question about 32 bit data > > jpeg-ycbcr overviews with default quality: 14 481 967 bytes > > > > -Jukka Rahkonen- > > > > *Lähettäjä:* gdal-dev *Puolesta *Duarte > Carreira > *Lähetetty:* torstai 18. marraskuuta 2021 13.45 > *Vastaanottaja:* gdal-dev@lists.osgeo.org > *Aihe:* [gdal-dev] setting MAX_Z_ERROR on overviews compressed with > LERC_DEFLATE > > > > Hi there. > > > > I am looking into compressing overviews for a DEM, with LERC_DEFLATE and > it works. But I'm trying to set the precision loss and get better > compression, since for overviews I don't really care that much. > > Ok, so the question is how to set the MAX_Z_ERROR for overviews compressed > with LERC_DEFLATE? > > > > More context: > > My reasoning is to compress losslessly the dem with deflate predictor 3, > and then compress "lossly" the overviews with lerc_deflate. > > Since this is 32bit dataset, I can't use jpeg on the overviews. > > > > Thanks in advance. > > Duarte > > > > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] setting MAX_Z_ERROR on overviews compressed with LERC_DEFLATE
Hi Even. I created a ticket: https://github.com/OSGeo/gdal/issues/4848 Thanks. >We would need a MAX_Z_ERROR_OVERVIEW config option for that. Please >file an enhancement ticket > >Even >Le 18/11/2021 à 12:44, Duarte Carreira a écrit : >>* Hi there. *>>>>* I am looking into compressing overviews for a DEM, with LERC_DEFLATE *>>* and it works. But I'm trying to set the precision loss and get better *>>* compression, since for overviews I don't really care that much. *>>* Ok, so the question is how to set the MAX_Z_ERROR for overviews *>>* compressed with LERC_DEFLATE? *>>>>* More context: *>>* My reasoning is to compress losslessly the dem with deflate predictor *>>* 3, and then compress "lossly" the overviews with lerc_deflate. *>>* Since this is 32bit dataset, I can't use jpeg on the overviews. *>>>>* Thanks in advance. *>>* Duarte* ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] setting MAX_Z_ERROR on overviews compressed with LERC_DEFLATE
Well, it didn't even occur to me to compress the overviews with translate... I was expecting a bit more compressing from lerc... About tif 32bit, what compression do you think would yield better ratios? Thanks. Rahkonen Jukka (MML) escreveu no dia quinta, 18/11/2021 à(s) 12:41: > Hi, > > > > I made some tests with an 8 bit RGB image. > > First observation was that gdaladdo supports lerc_deflate (even it is not > documented), but it does not support “MAX_Z_ERROR”. This yields same sized > ovr file with or without max_z_error. > > gdaladdo -ro lerc_def2.tif --config compress_overview lerc_deflate > --config MAX_Z_ERROR 10 > > > > Another observation was that this option would really save some disk > space. I compressed the lossless ovr file with gdal_translate by using -co > max_z_error=10 and file sizes were: > > > > source tiff: 432 072 372 bytes > > lecr_deflate cog without overviews: 311 258 085 bytes > > > > lossless lerc_deflate overviews: 118 954 009 bytes > > lerc_deflare overviews with z_error=10: 52 400 583 bytes > > > > For comparison even not relevant to original question about 32 bit data > > jpeg-ycbcr overviews with default quality: 14 481 967 bytes > > > > -Jukka Rahkonen- > > > > *Lähettäjä:* gdal-dev *Puolesta *Duarte > Carreira > *Lähetetty:* torstai 18. marraskuuta 2021 13.45 > *Vastaanottaja:* gdal-dev@lists.osgeo.org > *Aihe:* [gdal-dev] setting MAX_Z_ERROR on overviews compressed with > LERC_DEFLATE > > > > Hi there. > > > > I am looking into compressing overviews for a DEM, with LERC_DEFLATE and > it works. But I'm trying to set the precision loss and get better > compression, since for overviews I don't really care that much. > > Ok, so the question is how to set the MAX_Z_ERROR for overviews compressed > with LERC_DEFLATE? > > > > More context: > > My reasoning is to compress losslessly the dem with deflate predictor 3, > and then compress "lossly" the overviews with lerc_deflate. > > Since this is 32bit dataset, I can't use jpeg on the overviews. > > > > Thanks in advance. > > Duarte > > > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] setting MAX_Z_ERROR on overviews compressed with LERC_DEFLATE
Hi there. I am looking into compressing overviews for a DEM, with LERC_DEFLATE and it works. But I'm trying to set the precision loss and get better compression, since for overviews I don't really care that much. Ok, so the question is how to set the MAX_Z_ERROR for overviews compressed with LERC_DEFLATE? More context: My reasoning is to compress losslessly the dem with deflate predictor 3, and then compress "lossly" the overviews with lerc_deflate. Since this is 32bit dataset, I can't use jpeg on the overviews. Thanks in advance. Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] error in gdal.org certificate
Maybe this is known, but www.gdal.org is giving me a ERR_CERT_COMMON_NAME_INVALID error on chrome/win. Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] reading postgis xyzm with ogr2ogr
Hi. I am not that desperate ;) I was trying to figure out if the command ogr2ogr could use the hint in the ticket, by using the ewkt... Thanks, Duarte On Thu, Feb 12, 2015 at 9:49 AM, Rémi Cura remi.c...@gmail.com wrote: depending on how desperate you are you could : - use simple COPY statement to export you LINESTRINGZM from database - use python + python-ogr and manually create a X Y Z M file - use python + shapely In all case you can wrap it into a command line executable, liek ogr2ogr. Cheers, Rémi-C 2015-02-11 19:23 GMT+01:00 Duarte Carreira dncarre...@gmail.com: Thanks Paul. Yes, but I'm trying to export all tables from a schema, so I don't state the names of the tables. I think using sql would require 1 comand per table, right? So my command is something like this: ogr2ogr -progress PG:dbname='postgis' active_schema=schema1 schemas=schema1 connectioninfo -f SQLite mydb.sqlite I was hoping for a magic parameter to avoid the xyzm error... On Wed, Feb 11, 2015 at 5:55 PM, Paul Ramsey pram...@cleverelephant.ca wrote: You could use the OGR sql query option to wrap your geometry call in a force2d or force3d function call in PostGIS? -- Paul Ramsey http://cleverelephant.ca http://postgis.net On February 11, 2015 at 9:54:07 AM, Duarte Carreira ( dncarre...@gmail.com) wrote: Hi there. I’m trying to figure out how to read xyzm geometries from postgis with ogr2ogr, if at all possible. I get the usual is not supported error: ERROR 1: Reading EWKB with 4-dimensional coordinates (XYZM) is not supported But reading old tickets I get the idea there is a way to read and just ignore the 4th dimension, instead of getting null geometries. For instance: http://trac.osgeo.org/gdal/ticket/1323 From 8 years ago(!), says: The problem does not occur if a layer is accessed using OGRPGDataSource::GetLayerByname() Then, geometry is read in EWKT form and parsed correctly without any errors but M coordinate is omitted. Can this be done using og2ogr? Thanks, Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] reading postgis xyzm with ogr2ogr
Hi there. I’m trying to figure out how to read xyzm geometries from postgis with ogr2ogr, if at all possible. I get the usual is not supported error: ERROR 1: Reading EWKB with 4-dimensional coordinates (XYZM) is not supported But reading old tickets I get the idea there is a way to read and just ignore the 4th dimension, instead of getting null geometries. For instance: http://trac.osgeo.org/gdal/ticket/1323 From 8 years ago(!), says: The problem does not occur if a layer is accessed using OGRPGDataSource::GetLayerByname() Then, geometry is read in EWKT form and parsed correctly without any errors but M coordinate is omitted. Can this be done using og2ogr? Thanks, Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] reading postgis xyzm with ogr2ogr
Thanks Paul. Yes, but I'm trying to export all tables from a schema, so I don't state the names of the tables. I think using sql would require 1 comand per table, right? So my command is something like this: ogr2ogr -progress PG:dbname='postgis' active_schema=schema1 schemas=schema1 connectioninfo -f SQLite mydb.sqlite I was hoping for a magic parameter to avoid the xyzm error... On Wed, Feb 11, 2015 at 5:55 PM, Paul Ramsey pram...@cleverelephant.ca wrote: You could use the OGR sql query option to wrap your geometry call in a force2d or force3d function call in PostGIS? -- Paul Ramsey http://cleverelephant.ca http://postgis.net On February 11, 2015 at 9:54:07 AM, Duarte Carreira (dncarre...@gmail.com) wrote: Hi there. I’m trying to figure out how to read xyzm geometries from postgis with ogr2ogr, if at all possible. I get the usual is not supported error: ERROR 1: Reading EWKB with 4-dimensional coordinates (XYZM) is not supported But reading old tickets I get the idea there is a way to read and just ignore the 4th dimension, instead of getting null geometries. For instance: http://trac.osgeo.org/gdal/ticket/1323 From 8 years ago(!), says: The problem does not occur if a layer is accessed using OGRPGDataSource::GetLayerByname() Then, geometry is read in EWKT form and parsed correctly without any errors but M coordinate is omitted. Can this be done using og2ogr? Thanks, Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] how to create just the msk file from a rgba vrt file
Hi Even, and many thanks! Your pointers have solved my question. As for the problem building overviews it seems that it occurs when you enter a OVERVIEW tag pointing to a regular image, that doesn't have the INTERNAL_MASK_FLAGS_# set. On the other hand these flags allowed me to create a mask file even faster than using gdal_translate. The process is: 1) get a shapefile that represents your mask (maybe use gdaltindex and dissolve the result) 2) use gdalinfo to get extent and cell size of your vrt mosaic 3) use gdal_rasterize to get your mask raster, burning the shapefile to a new image initiated with 0 and burnt with 255 4) set the metadata tags to make this raster a mask (I used a small python script for this) 5) rename your raster mask to match your vrt mosaic (eg mosaic.vrt.msk) That's it. Instant (almost) mask file: instead of 4h41min it took 10min to create my mask. (The vrt mosaic references 487 rgb tiff files, totaling 240001x25 pixels in size.) The command to build the mask was the following: gdal_rasterize --config GDAL_CACHEMAX 512 -of GTiff -co compress=deflate -co tiled=yes -l quads_diss_etrs -te -79997.180 -22.390 40002.909 -104999.818 -tr 0.44258518990 0.44258518990 -ot Byte -burn 255 -init 0 -co INTERLEAVE=BAND -co NBITS=1 quads_diss_etrs.shp maskfile.tif I'm not sure about the NBITS command effect though... Then to insert the mask flags I used this python script: # import gdal from gdalconst import * import sys gdal.UseExceptions() try: filename = sys.argv[1] dataset = gdal.Open(filename, GA_Update) dataset.SetMetadataItem( INTERNAL_MASK_FLAGS_1, 2, None ) dataset.SetMetadataItem( INTERNAL_MASK_FLAGS_2, 2, None ) dataset.SetMetadataItem( INTERNAL_MASK_FLAGS_3, 2, None ) except Exception, e: print Error: + str(e) dataset = None ## Thanks again. Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: domingo, 9 de Fevereiro de 2014 09:56 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira; Brian Case Assunto: Re: [gdal-dev] how to create just the msk file from a rgba vrt file Le vendredi 07 février 2014 10:55:06, Duarte Carreira a écrit : Thanks Brian. But this way you rewrite the whole image to disk. It uses lots of disk space and takes forever. I want to avoid that and just get the mask out to a msk file as fast as possible. I don't want to convert my rgba vrt mosaic. The final objective is to get a msk file I can use with the simple rgb vrt mosaic. Then I'll be able to build overviews with jpeg/ycbcr compression which I can't do with rgba vrt because of the 4 bands. For now I have tried 3 ways: 1) use gdal_rasterize to create a mask directly from the mask polygon shapefile. Then just edit the rgb vrt mosaic and add a maskband to it. The problem here is gdaladdo does not honor the maskband. This is the fastest way I know, pitty it doesn't work in the end. I'd be curious that you provide ways of reproducing this. The overview computation code has explicit code to deal with mask bands. 2) use gdal_translate like you suggested but use -scale to write all 0s in all 3 bands, and compress with deflate. You get a valid mask and a very, very small useless mosaic. This works but takes a while still. 3) use gdal_translate like you suggested but exaggerate the jpeg compression so it errors out (jpeg_quality=15). You get an invalid 1kb mosaic and an apparently good msk. But it's corrupted in some way. So doesn't work. I think #1 has potential. If there was a way to somehow turn the tif created by gdal_rasterize into a true mask file and have it honored by gdaladdo we would have a winner. Maybe there's a way to directly export the alpha band from the rgba vrt mosaic to a mks file without writing anything else? That I guess would be the fastest way of all. You can generate a valid .msk file with the following gdal_translate command : gdal_translate -b 4 rgba.tif out.tif.msk \ -mo INTERNAL_MASK_FLAGS_1=2 -mo INTERNAL_MASK_FLAGS_2=2 \ -mo INTERNAL_MASK_FLAGS_3=2 -co COMPRESS=DEFLATE -co INTERLEAVE=BAND \ -co NBITS=1 You need to provide as many -mo INTERNAL_MASK_FLAGS_X=2 option as there are bands (so 3 for a RGB dataset as in the above example). This is the important option that will make a .msk file being recognized as a mask band. Even -- Geospatial professional services http://even.rouault.free.fr/services.html ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] how to create just the msk file from a rgba vrt file
Thanks Brian. But this way you rewrite the whole image to disk. It uses lots of disk space and takes forever. I want to avoid that and just get the mask out to a msk file as fast as possible. I don't want to convert my rgba vrt mosaic. The final objective is to get a msk file I can use with the simple rgb vrt mosaic. Then I'll be able to build overviews with jpeg/ycbcr compression which I can't do with rgba vrt because of the 4 bands. For now I have tried 3 ways: 1) use gdal_rasterize to create a mask directly from the mask polygon shapefile. Then just edit the rgb vrt mosaic and add a maskband to it. The problem here is gdaladdo does not honor the maskband. This is the fastest way I know, pitty it doesn't work in the end. 2) use gdal_translate like you suggested but use -scale to write all 0s in all 3 bands, and compress with deflate. You get a valid mask and a very, very small useless mosaic. This works but takes a while still. 3) use gdal_translate like you suggested but exaggerate the jpeg compression so it errors out (jpeg_quality=15). You get an invalid 1kb mosaic and an apparently good msk. But it's corrupted in some way. So doesn't work. I think #1 has potential. If there was a way to somehow turn the tif created by gdal_rasterize into a true mask file and have it honored by gdaladdo we would have a winner. Maybe there's a way to directly export the alpha band from the rgba vrt mosaic to a mks file without writing anything else? That I guess would be the fastest way of all. Duarte -Mensagem original- De: Brian Case [mailto:r...@winkey.org] Enviada: quinta-feira, 6 de Fevereiro de 2014 21:51 Para: Duarte Carreira Cc: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] how to create just the msk file from a rgba vrt file Duarte gdaltranslate -of outfrmt -b 1 -b 2 -b 3 -mask 4 infile outfile brian you can output to vrt's just dont try to mosaic masked or aphad files with vrts brian On Thu, 2014-02-06 at 13:20 +, Duarte Carreira wrote: Hi there. Continuing with trying to cut processing times, I am trying to get a msk file from a vrt mosaic, without converting the entire vrt to tif just to get the msk file. In other words, how can we create a msk file from the alpha band without rewriting the whole image? Thanks again. Duarte Duarte Carreira Diretor | Dep. Informação Geográfica e Cartografia www.edia.pt www.alqueva.com.pt Tel. +351 284315100 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] how to create just the msk file from a rgba vrt file
Hi there. Continuing with trying to cut processing times, I am trying to get a msk file from a vrt mosaic, without converting the entire vrt to tif just to get the msk file. In other words, how can we create a msk file from the alpha band without rewriting the whole image? Thanks again. Duarte Duarte Carreira Diretor | Dep. Informa??o Geogr?fica e Cartografia www.edia.pthttp://www.edia.pt www.alqueva.com.pthttp://www.alqueva.com.pt Tel. +351 284315100 [http://www.edia.pt/edia/images/edia_logo2.gif]http://www.edia.pt ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] suggestion for faster overviews building
Hi there. I am again in the course of creating overviews using gdaladdo and can't help feeling this could be faster... Gdalwarp is much faster when resampling an image. You can use -multi and tune memory usage. So why can't gdaladdo? Ok, lets just assume this is a fact of life we have to deal with... How about using gdalwarp to create our resampled images one at a time (which we can do several in parallel) and using them as overviews? Can this be done? Is there a way to merge de resampled images into an .ovr? Or maybe reference the resampled images in a vrt and have them used as overviews? Appreciate any ideas on this. Thanks. Duarte Duarte Carreira Diretor | Dep. Informa??o Geogr?fica e Cartografia www.edia.pthttp://www.edia.pt www.alqueva.com.pthttp://www.alqueva.com.pt Tel. +351 284315100 [http://www.edia.pt/edia/images/edia_logo2.gif]http://www.edia.pt ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] suggestion for faster overviews building
Hi Jukka. Now that you mentioned it I recall seeing that discussion online. I'll give it a try. It would be more elegant if vrt format supported several overview levels. Thanks. Duarte -Mensagem original- De: Jukka Rahkonen [mailto:jukka.rahko...@mmmtike.fi] Enviada: terça-feira, 4 de Fevereiro de 2014 15:19 Para: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] suggestion for faster overviews building Duarte Carreira DCarreira at edia.pt writes: Is there a way to merge de resampled images into an .ovr? Or maybe reference the resampled images in a vrt and have them used as overviews? Hi, Even Rouault helped me when I was experimenting with something like this. You can catch the discussion from here http://permalink.gmane.org/gmane.comp.gis.gdal.devel/31829 The zip file is also still available http://latuviitta.org/documents/GDAL_Scale_dependent_VRT_test.zip This is more fun than doing by hand what gdaladdo normally does because the overviews are not subsampled but they come from totally different data. -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Question about gdaladdo and raster mosaicking
Try without internal mask. Duarte _ De: Ammar [mailto:ammar8...@yahoo.com] Enviada: quinta-feira, 23 de Janeiro de 2014 14:28 Para: Etienne Tourigny Cc: gdal dev Assunto: Re: [gdal-dev] Question about gdaladdo and raster mosaicking Etienne, I tried the following: gdalbuildvrt -srcnodata 255 -vrtnodata 255 -addalpha -input_file_list tiff_list.txt mosaic02.vrt gdal_translate -of GTiff -b 1 -b 2 -b 3 -mask 4 -a_srs EPSG:3011 -co TILED=yes -co compress=jpeg -co jpeg_quality=80 -co blockxsize=512 -co blockysize=512 -co photometric=ycbcr --config gdal_tiff_internal_mask yes mosaic02.vrt mosaic02.tif gdaladdo mosaic02.tif -r average --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 60 --config INTERLEAVE_OVERVIEW PIXEL --config PHOTOMETRIC_OVERVIEW YCBCR 2 4 8 16 32 64 128 256 512 Did I implement the mask the right way? The results I got were similar to the JPEG compression with no mask, in other words I got white lines too! From: Etienne Tourigny etourigny@gmail.commailto:etourigny@gmail.com To: Ammar ammar8...@yahoo.commailto:ammar8...@yahoo.com Cc: Chaitanya kumar CH chaitanya...@gmail.commailto:chaitanya...@gmail.com; gdal dev gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org Sent: Thursday, January 23, 2014 12:25 PM Subject: Re: [gdal-dev] Question about gdaladdo and raster mosaicking As was suggested, you could try using a mask instead. On Thu, Jan 23, 2014 at 7:52 AM, Ammar ammar8...@yahoo.commailto:ammar8...@yahoo.com wrote: Chaitanya, Thank you very much for your reply and suggestion. I have used LZW with predictor 2 compression and added external overviews compressed in LZW too and the image came perfect. The only downside is the size of each of the images has expanded from approx. 2 GB to 18-20 GB. So now I have to choose between quality and size! Attached are two screen shots showing the difference. Thanks again! Regards, Ammar From: Chaitanya kumar CH chaitanya...@gmail.commailto:chaitanya...@gmail.com To: ammar8...@yahoo.commailto:ammar8...@yahoo.com Cc: gdal dev gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org Sent: Tuesday, January 21, 2014 6:27 PM Subject: Re: Question about gdaladdo and raster mosaicking Ammar, You are using lossy JPEG compression. I'm guessing that the white lines are are actually pixels with values near but not equal to 255. So, they are not treated as nodata pixels. My suggestion is to use another format with lossless compression for the 21 images or use a mask band if you prefer. Let me know how it goes. -- Best regards, Chaitanya Kumar CH On 21-Jan-2014 5:31 pm, ammar8...@yahoo.commailto:ammar8...@yahoo.com wrote: Hello Chaitanya, I was looking online for solutions to my problem and I cam across some of your posts so I thought you might help me with my problem. I have 10,000+ TIFF files that I want to merge into one big Image and serve using GeoServer. I merged every 500 images at a time creating using gdalbuildvrt and gdal_translate along with JPEG compression creating 21 TIFF files. I added overviews then to each of the new 21 files. When adding the images together, once directly as files using QGIS and another time as WMSs using GeoServer, I got white lines/gaps precisely at the border of ever 2 images. I thought the problem is with no data values but I decided to take another approach and merged all the 10,000 files into one huge image and this time I got one perfect image with no lines or gaps. I failed to create overviews for the big image because the process took days or gdaladdo running with no progress. I tested then on smaller images with and without overviews and the ones with no overview came perfect while the one with gdaladdo overviews came with the same problem of lines and gaps. The commands I m using are the following: gdalbuildvrt -srcnodata 255 -vrtnodata 255 -a_srs EPSG:27700 -input_file_list tiff_list.txt mosaic.vrt gdal_translate -of GTiff -co TILED=YES -co BIGTIFF=YES -co COMPRESS=JPEG -co JPEG_QUALITY=80 -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co PHOTOMETRIC=YCBCR mosaic.vrt mosaic.tif gdaladdo mosaic.tif -r average --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 60 --config INTERLEAVE_OVERVIEW PIXEL --config PHOTOMETRIC_OVERVIEW YCBCR 2 4 8 16 32 64 128 256 512 Any ideas, tips or recommendations would be appreciated! Best regards, Ammar _ Sent from http://osgeo-org.1560.x6.nabble.comhttp://osgeo-org.1560.x6.nabble.com/ ___ gdal-dev mailing list gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] create mask file only
Is there any way to create a mask file from an rgba without creating anything else, just the .msk file? Thanks, Duarte Duarte Carreira Diretor | Dep. Informa??o Geogr?fica e Cartografia www.edia.pthttp://www.edia.pt www.alqueva.com.pthttp://www.alqueva.com.pt Tel. +351 284315100 [http://www.edia.pt/edia/images/edia_logo2.gif]http://www.edia.pt ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gtiff with internal mask
To solve the overviews I created an external mask file instead, in step 3, by removing the option --config GDAL_TIFF_INTERNAL_MASK YES from the command. As I am using GDAL 1.9.2 I don't know if this issue is caused by the solved issue in 1.10 (https://trac.osgeo.org/gdal/wiki/Release/1.10.0-News): When building overviews, if the image has already an internal mask, then build internal overviews for the mask implicitely I ended up with this set of files: Mosaic.tif Mosaic.tif.msk Mosaic.tif.msk.ovr Mosaic.tif.ovr And to use in qgis, corresponding to step 4: Mosaic_rgba_qgis.vrt Duarte -Mensagem original- De: Duarte Carreira Enviada: quinta-feira, 18 de Julho de 2013 13:01 Para: 'Even Rouault' Cc: 'gdal-dev@lists.osgeo.org' Assunto: RE: [gdal-dev] gtiff with internal mask So... now I 've got issues building the overviews. Gdaladdo loses the transparency so I get to see black areas where I was supposed to see nothing. When I zoom in black disappears. Any known way to overcome this? Thanks, Duarte -Mensagem original- De: Duarte Carreira Enviada: quarta-feira, 17 de Julho de 2013 19:41 Para: 'Even Rouault' Cc: gdal-dev@lists.osgeo.org Assunto: RE: [gdal-dev] gtiff with internal mask Well, you're right... I messed up recreating the footsteps... So, cleaning up: 1) create a rgba vrt using gdalwarp to cut the original mosaic with a shapefile gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 312 -co alpha=yes -dstalpha -cutline shapes\index_diss.shp -of vrt originalmosaic.vrt test_rgba_uncomp.vrt 2) create a masked uncompressed vrt using gdal_translate (because I couldn't compress right away, with errors of missing StripOffsets field) gdal_translate -b 1 -b 2 -b 3 -mask 4 -co alpha=no -co photometric=rgb -co interleave=pixel -co tiled=yes --config GDAL_TIFF_INTERNAL_MASK YES -of vrt test_rgba_uncomp.vrt testmask_uncomp.vrt 3) create final masked compressed tiff using gdal_translate gdal_translate -co alpha=no -co photometric=ycbcr -co interleave=pixel -co tiled=yes -co compress=jpeg --config GDAL_TIFF_INTERNAL_MASK YES testmask_uncomp.vrt test_finalmask.tif The power of VRT amazes me! Also regarding sizes: I still got the same ratio of 8x smaller. To use this in QGIS you still have to create a rgba vrt: 4) gdal_translate -of VRT -b 1 -b 2 -b 3 -b mask test_finalmask.tif test_finalmask_qgis.vrt (couldn't get the 4ª band to be recognized as an alpha band though) So I think now it's correct... I am going to apply this to a real-case mosaic and report back... Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: quarta-feira, 17 de Julho de 2013 13:45 Para: Duarte Carreira Cc: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] gtiff with internal mask Le mercredi 17 juillet 2013 14:23:29, Duarte Carreira a écrit : Hi Even. Thanks so much for your tip! It works. I did have to specify I did not want an alpha band when cutting with the shapefile: gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 256 -co photometric=ycbcr -co compress=jpeg -co alpha=no --config GDAL_TIFF_INTERNAL_MASK YES -cutline shapes\index_diss.shp vrtini.vrt testmask.tif Hum, what I don't understand is how the above will produce a mask band in the output file... Does gdalinfo on testmask.tif show something like : Band 1 Block= Type=Byte, ColorInterp=Red Mask Flags: PER_DATASET Band 2 Block= Type=Byte, ColorInterp=Green Mask Flags: PER_DATASET Band 3 Block=... Type=Byte, ColorInterp=Blue Mask Flags: PER_DATASET I think you would need to do the above in 2 steps: 1) gdalwarp to an uncompressed RGBA TIFF 2) gdal_translate to turn the RGBA into a YCbCr JPEG-compressed GTiff + mask band Seems arcgis recognizes the masked tiff, but is much much slower displaying than using a normal alpha band. Perhaps adding -co TILED=YES will help ? (random guess) QGIS does need the vrt trick but then is as fast as usual. Also, the size is must smaller when using an internal mask: from a 219MB rgba,band interleaved,jpegd image to a 27MB ycbcr,pixel interleaved,jpegd masked tiff (8x smaller). Unless I missed something... Not completely surprising although more important than I would have expected. YCbCr compression usually gives a 2x to 3x compression bonus, and due to the usual nature of mask bands, the 1bit deflate compression of the mask band will give better results (both visually and in size) that the 8bit JPEG compression of the alpha band. Thanks again! Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: terça-feira, 16 de Julho de 2013 18:58 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira Assunto: Re: [gdal-dev] gtiff with internal mask Le mardi 16 juillet 2013 12:29:37, Duarte Carreira a écrit : I'm trying to create gtiffs with transparent areas where there are voids between my original images. I have successfully done it by using an alpha
Re: [gdal-dev] gtiff with internal mask
Please ignore previous email... should be a note to self... -Mensagem original- De: Duarte Carreira Enviada: sexta-feira, 19 de Julho de 2013 12:27 Para: 'Even Rouault' Cc: 'gdal-dev@lists.osgeo.org' Assunto: RE: [gdal-dev] gtiff with internal mask Ok, para conseguir construir as pirâmides temos de ter a máscara separada e não incluída no próprio tiff. Para isso basta alterar o 3º passo que cria efectivamente o tiff a partir do vrt, removendo a opção de criar a máscara interna, ficando assim: gdal_translate -co alpha=no -co photometric=ycbcr -co interleave=pixel -co tiled=yes -co compress=jpeg testmask_uncomp.vrt test_finalmask.tif Isto basta para que seja criado um tif rgb e um ficheiro .msk separados. *** Para criar um rgb+iv num só tiff, com máscara! *** Alterar o mosaico vrt criado no 2º passo, que já inclui a máscara, e criar uma banda Gray com origem no mosaico vrt dos infravermelhos. Depois é só converter mas usando as opções: -co alpha=no -co photometric=rgb e talvez -co interleave=band (para não dar erro). Não podemos usar ycbcr porque só dá para 3 bandas e nós temos 4. DC -Mensagem original- De: Duarte Carreira Enviada: quinta-feira, 18 de Julho de 2013 13:01 Para: 'Even Rouault' Cc: 'gdal-dev@lists.osgeo.org' Assunto: RE: [gdal-dev] gtiff with internal mask So... now I 've got issues building the overviews. Gdaladdo loses the transparency so I get to see black areas where I was supposed to see nothing. When I zoom in black disappears. Any known way to overcome this? Thanks, Duarte -Mensagem original- De: Duarte Carreira Enviada: quarta-feira, 17 de Julho de 2013 19:41 Para: 'Even Rouault' Cc: gdal-dev@lists.osgeo.org Assunto: RE: [gdal-dev] gtiff with internal mask Well, you're right... I messed up recreating the footsteps... So, cleaning up: 1) create a rgba vrt using gdalwarp to cut the original mosaic with a shapefile gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 312 -co alpha=yes -dstalpha -cutline shapes\index_diss.shp -of vrt originalmosaic.vrt test_rgba_uncomp.vrt 2) create a masked uncompressed vrt using gdal_translate (because I couldn't compress right away, with errors of missing StripOffsets field) gdal_translate -b 1 -b 2 -b 3 -mask 4 -co alpha=no -co photometric=rgb -co interleave=pixel -co tiled=yes --config GDAL_TIFF_INTERNAL_MASK YES -of vrt test_rgba_uncomp.vrt testmask_uncomp.vrt 3) create final masked compressed tiff using gdal_translate gdal_translate -co alpha=no -co photometric=ycbcr -co interleave=pixel -co tiled=yes -co compress=jpeg --config GDAL_TIFF_INTERNAL_MASK YES testmask_uncomp.vrt test_finalmask.tif The power of VRT amazes me! Also regarding sizes: I still got the same ratio of 8x smaller. To use this in QGIS you still have to create a rgba vrt: 4) gdal_translate -of VRT -b 1 -b 2 -b 3 -b mask test_finalmask.tif test_finalmask_qgis.vrt (couldn't get the 4ª band to be recognized as an alpha band though) So I think now it's correct... I am going to apply this to a real-case mosaic and report back... Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: quarta-feira, 17 de Julho de 2013 13:45 Para: Duarte Carreira Cc: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] gtiff with internal mask Le mercredi 17 juillet 2013 14:23:29, Duarte Carreira a écrit : Hi Even. Thanks so much for your tip! It works. I did have to specify I did not want an alpha band when cutting with the shapefile: gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 256 -co photometric=ycbcr -co compress=jpeg -co alpha=no --config GDAL_TIFF_INTERNAL_MASK YES -cutline shapes\index_diss.shp vrtini.vrt testmask.tif Hum, what I don't understand is how the above will produce a mask band in the output file... Does gdalinfo on testmask.tif show something like : Band 1 Block= Type=Byte, ColorInterp=Red Mask Flags: PER_DATASET Band 2 Block= Type=Byte, ColorInterp=Green Mask Flags: PER_DATASET Band 3 Block=... Type=Byte, ColorInterp=Blue Mask Flags: PER_DATASET I think you would need to do the above in 2 steps: 1) gdalwarp to an uncompressed RGBA TIFF 2) gdal_translate to turn the RGBA into a YCbCr JPEG-compressed GTiff + mask band Seems arcgis recognizes the masked tiff, but is much much slower displaying than using a normal alpha band. Perhaps adding -co TILED=YES will help ? (random guess) QGIS does need the vrt trick but then is as fast as usual. Also, the size is must smaller when using an internal mask: from a 219MB rgba,band interleaved,jpegd image to a 27MB ycbcr,pixel interleaved,jpegd masked tiff (8x smaller). Unless I missed something... Not completely surprising although more important than I would have expected. YCbCr compression usually gives a 2x to 3x compression bonus, and due to the usual nature of mask bands, the 1bit deflate compression of the mask band will give better results (both visually and in size) that the 8bit
Re: [gdal-dev] gtiff with internal mask
Ok, para conseguir construir as pirâmides temos de ter a máscara separada e não incluída no próprio tiff. Para isso basta alterar o 3º passo que cria efectivamente o tiff a partir do vrt, removendo a opção de criar a máscara interna, ficando assim: gdal_translate -co alpha=no -co photometric=ycbcr -co interleave=pixel -co tiled=yes -co compress=jpeg testmask_uncomp.vrt test_finalmask.tif Isto basta para que seja criado um tif rgb e um ficheiro .msk separados. *** Para criar um rgb+iv num só tiff, com máscara! *** Alterar o mosaico vrt criado no 2º passo, que já inclui a máscara, e criar uma banda Gray com origem no mosaico vrt dos infravermelhos. Depois é só converter mas usando as opções: -co alpha=no -co photometric=rgb e talvez -co interleave=band (para não dar erro). Não podemos usar ycbcr porque só dá para 3 bandas e nós temos 4. DC -Mensagem original- De: Duarte Carreira Enviada: quinta-feira, 18 de Julho de 2013 13:01 Para: 'Even Rouault' Cc: 'gdal-dev@lists.osgeo.org' Assunto: RE: [gdal-dev] gtiff with internal mask So... now I 've got issues building the overviews. Gdaladdo loses the transparency so I get to see black areas where I was supposed to see nothing. When I zoom in black disappears. Any known way to overcome this? Thanks, Duarte -Mensagem original- De: Duarte Carreira Enviada: quarta-feira, 17 de Julho de 2013 19:41 Para: 'Even Rouault' Cc: gdal-dev@lists.osgeo.org Assunto: RE: [gdal-dev] gtiff with internal mask Well, you're right... I messed up recreating the footsteps... So, cleaning up: 1) create a rgba vrt using gdalwarp to cut the original mosaic with a shapefile gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 312 -co alpha=yes -dstalpha -cutline shapes\index_diss.shp -of vrt originalmosaic.vrt test_rgba_uncomp.vrt 2) create a masked uncompressed vrt using gdal_translate (because I couldn't compress right away, with errors of missing StripOffsets field) gdal_translate -b 1 -b 2 -b 3 -mask 4 -co alpha=no -co photometric=rgb -co interleave=pixel -co tiled=yes --config GDAL_TIFF_INTERNAL_MASK YES -of vrt test_rgba_uncomp.vrt testmask_uncomp.vrt 3) create final masked compressed tiff using gdal_translate gdal_translate -co alpha=no -co photometric=ycbcr -co interleave=pixel -co tiled=yes -co compress=jpeg --config GDAL_TIFF_INTERNAL_MASK YES testmask_uncomp.vrt test_finalmask.tif The power of VRT amazes me! Also regarding sizes: I still got the same ratio of 8x smaller. To use this in QGIS you still have to create a rgba vrt: 4) gdal_translate -of VRT -b 1 -b 2 -b 3 -b mask test_finalmask.tif test_finalmask_qgis.vrt (couldn't get the 4ª band to be recognized as an alpha band though) So I think now it's correct... I am going to apply this to a real-case mosaic and report back... Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: quarta-feira, 17 de Julho de 2013 13:45 Para: Duarte Carreira Cc: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] gtiff with internal mask Le mercredi 17 juillet 2013 14:23:29, Duarte Carreira a écrit : Hi Even. Thanks so much for your tip! It works. I did have to specify I did not want an alpha band when cutting with the shapefile: gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 256 -co photometric=ycbcr -co compress=jpeg -co alpha=no --config GDAL_TIFF_INTERNAL_MASK YES -cutline shapes\index_diss.shp vrtini.vrt testmask.tif Hum, what I don't understand is how the above will produce a mask band in the output file... Does gdalinfo on testmask.tif show something like : Band 1 Block= Type=Byte, ColorInterp=Red Mask Flags: PER_DATASET Band 2 Block= Type=Byte, ColorInterp=Green Mask Flags: PER_DATASET Band 3 Block=... Type=Byte, ColorInterp=Blue Mask Flags: PER_DATASET I think you would need to do the above in 2 steps: 1) gdalwarp to an uncompressed RGBA TIFF 2) gdal_translate to turn the RGBA into a YCbCr JPEG-compressed GTiff + mask band Seems arcgis recognizes the masked tiff, but is much much slower displaying than using a normal alpha band. Perhaps adding -co TILED=YES will help ? (random guess) QGIS does need the vrt trick but then is as fast as usual. Also, the size is must smaller when using an internal mask: from a 219MB rgba,band interleaved,jpegd image to a 27MB ycbcr,pixel interleaved,jpegd masked tiff (8x smaller). Unless I missed something... Not completely surprising although more important than I would have expected. YCbCr compression usually gives a 2x to 3x compression bonus, and due to the usual nature of mask bands, the 1bit deflate compression of the mask band will give better results (both visually and in size) that the 8bit JPEG compression of the alpha band. Thanks again! Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: terça-feira, 16 de Julho de 2013 18:58 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira
Re: [gdal-dev] gtiff with internal mask
So... now I 've got issues building the overviews. Gdaladdo loses the transparency so I get to see black areas where I was supposed to see nothing. When I zoom in black disappears. Any known way to overcome this? Thanks, Duarte -Mensagem original- De: Duarte Carreira Enviada: quarta-feira, 17 de Julho de 2013 19:41 Para: 'Even Rouault' Cc: gdal-dev@lists.osgeo.org Assunto: RE: [gdal-dev] gtiff with internal mask Well, you're right... I messed up recreating the footsteps... So, cleaning up: 1) create a rgba vrt using gdalwarp to cut the original mosaic with a shapefile gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 312 -co alpha=yes -dstalpha -cutline shapes\index_diss.shp -of vrt originalmosaic.vrt test_rgba_uncomp.vrt 2) create a masked uncompressed vrt using gdal_translate (because I couldn't compress right away, with errors of missing StripOffsets field) gdal_translate -b 1 -b 2 -b 3 -mask 4 -co alpha=no -co photometric=rgb -co interleave=pixel -co tiled=yes --config GDAL_TIFF_INTERNAL_MASK YES -of vrt test_rgba_uncomp.vrt testmask_uncomp.vrt 3) create final masked compressed tiff using gdal_translate gdal_translate -co alpha=no -co photometric=ycbcr -co interleave=pixel -co tiled=yes -co compress=jpeg --config GDAL_TIFF_INTERNAL_MASK YES testmask_uncomp.vrt test_finalmask.tif The power of VRT amazes me! Also regarding sizes: I still got the same ratio of 8x smaller. To use this in QGIS you still have to create a rgba vrt: 4) gdal_translate -of VRT -b 1 -b 2 -b 3 -b mask test_finalmask.tif test_finalmask_qgis.vrt (couldn't get the 4ª band to be recognized as an alpha band though) So I think now it's correct... I am going to apply this to a real-case mosaic and report back... Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: quarta-feira, 17 de Julho de 2013 13:45 Para: Duarte Carreira Cc: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] gtiff with internal mask Le mercredi 17 juillet 2013 14:23:29, Duarte Carreira a écrit : Hi Even. Thanks so much for your tip! It works. I did have to specify I did not want an alpha band when cutting with the shapefile: gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 256 -co photometric=ycbcr -co compress=jpeg -co alpha=no --config GDAL_TIFF_INTERNAL_MASK YES -cutline shapes\index_diss.shp vrtini.vrt testmask.tif Hum, what I don't understand is how the above will produce a mask band in the output file... Does gdalinfo on testmask.tif show something like : Band 1 Block= Type=Byte, ColorInterp=Red Mask Flags: PER_DATASET Band 2 Block= Type=Byte, ColorInterp=Green Mask Flags: PER_DATASET Band 3 Block=... Type=Byte, ColorInterp=Blue Mask Flags: PER_DATASET I think you would need to do the above in 2 steps: 1) gdalwarp to an uncompressed RGBA TIFF 2) gdal_translate to turn the RGBA into a YCbCr JPEG-compressed GTiff + mask band Seems arcgis recognizes the masked tiff, but is much much slower displaying than using a normal alpha band. Perhaps adding -co TILED=YES will help ? (random guess) QGIS does need the vrt trick but then is as fast as usual. Also, the size is must smaller when using an internal mask: from a 219MB rgba,band interleaved,jpegd image to a 27MB ycbcr,pixel interleaved,jpegd masked tiff (8x smaller). Unless I missed something... Not completely surprising although more important than I would have expected. YCbCr compression usually gives a 2x to 3x compression bonus, and due to the usual nature of mask bands, the 1bit deflate compression of the mask band will give better results (both visually and in size) that the 8bit JPEG compression of the alpha band. Thanks again! Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: terça-feira, 16 de Julho de 2013 18:58 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira Assunto: Re: [gdal-dev] gtiff with internal mask Le mardi 16 juillet 2013 12:29:37, Duarte Carreira a écrit : I'm trying to create gtiffs with transparent areas where there are voids between my original images. I have successfully done it by using an alpha band but this process created 3x larger images. I think partly because of an additional 4th band and partly because I cannot use ycbcr to compress with jpeg. So I thought of using internal masks expecting to get smaller images but cannot get it to work. When converting from the rgba image using -mask 4 I cannot see the transparency and get black areas: gdal_translate -b 1 -b 2 -b 3 -ma sk 4 --config GDAL_TIFF_INTERNAL_MASK YES test_alpha.tif test_internalmask.tif I tried looking at the images using qgis and paint.net. I know QGIS doesn't use the GDALGetMaskBand() API, so no real surprise (that would likely be a possible QGIS enhancement). And paint.net probably doesn't support this feature of the TIFF format either. MapServer does support mask
Re: [gdal-dev] gtiff with internal mask
Hi Even. Thanks so much for your tip! It works. I did have to specify I did not want an alpha band when cutting with the shapefile: gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 256 -co photometric=ycbcr -co compress=jpeg -co alpha=no --config GDAL_TIFF_INTERNAL_MASK YES -cutline shapes\index_diss.shp vrtini.vrt testmask.tif Seems arcgis recognizes the masked tiff, but is much much slower displaying than using a normal alpha band. QGIS does need the vrt trick but then is as fast as usual. Also, the size is must smaller when using an internal mask: from a 219MB rgba,band interleaved,jpegd image to a 27MB ycbcr,pixel interleaved,jpegd masked tiff (8x smaller). Unless I missed something... Thanks again! Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: terça-feira, 16 de Julho de 2013 18:58 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira Assunto: Re: [gdal-dev] gtiff with internal mask Le mardi 16 juillet 2013 12:29:37, Duarte Carreira a écrit : I'm trying to create gtiffs with transparent areas where there are voids between my original images. I have successfully done it by using an alpha band but this process created 3x larger images. I think partly because of an additional 4th band and partly because I cannot use ycbcr to compress with jpeg. So I thought of using internal masks expecting to get smaller images but cannot get it to work. When converting from the rgba image using -mask 4 I cannot see the transparency and get black areas: gdal_translate -b 1 -b 2 -b 3 -ma sk 4 --config GDAL_TIFF_INTERNAL_MASK YES test_alpha.tif test_internalmask.tif I tried looking at the images using qgis and paint.net. I know QGIS doesn't use the GDALGetMaskBand() API, so no real surprise (that would likely be a possible QGIS enhancement). And paint.net probably doesn't support this feature of the TIFF format either. MapServer does support mask bands however. For QGIS, you could workaround that however. Imagine that you have a YCbCr TIFF + internal mask in.tif. You can do : gdal_translate -of VRT -b 1 -b 2 -b 3 -b mask in.tif out.vrt And QGIS should handle out.vrt as a regular RGBA dataset. Any hints or help appreciated. Thanks, Duarte Duarte Carreira Diretor | Dep. Informa??o Geogr?fica e Cartografia www.edia.pthttp://www.edia.pt www.alqueva.com.pthttp://www.alqueva.com.pt Tel. +351 284315100 [http://www.edia.pt/edia/images/edia_logo2.gif]http://www.edia.pt -- Geospatial professional services http://even.rouault.free.fr/services.html ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] gtiff with internal mask
Well, you're right... I messed up recreating the footsteps... So, cleaning up: 1) create a rgba vrt using gdalwarp to cut the original mosaic with a shapefile gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 312 -co alpha=yes -dstalpha -cutline shapes\index_diss.shp -of vrt originalmosaic.vrt test_rgba_uncomp.vrt 2) create a masked uncompressed vrt using gdal_translate (because I couldn't compress right away, with errors of missing StripOffsets field) gdal_translate -b 1 -b 2 -b 3 -mask 4 -co alpha=no -co photometric=rgb -co interleave=pixel -co tiled=yes --config GDAL_TIFF_INTERNAL_MASK YES -of vrt test_rgba_uncomp.vrt testmask_uncomp.vrt 3) create final masked compressed tiff using gdal_translate gdal_translate -co alpha=no -co photometric=ycbcr -co interleave=pixel -co tiled=yes -co compress=jpeg --config GDAL_TIFF_INTERNAL_MASK YES testmask_uncomp.vrt test_finalmask.tif The power of VRT amazes me! Also regarding sizes: I still got the same ratio of 8x smaller. To use this in QGIS you still have to create a rgba vrt: 4) gdal_translate -of VRT -b 1 -b 2 -b 3 -b mask test_finalmask.tif test_finalmask_qgis.vrt (couldn't get the 4ª band to be recognized as an alpha band though) So I think now it's correct... I am going to apply this to a real-case mosaic and report back... Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: quarta-feira, 17 de Julho de 2013 13:45 Para: Duarte Carreira Cc: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] gtiff with internal mask Le mercredi 17 juillet 2013 14:23:29, Duarte Carreira a écrit : Hi Even. Thanks so much for your tip! It works. I did have to specify I did not want an alpha band when cutting with the shapefile: gdalwarp -multi -wm 480 --config GDAL_CACHEMAX 256 -co photometric=ycbcr -co compress=jpeg -co alpha=no --config GDAL_TIFF_INTERNAL_MASK YES -cutline shapes\index_diss.shp vrtini.vrt testmask.tif Hum, what I don't understand is how the above will produce a mask band in the output file... Does gdalinfo on testmask.tif show something like : Band 1 Block= Type=Byte, ColorInterp=Red Mask Flags: PER_DATASET Band 2 Block= Type=Byte, ColorInterp=Green Mask Flags: PER_DATASET Band 3 Block=... Type=Byte, ColorInterp=Blue Mask Flags: PER_DATASET I think you would need to do the above in 2 steps: 1) gdalwarp to an uncompressed RGBA TIFF 2) gdal_translate to turn the RGBA into a YCbCr JPEG-compressed GTiff + mask band Seems arcgis recognizes the masked tiff, but is much much slower displaying than using a normal alpha band. Perhaps adding -co TILED=YES will help ? (random guess) QGIS does need the vrt trick but then is as fast as usual. Also, the size is must smaller when using an internal mask: from a 219MB rgba,band interleaved,jpegd image to a 27MB ycbcr,pixel interleaved,jpegd masked tiff (8x smaller). Unless I missed something... Not completely surprising although more important than I would have expected. YCbCr compression usually gives a 2x to 3x compression bonus, and due to the usual nature of mask bands, the 1bit deflate compression of the mask band will give better results (both visually and in size) that the 8bit JPEG compression of the alpha band. Thanks again! Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: terça-feira, 16 de Julho de 2013 18:58 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira Assunto: Re: [gdal-dev] gtiff with internal mask Le mardi 16 juillet 2013 12:29:37, Duarte Carreira a écrit : I'm trying to create gtiffs with transparent areas where there are voids between my original images. I have successfully done it by using an alpha band but this process created 3x larger images. I think partly because of an additional 4th band and partly because I cannot use ycbcr to compress with jpeg. So I thought of using internal masks expecting to get smaller images but cannot get it to work. When converting from the rgba image using -mask 4 I cannot see the transparency and get black areas: gdal_translate -b 1 -b 2 -b 3 -ma sk 4 --config GDAL_TIFF_INTERNAL_MASK YES test_alpha.tif test_internalmask.tif I tried looking at the images using qgis and paint.net. I know QGIS doesn't use the GDALGetMaskBand() API, so no real surprise (that would likely be a possible QGIS enhancement). And paint.net probably doesn't support this feature of the TIFF format either. MapServer does support mask bands however. For QGIS, you could workaround that however. Imagine that you have a YCbCr TIFF + internal mask in.tif. You can do : gdal_translate -of VRT -b 1 -b 2 -b 3 -b mask in.tif out.vrt And QGIS should handle out.vrt as a regular RGBA dataset. Any hints or help appreciated. Thanks, Duarte Duarte Carreira Diretor | Dep. Informa??o Geogr?fica e Cartografia
[gdal-dev] gtiff with internal mask
I'm trying to create gtiffs with transparent areas where there are voids between my original images. I have successfully done it by using an alpha band but this process created 3x larger images. I think partly because of an additional 4th band and partly because I cannot use ycbcr to compress with jpeg. So I thought of using internal masks expecting to get smaller images but cannot get it to work. When converting from the rgba image using -mask 4 I cannot see the transparency and get black areas: gdal_translate -b 1 -b 2 -b 3 -ma sk 4 --config GDAL_TIFF_INTERNAL_MASK YES test_alpha.tif test_internalmask.tif I tried looking at the images using qgis and paint.net. Any hints or help appreciated. Thanks, Duarte Duarte Carreira Diretor | Dep. Informa??o Geogr?fica e Cartografia www.edia.pthttp://www.edia.pt www.alqueva.com.pthttp://www.alqueva.com.pt Tel. +351 284315100 [http://www.edia.pt/edia/images/edia_logo2.gif]http://www.edia.pt ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Tiling aerial photos
Hi Paul. I had to build a big vrt. The one trick that got it loading very fast into qgis was adding statistics to each rasterband: Metadata MDI key=STATISTICS_MAXIMUM255/MDI MDI key=STATISTICS_MEAN111.6784426525/MDI MDI key=STATISTICS_MINIMUM0/MDI MDI key=STATISTICS_STDDEV52.100074055799/MDI /Metadata I just got stats from a subset and used that for all bands. Duarte De: Paul Meems (Top-X) [mailto:p.me...@topx-group.nl] Enviada: quinta-feira, 12 de Julho de 2012 12:11 Para: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] Tiling aerial photos Thanks Dmitry and Even, The aerial photos are north-up and are in the same projection and I think also in the same resolution. It are commercial aerial photos. The Correlator project sounds very interesting but not necessary in my case. I'll try to implement the VRT format and see what the performance will be. If it is fast we don't need to tile. Thanks, Paul 2012/7/12 Even Rouault even.roua...@mines-paris.orgmailto:even.roua...@mines-paris.org Selon Paul Meems (Top-X) p.me...@topx-group.nlmailto:p.me...@topx-group.nl: Thanks Even, http://www.gdal.org/gdalbuildvrt.html does seems very interesting. As I understand it, it will do the merging part (without actually merging). The VRT driver will do on-the-fly merging of tiles that have overlapping. The VRT itself is just a XML file. But it doesn't do the tiling part, right? No, I wasn't sure if your photos were already regularly tiled or not. Note that the VRT accepts non regularly tiled images. They can have overlapping, gaps, different resolutions, etc. The main constraints are : - they are in the same projection - they are north-up, that is to say there is no rotation or skewing term in their geotransform matrix - they have the same number of bands Or is the vrt so optimized tiling is no longer necessary? Not necessary. Note that the VRT has no internal spatial indexing, so if you have several dozains of thousands of images in a VRT, it might slow down because it will iterate over all the image descriptions (without needing to open them however, all the information is in the VRT) to see if they intersect with the request window. But I'd expect the number of images in the VRT to be really high for that effect to become noticeable. Thanks, Paul 2012/7/12 Dmitry Baryshnikov poli...@mail.rumailto:poli...@mail.ru 12.07.2012 14:36, Paul Meems (Top-X) пишет: Thanks Even, http://www.gdal.org/gdalbuildvrt.html does seems very interesting. As I understand it, it will do the merging part (without actually merging). But it doesn't do the tiling part, right? Or is the vrt so optimized tiling is no longer necessary? Thanks, Paul 2012/7/12 Even Rouault even.roua...@mines-paris.orgmailto:even.roua...@mines-paris.org Selon Paul Meems (Top-X) p.me...@topx-group.nlmailto:p.me...@topx-group.nl: Hi list, I have several aerial photos and I use MapWindow GIS to view them. MapWindow is already using GDAL v1.8 Instead of loading each aerial photo as an individual layer I want to create a local tiles store of the photos. I can then just load the tiles I need based on the scale and view. This will most likely improve the performance. The process will be something like this: 1. Get the filenames of the photos 2. Merge them into one 3. Create tiles 4. Put the tiles in a SQLite database My first question: Is the above suggested process correct or should I do it differently? My second question: What GDAL functions should I look into to accomplish my process? You could try to make a VRT from all your photos. It will be seen as a single GDAL dataset, and will take care of the burden of opening the underlying photos when needed. You can use the gdalbuildvrt utility to create the VRT from the photos. Thanks, Paul Meems The Netherlands ___ gdal-dev mailing list gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev There is an interesting work connected aerial imagery: http://trac.osgeo.org/gdal/wiki/Correlator http://correlatorgsoc2012.blogspot.com/ Best regards, Dmitry ___ gdal-dev mailing list gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Tiling aerial photos
For some reason stats in the individual files did not solve the issue, had to put it in the vrt. Duarte -Mensagem original- De: Etienne Tourigny [mailto:etourigny@gmail.com] Enviada: quinta-feira, 12 de Julho de 2012 13:28 Para: Duarte Carreira Cc: p.me...@topx-group.nl; gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] Tiling aerial photos You should generate the stats for every file, as you will probably get incorrect results with your method for f in *.tif; do gdalinfo -stats $f ; done On Thu, Jul 12, 2012 at 9:03 AM, Duarte Carreira dcarre...@edia.pt wrote: Hi Paul. I had to build a big vrt. The one trick that got it loading very fast into qgis was adding statistics to each rasterband: Metadata MDI key=STATISTICS_MAXIMUM255/MDI MDI key=STATISTICS_MEAN111.6784426525/MDI MDI key=STATISTICS_MINIMUM0/MDI MDI key=STATISTICS_STDDEV52.100074055799/MDI /Metadata I just got stats from a subset and used that for all bands. Duarte De: Paul Meems (Top-X) [mailto:p.me...@topx-group.nl] Enviada: quinta-feira, 12 de Julho de 2012 12:11 Para: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] Tiling aerial photos Thanks Dmitry and Even, The aerial photos are north-up and are in the same projection and I think also in the same resolution. It are commercial aerial photos. The Correlator project sounds very interesting but not necessary in my case. I'll try to implement the VRT format and see what the performance will be. If it is fast we don't need to tile. Thanks, Paul 2012/7/12 Even Rouault even.roua...@mines-paris.org Selon Paul Meems (Top-X) p.me...@topx-group.nl: Thanks Even, http://www.gdal.org/gdalbuildvrt.html does seems very interesting. As I understand it, it will do the merging part (without actually merging). The VRT driver will do on-the-fly merging of tiles that have overlapping. The VRT itself is just a XML file. But it doesn't do the tiling part, right? No, I wasn't sure if your photos were already regularly tiled or not. Note that the VRT accepts non regularly tiled images. They can have overlapping, gaps, different resolutions, etc. The main constraints are : - they are in the same projection - they are north-up, that is to say there is no rotation or skewing term in their geotransform matrix - they have the same number of bands Or is the vrt so optimized tiling is no longer necessary? Not necessary. Note that the VRT has no internal spatial indexing, so if you have several dozains of thousands of images in a VRT, it might slow down because it will iterate over all the image descriptions (without needing to open them however, all the information is in the VRT) to see if they intersect with the request window. But I'd expect the number of images in the VRT to be really high for that effect to become noticeable. Thanks, Paul 2012/7/12 Dmitry Baryshnikov poli...@mail.ru 12.07.2012 14:36, Paul Meems (Top-X) пишет: Thanks Even, http://www.gdal.org/gdalbuildvrt.html does seems very interesting. As I understand it, it will do the merging part (without actually merging). But it doesn't do the tiling part, right? Or is the vrt so optimized tiling is no longer necessary? Thanks, Paul 2012/7/12 Even Rouault even.roua...@mines-paris.org Selon Paul Meems (Top-X) p.me...@topx-group.nl: Hi list, I have several aerial photos and I use MapWindow GIS to view them. MapWindow is already using GDAL v1.8 Instead of loading each aerial photo as an individual layer I want to create a local tiles store of the photos. I can then just load the tiles I need based on the scale and view. This will most likely improve the performance. The process will be something like this: 1. Get the filenames of the photos 2. Merge them into one 3. Create tiles 4. Put the tiles in a SQLite database My first question: Is the above suggested process correct or should I do it differently? My second question: What GDAL functions should I look into to accomplish my process? You could try to make a VRT from all your photos. It will be seen as a single GDAL dataset, and will take care of the burden of opening the underlying photos when needed. You can use the gdalbuildvrt utility to create the VRT from the photos. Thanks, Paul Meems The Netherlands ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev There is an interesting work connected aerial imagery: http://trac.osgeo.org/gdal/wiki/Correlator http://correlatorgsoc2012.blogspot.com/ Best regards, Dmitry ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org
[gdal-dev] ogrinfo and oracle sql expressions, maybe arcsde
Hi again.
I am trying to determine why I cannot get any sql queries to work against
oracle and through arcsde driver.
It seems the sql sentence keeps adding single-quotes to both field and value,
which is not valid.
For instance, a simple numeric clause:
Ogrinfo -so -ro -al -sql SELECT * FROM 'USER.TABLE' WHERE FEATID=1234
SDE:connection string
Results in errors being reported in the client:
(...)
using driver `SDE' successful.
Layer name: GDBMAN.SREGA_PERIMETROS_REGA
Geometry: Unknown (any)
ERROR 1: SE_stream_fetch: -51/Underlying DBMS error
Feature Count: -1
OGR_SDE: Loading GDBMAN.SREGA_PERIMETROS_REGA layerinfo.
ERROR 1: SE_stream_fetch: -51/Underlying DBMS error
Layer SRS WKT:
(...)
In ArcSDE logs I can see this sql query being reported:
WhereClause: ('OBJECTID_1') = (21522)
Which in turn raises an error in Oracle, since single-quotes are meant for
values and not for field names. For field names oracle allows double-quotes. So
it should really be: (OBJECTID_2)=(21522).
Then I tried a string query. And that doesn't work either... I can't figure out
how to build a sql where clause in ogr with a string field that works...
So, has anyone any successful experience with ogr -sql queries in Oracle or
ArcSDE? Any tips? I'm not sure where the error happens, either in sde driver or
ogr sql parser...
Thanks,
Duarte
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] ogr, vrt - connecting to arcsde errors only with srcsql
Hi Even. Thanks, using single-quotes around the fully qualified table name solved it. But I cannot use a where clause... it always results in an error. I have tried everything I could remember: single quotes, double-quotes, fully qualified field name, quotes and double-quotes on the value... nothing worked. A simple query with a numeric field: SELECT * FROM 'GDBMAN.SREGA_PERIMETROS_REGA' WHERE OBJECTID_1=21522 Gives this error: Layer name: regadio ERROR 1: SE_stream_fetch: -51/Underlying DBMS error If I use the fully qualified field name: SELECT * FROM 'GDBMAN.SREGA_PERIMETROS_REGA' WHERE 'GDBMAN.SREGA_PERIMETROS_REGA.OBJECTID_1'=21522 Results in this error: ERROR 1: Type mismatch or improper type of arguments to = operator. ERROR 1: SQL statement failed, or returned no layer result: SELECT * FROM 'GDBMAN.SREGA_PERIMETROS_REGA' WHERE 'GDBMAN.SREGA_PERIMETROS_REGA .OBJECTID_1'=21522 FAILURE: Unable to open datasource `sde_Regadio_SQL.vrt' with the following drivers. If I take out the single-quotes around the field name I get another error: ERROR 1: SQL Expression Parsing Error: syntax error ERROR 1: SQL statement failed, or returned no layer result: SELECT * FROM 'GDBMAN.SREGA_PERIMETROS_REGA' WHERE GDBMAN.SREGA_PERIMETROS_REGA. OBJECTID_1=21522 FAILURE: Unable to open datasource `sde_Regadio_SQL.vrt' with the following drivers. Any clues on how to apply a SQL query here? Thanks again, Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: segunda-feira, 3 de Outubro de 2011 18:51 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira Assunto: Re: [gdal-dev] ogr, vrt - connecting to arcsde errors only with srcsql Le lundi 03 octobre 2011 19:17:25, Duarte Carreira a écrit : Hi there. I'm having trouble using a SrcSQL tag in a vrt that connects to an ArcSDE feature class. If I use SrcLayer everything works. But if I remove SrcLayer, and add a SrcSQL with the simplest of queries (SELECT * FROM USER.TABLE1) I get errors in ogrinfo: ERROR 1: SQL Expression Parsing Error: syntax error ERROR 1: SQL statement failed, or returned no layer result: SELECT * FROM GDBMAN.SREGA_PERIMETROS_REGA ERROR 1: SQL Expression Parsing Error: syntax error ERROR 1: SQL statement failed, or returned no layer result: SELECT * FROM GDBMAN.SREGA_PERIMETROS_REGA FAILURE: Unable to open datasource `sde_Regadio_SQL.vrt' with the following drivers. Any idea why this is? I guess it is because of the . in the table name. You should surround the table name by simple quote character. SELECT * FROM 'USER.TABLE1' Thanks, Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] ogr, vrt - connecting to arcsde errors only with srcsql
Hi there. I'm having trouble using a SrcSQL tag in a vrt that connects to an ArcSDE feature class. If I use SrcLayer everything works. But if I remove SrcLayer, and add a SrcSQL with the simplest of queries (SELECT * FROM USER.TABLE1) I get errors in ogrinfo: ERROR 1: SQL Expression Parsing Error: syntax error ERROR 1: SQL statement failed, or returned no layer result: SELECT * FROM GDBMAN.SREGA_PERIMETROS_REGA ERROR 1: SQL Expression Parsing Error: syntax error ERROR 1: SQL statement failed, or returned no layer result: SELECT * FROM GDBMAN.SREGA_PERIMETROS_REGA FAILURE: Unable to open datasource `sde_Regadio_SQL.vrt' with the following drivers. Any idea why this is? Thanks, Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] SDE driver: connecting to rasters never returns
Hello listers. So now that OGR connecting is solved by specifying the version name (or maybe by making a readonly connection), I'm stumbling on the same effect when connecting to a raster with gdalinfo or gdal_translate. But there is no option to specify version or readonly... My gdalinfo/translate commands never end... they keep going with 0 cpu... This is the same raster that I used when reporting #2063http://trac.osgeo.org/gdal/ticket/2063... One example command: gdal_translate -of vrt SDE:server,5151,database,user,pwd,user.MOSAIC_ORTO_10K_IGEOE,RASTER c:\temp\gdalteste.vrt Anyway I can debug this? Thanks. Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] SDE driver: connecting to rasters never returns
Howard, This is the output: GDAL: Auto register C:\OSGeo4W\bin\gdalplugins\1.8\gdal_MG4Lidar.dll using GDALR egister_MG4Lidar. GDAL: Auto register C:\OSGeo4W\bin\gdalplugins\1.8\gdal_MrSID.dll using GDALRegi ster_MrSID. GDAL: Auto register C:\OSGeo4W\bin\gdalplugins\1.8\gdal_SDE.dll using GDALRegist er_SDE. SDERASTER: Open(SDE:xxx,5151,database,,,sig_owner. MOSAIC_ORTO_10K_IGEOE,RASTER) revealed 7 tokens. SDERASTER: SDE Column name is 'RASTER' SDERASTER: 'sig_owner.MOSAIC_ORTO_10K_IGEOE' raster layer specified... using it directly with 'RASTER' as the raster column name. SDERASTER: minx: 174950.2, miny: 99950.2, maxx: 295049.8, maxy: 2100 49.8 SDERASTER: Tile origin: 174914.2, 210083.4 GDAL: GDALDeregister_GTiff() called. So it stops at SDERASTER: Tile origin: ... Because the last line appears only after pressing Ctrl-C (GDAL: GDALDeregister GTiff). I'm really sorry you don't have access to ArcSDE anymore... I can offer my time to test though. Duarte -Mensagem original- De: Howard Butler [mailto:hobu@gmail.com] Enviada: segunda-feira, 13 de Junho de 2011 16:28 Para: Duarte Carreira Cc: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] SDE driver: connecting to rasters never returns On Jun 13, 2011, at 9:38 AM, Duarte Carreira wrote: Hello listers. So now that OGR connecting is solved by specifying the version name (or maybe by making a readonly connection), I'm stumbling on the same effect when connecting to a raster with gdalinfo or gdal_translate. But there is no option to specify version or readonly... My gdalinfo/translate commands never end... they keep going with 0 cpu... This is the same raster that I used when reporting #2063... One example command: gdal_translate -of vrt SDE:server,5151,database,user,pwd,user.MOSAIC_ORTO_10K_IGEOE,RASTER c:\temp\gdalteste.vrt Anyway I can debug this? set the environment variable CPL_DEBUG=on This should allow you to watch the connection happen and at the very least, see where it is hanging in either the SDE connection or query process. Note that I wrote the SDE driver, but I haven't had a running SDE instance for years, and I am in no position to test. sorry. Howard ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] Table Names (FGDB)
Paul, The schema.table is the most similar to the structure of the fgdb, but may be unwanted complexity. Can this be an option? Like yes, consider the dataset name? Or no, discard dataset name. I can see myself in both situations... Is the driver read only? If not, what will happen when you try the reverse? From PgSql to fgdb? Duarte -Mensagem original- De: Paul Ramsey [mailto:pram...@cleverelephant.ca] Enviada: segunda-feira, 30 de Maio de 2011 22:47 Para: gdal-dev@lists.osgeo.org Assunto: [gdal-dev] Table Names (FGDB) So, my goal is to map information from FGDB to PostGIS and as much fidelity as possible. FGDB includes a class called Feature Dataset which is basically a folder that holds Feature Class objects, which map directly to OGR layers. So the Feature Dataset then acts a good deal like a schema in PostGIS. I've noticed that the PgSQL driver supports a SCHEMA keyword to allow you to write to a schema without schema-qualified table names, and that it also notices when table names are schema qualified and puts them in the appropriate place. So the name of a layer read from a schema would show up in OGR as schemaname.tablename. In FGDB, we have the path of a Feature Class or Table, which looks like this \FeatureDataset\FeatureClass. Similar to the schema qualification, yet different. As it stands right now, the FeatureDataset portion of the path is discarded in the public API, so that the OGR layer name is just the FeatureClass, with no reference to the FeatureDataset. That makes it hard to create a mapping from FGDB (\FeatureDataset\FeatureClass) to PostGIS (FeatureDataset.FeatureClass). The FGDB implementation currently returns GeoDatabase.GetQueryName() as the layer name, which Gets the query name (the name to use in SQL statements) of a table based on its path. If it returned the path name, then the FeatureDataset qualification information could be preserved in other contexts. On the other hand, perhaps the QueryName is more useful to more users than the path? It looks like the QueryName is expected to be unique, so probably FeatureClass and Table names have to be unique regardless of what FeatureDataset they appear in (true?). So my options are: (a) - change the current OGR layer name to the path name (b) - change the current OGR layer name to a schema.table analogue and handle appropriately (c) - leave the current OGR layer name as is (tablename regardless of containing folders) and make a FGDB-only method for accessing the pathname for my particular purposes Preferences? P. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] Table Names (FGDB)
Well, I'm no expert!! As far as applications go, the few I use *had* problems with schemas other than public, but I think this has been worked out for some time now. So I think schemas are well supported - but it may be worthwhile asking qgis and gvsig how they are faring with these today... I can hook up some people from qgis to this thread (if they're not following already...). Duarte -Mensagem original- De: Paul Ramsey [mailto:pram...@cleverelephant.ca] Enviada: terça-feira, 31 de Maio de 2011 17:43 Para: Duarte Carreira Cc: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] Table Names (FGDB) This is why I'm looking for guidance from the OGR experts... what layer name styles work best for interoperation? If I emit schema.table layers on read, will apps be happy or sad? The driver is moving towards read/write. I will have no trouble dealing with schema.table names as I can just map them to \FeatureDataset\FeatureClass under the covers, and create new FeatureDataset folders as necessary. P. On Tue, May 31, 2011 at 5:05 AM, Duarte Carreira dcarre...@edia.pt wrote: Paul, The schema.table is the most similar to the structure of the fgdb, but may be unwanted complexity. Can this be an option? Like yes, consider the dataset name? Or no, discard dataset name. I can see myself in both situations... Is the driver read only? If not, what will happen when you try the reverse? From PgSql to fgdb? Duarte -Mensagem original- De: Paul Ramsey [mailto:pram...@cleverelephant.ca] Enviada: segunda-feira, 30 de Maio de 2011 22:47 Para: gdal-dev@lists.osgeo.org Assunto: [gdal-dev] Table Names (FGDB) So, my goal is to map information from FGDB to PostGIS and as much fidelity as possible. FGDB includes a class called Feature Dataset which is basically a folder that holds Feature Class objects, which map directly to OGR layers. So the Feature Dataset then acts a good deal like a schema in PostGIS. I've noticed that the PgSQL driver supports a SCHEMA keyword to allow you to write to a schema without schema-qualified table names, and that it also notices when table names are schema qualified and puts them in the appropriate place. So the name of a layer read from a schema would show up in OGR as schemaname.tablename. In FGDB, we have the path of a Feature Class or Table, which looks like this \FeatureDataset\FeatureClass. Similar to the schema qualification, yet different. As it stands right now, the FeatureDataset portion of the path is discarded in the public API, so that the OGR layer name is just the FeatureClass, with no reference to the FeatureDataset. That makes it hard to create a mapping from FGDB (\FeatureDataset\FeatureClass) to PostGIS (FeatureDataset.FeatureClass). The FGDB implementation currently returns GeoDatabase.GetQueryName() as the layer name, which Gets the query name (the name to use in SQL statements) of a table based on its path. If it returned the path name, then the FeatureDataset qualification information could be preserved in other contexts. On the other hand, perhaps the QueryName is more useful to more users than the path? It looks like the QueryName is expected to be unique, so probably FeatureClass and Table names have to be unique regardless of what FeatureDataset they appear in (true?). So my options are: (a) - change the current OGR layer name to the path name (b) - change the current OGR layer name to a schema.table analogue and handle appropriately (c) - leave the current OGR layer name as is (tablename regardless of containing folders) and make a FGDB-only method for accessing the pathname for my particular purposes Preferences? P. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] ArcSDE: connection times too long
Hi listers. I'm seing a very long delay while connecting to ArcSDE. Even if I just do ogrinfo on a single layer I still have to wait a few minutes before I get anything back. Any parameter I can use? Thanks, Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] ArcSDE: connection times too long
Hi Frank. I do that already: ogrinfo -so -ro SDE:server,5151,,user,pdw user.featureclass Notice that I separate the layer name with a blank space. Using a comma does not work for me (wonder if the docs are right on that??). I still wait quite a bit. This is a mid-sized db, with some 1200 feature classes... Duarte -Mensagem original- De: Frank Warmerdam [mailto:warmer...@pobox.com] Enviada: terça-feira, 12 de Abril de 2011 15:47 Para: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] ArcSDE: connection times too long On 11-04-12 10:33 AM, Duarte Carreira wrote: Hi listers. I'm seing a very long delay while connecting to ArcSDE. Even if I just do ogrinfo on a single layer I still have to wait a few minutes before I get anything back. Any parameter I can use? Duarte, The SDE driver docs at http://www.gdal.org/ogr/drv_sde.html says: If the layer parameter is specified then the SDE driver is able to skip reading the summary metadata for each layer; skipping this step can be a significant time savings. So try listing the layer of interest in the connection string. SDE:server,instance,database,username,password,layer Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] ArcSDE: connection times too long
That never occurred to me! Ok, so I measured 48secs sharp to get ogrinfo results, by including the featureclass name in the connection and after that too (like you pointed out). Then got 45secs with just the name in the connection, and got all layers listed. Don't know what to make of this... Duarte -Mensagem original- De: Frank Warmerdam [mailto:warmer...@pobox.com] Enviada: terça-feira, 12 de Abril de 2011 17:32 Para: Duarte Carreira Cc: gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] ArcSDE: connection times too long On 11-04-12 12:12 PM, Duarte Carreira wrote: Hi Frank. I do that already: ogrinfo -so -ro SDE:server,5151,,user,pdw user.featureclass Notice that I separate the layer name with a blank space. Using a comma does not work for me (wonder if the docs are right on that??). I still wait quite a bit. This is a mid-sized db, with some 1200 feature classes... Duarte, In your case you are passing user.featureclass as the layer name to ogrinfo telling it that you only want to report on that layer. But to avoid opening all layers, you should also include it in the connection string. eg. ogrinfo -so -ro SDE:server,5151,,user,pdw,user.featureclass user.featureclass Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] external overviews format
What is the format of external overviews? Is the .ovr file in the same format as the original image? Or is it TIFF format, unless you choose Imagine format? I'm supposing that internal overviews are in the same format as the original image? Thanks, Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] new DWG specifications made public by ODA
I haven't seen this anywhere else so maybe it went unnoticed... Announcement here: http://www.opendesign.com/ODANEWS020610 Download here: http://www.opendesign.com/guestfiles It seems the document is open to all... not just members... Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] Re: ESRI file geodatabase support
Ragi, That is a good point, as always. Unfortunately I am not able to do that, but I can help in any other way... The zigGIS driver (I think ) is for reading/writing PostGIS using ArcMap, without having ArcSDE in the middle. As far as I know it does not read File geodatabases... I think for now you would have to reverse engineer the way fgdb's are written... given the effort, the discussion on whether to invest the time on fgbd or invest it on SL is at least interesting... and even fun for some of us. Duarte De: Ragi Burhum [mailto:r...@burhum.com] Enviada: quinta-feira, 17 de Junho de 2010 21:57 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira; Eric Wolf; Peter J Halls; Matt Wilkie Assunto: RE: [gdal-dev] Re: ESRI file geodatabase support From: Duarte Carreira dcarre...@edia.ptmailto:dcarre...@edia.pt Subject: RE: [gdal-dev] Re: ESRI file geodatabase support To: Eric Wolf ebw...@gmail.commailto:ebw...@gmail.com, Peter J Halls p.ha...@york.ac.ukmailto:p.ha...@york.ac.uk Cc: gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org, Matt Wilkie map...@gmail.commailto:map...@gmail.com Well, if SpatiaLite offers some proper benefits and disseminates through all of the FOSS world, then it may get a strong enough push even for ESRI to pick it up. It happened before... (kml?) If SL would: 1) Be as fast as shapefile in production settings, desktop and webgis 2) Offer SQL support, spatial and otherwise, also through desktop tools like QGIS 3) Allow editing while serving (even if for 1 editor only) 4) Better support in QGIS than for shapefile (take advantage of Spatial SQL, all other functionality) 5) Same for MapServer, GeoServer, gvSIG, et al. 6) Allow easy managing of rasters inside the .db file, through QGIS 7) ??more ideas/requests?? Then it would be a very, very good contender... and the ball would be kicked to the other side. And it seems we're already there for some of the listed features. Duarte Or even better, instead of waiting and complaining, we could just write a GeoDatabase OGR (ESRI) Workspace ourselves. There are already examples of working ones out there http://svn.obtusesoft.com/core/trunk/ So instead of hoping and pleading for support, someone should just sit down and write it. I started one at one point (C++), got side tracked with other things. If anyone is interested in that source code, I would be happy to share that too. - Ragi ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] Re: ESRI file geodatabase support
Re-reading your email it seems I misunderstood what you meant... So using the zigGIS ArcMap/PostGIS provider one could adapt it to read OGR datasources, like QGIS does. You would then access SpatiaLite and potentially all other OGR formats from ArcMap. Good idea. Duarte De: Ragi Burhum [mailto:r...@burhum.com] Enviada: quinta-feira, 17 de Junho de 2010 21:57 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira; Eric Wolf; Peter J Halls; Matt Wilkie Assunto: RE: [gdal-dev] Re: ESRI file geodatabase support From: Duarte Carreira dcarre...@edia.ptmailto:dcarre...@edia.pt Subject: RE: [gdal-dev] Re: ESRI file geodatabase support To: Eric Wolf ebw...@gmail.commailto:ebw...@gmail.com, Peter J Halls p.ha...@york.ac.ukmailto:p.ha...@york.ac.uk Cc: gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org, Matt Wilkie map...@gmail.commailto:map...@gmail.com Well, if SpatiaLite offers some proper benefits and disseminates through all of the FOSS world, then it may get a strong enough push even for ESRI to pick it up. It happened before... (kml?) If SL would: 1) Be as fast as shapefile in production settings, desktop and webgis 2) Offer SQL support, spatial and otherwise, also through desktop tools like QGIS 3) Allow editing while serving (even if for 1 editor only) 4) Better support in QGIS than for shapefile (take advantage of Spatial SQL, all other functionality) 5) Same for MapServer, GeoServer, gvSIG, et al. 6) Allow easy managing of rasters inside the .db file, through QGIS 7) ??more ideas/requests?? Then it would be a very, very good contender... and the ball would be kicked to the other side. And it seems we're already there for some of the listed features. Duarte Or even better, instead of waiting and complaining, we could just write a GeoDatabase OGR (ESRI) Workspace ourselves. There are already examples of working ones out there http://svn.obtusesoft.com/core/trunk/ So instead of hoping and pleading for support, someone should just sit down and write it. I started one at one point (C++), got side tracked with other things. If anyone is interested in that source code, I would be happy to share that too. - Ragi ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] Re: ESRI file geodatabase support
Just a quick note: SQLite is supported through the FME add-on aka Interoperability Extension to ArcMap. Duarte -Mensagem original- De: Peter J Halls [mailto:p.ha...@york.ac.uk] Enviada: sexta-feira, 18 de Junho de 2010 14:49 Para: Duarte Carreira Cc: Ragi Burhum; gdal-dev@lists.osgeo.org; Eric Wolf; Matt Wilkie Assunto: Re: [gdal-dev] Re: ESRI file geodatabase support And ... I think I may have found the file geodatabase documentation promised ... in the online ArecGIS 10 documentation, Administrator Library, Architecture of a geodatabase, Geodatabase XML. It would appear that, in addition to the XML Schema they published several years ago, the XML documentation has been augmented with a view to interoperability being achieved at the XML level. So, maybe no, they have not published the binary structure but they may have published enough to enable XML exchange. I've not read all the material - and I would need to think about much of it before I understood it. Whether there is enough there I cannot say ... I was looking at http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html Best wishes, Peter Duarte Carreira wrote: Ragi, That is a good point, as always. Unfortunately I am not able to do that, but I can help in any other way... The zigGIS driver (I think ) is for reading/writing PostGIS using ArcMap, without having ArcSDE in the middle. As far as I know it does not read File geodatabases... I think for now you would have to reverse engineer the way fgdb's are written... given the effort, the discussion on whether to invest the time on fgbd or invest it on SL is at least interesting... and even fun for some of us. Duarte De: Ragi Burhum [mailto:r...@burhum.com] Enviada: quinta-feira, 17 de Junho de 2010 21:57 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira; Eric Wolf; Peter J Halls; Matt Wilkie Assunto: RE: [gdal-dev] Re: ESRI file geodatabase support From: Duarte Carreira dcarre...@edia.ptmailto:dcarre...@edia.pt Subject: RE: [gdal-dev] Re: ESRI file geodatabase support To: Eric Wolf ebw...@gmail.commailto:ebw...@gmail.com, Peter J Halls p.ha...@york.ac.ukmailto:p.ha...@york.ac.uk Cc: gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org, Matt Wilkie map...@gmail.commailto:map...@gmail.com Well, if SpatiaLite offers some proper benefits and disseminates through all of the FOSS world, then it may get a strong enough push even for ESRI to pick it up. It happened before... (kml?) If SL would: 1) Be as fast as shapefile in production settings, desktop and webgis 2) Offer SQL support, spatial and otherwise, also through desktop tools like QGIS 3) Allow editing while serving (even if for 1 editor only) 4) Better support in QGIS than for shapefile (take advantage of Spatial SQL, all other functionality) 5) Same for MapServer, GeoServer, gvSIG, et al. 6) Allow easy managing of rasters inside the .db file, through QGIS 7) ??more ideas/requests?? Then it would be a very, very good contender... and the ball would be kicked to the other side. And it seems we're already there for some of the listed features. Duarte Or even better, instead of waiting and complaining, we could just write a GeoDatabase OGR (ESRI) Workspace ourselves. There are already examples of working ones out there http://svn.obtusesoft.com/core/trunk/ So instead of hoping and pleading for support, someone should just sit down and write it. I started one at one point (C++), got side tracked with other things. If anyone is interested in that source code, I would be happy to share that too. - Ragi -- Peter J Halls, GIS Advisor, University of York Telephone: 01904 433806 Fax: 01904 433740 Snail mail: Computing Service, University of York, Heslington, York YO10 5DD This message has the status of a private and personal communication ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] Re: ESRI file geodatabase support
Well, I'm sold. What you need to do it?? Duarte De: Ragi Burhum [mailto:r...@burhum.com] Enviada: sexta-feira, 18 de Junho de 2010 16:31 Para: Peter J Halls Cc: Duarte Carreira; gdal-dev@lists.osgeo.org; Eric Wolf; Matt Wilkie Assunto: Re: [gdal-dev] Re: ESRI file geodatabase support This is meant to be an export/import type of thing. So you won't be able to open something in ArcMap and start manipulating it live using the ArcMap/ArcCatalog tools. The other approach I am proposing would be OGR as a seamless datasource that integrates with ArcGIS at the data access layer level. It would be transparent to the other tools. - Ragi On Fri, Jun 18, 2010 at 5:49 AM, Peter J Halls p.ha...@york.ac.ukmailto:p.ha...@york.ac.uk wrote: And ... I think I may have found the file geodatabase documentation promised ... in the online ArecGIS 10 documentation, Administrator Library, Architecture of a geodatabase, Geodatabase XML. It would appear that, in addition to the XML Schema they published several years ago, the XML documentation has been augmented with a view to interoperability being achieved at the XML level. So, maybe no, they have not published the binary structure but they may have published enough to enable XML exchange. I've not read all the material - and I would need to think about much of it before I understood it. Whether there is enough there I cannot say ... I was looking at http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html Best wishes, Peter ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] Re: ESRI file geodatabase support
Well, if SpatiaLite offers some proper benefits and disseminates through all of the FOSS world, then it may get a strong enough push even for ESRI to pick it up. It happened before... (kml?) If SL would: 1) Be as fast as shapefile in production settings, desktop and webgis 2) Offer SQL support, spatial and otherwise, also through desktop tools like QGIS 3) Allow editing while serving (even if for 1 editor only) 4) Better support in QGIS than for shapefile (take advantage of Spatial SQL, all other functionality) 5) Same for MapServer, GeoServer, gvSIG, et al. 6) Allow easy managing of rasters inside the .db file, through QGIS 7) ??more ideas/requests?? Then it would be a very, very good contender... and the ball would be kicked to the other side. And it seems we're already there for some of the listed features. Duarte De: Eric Wolf [mailto:ebw...@gmail.com] Enviada: quinta-feira, 17 de Junho de 2010 17:33 Para: Peter J Halls Cc: gdal-dev@lists.osgeo.org; Matt Wilkie Assunto: Re: [gdal-dev] Re: ESRI file geodatabase support Matt's chicken-and-egg point seems dead-on. Except that Jack Dangermond abhors a vacuum and ESRI has been focused on higher-order issues than file formats. They are trying to provide topological constraints in the database (or file) and things like geometric networks (which are really just a set of topologically consistent linear features). The good ol' Shapefile doesn't even come close to cutting the mustard. ArcSDE imposes these on other RDBMS. The Personal Geodatabases did it in MDBs. And the File Geodatabase does it without dependency on Microsoft Jet. Another way to look at the open spec issue (which echoes ESRIs sentiment) is that it's rather easy to screw up topology constraints. As Peter mentioned, SDE sometimes doesn't like Oracle tables created by GDAL. The format of the tables may be fine - but the topological relations may not be right. I'm betting ESRI created File Geodatabase mainly to get away from the Personal Geodatabase because it was too easy to muck with the MDB in Access and screw up the higher order relationships. File Geodatabases, like Shapefiles and Personal Geodatabases, are intended as a means to exchange data. You export your data from ArcSDE into one of these formats and give it to someone to use. Shapefiles are stripped of topology. Personal Geodatabases only really work on platforms Microsoft supports. File Geodatabases are the next logical step. SpatialLite seems like a really strong contender. How do we get ESRI to play along? -=--=---===---=--=-=--=---==---=--=-=- Eric B. WolfNew! 720-334-7734 USGS Geographer Center of Excellence in GIScience PhD Student CU-Boulder - Geography GPG Public Key: http://www.h4h.net/ebwolf.public.key.txt On Thu, Jun 17, 2010 at 1:21 AM, Peter J Halls p.ha...@york.ac.ukmailto:p.ha...@york.ac.uk wrote: Another way to achieve interoperability is via a DBMS for which there is an SDE implementation, although this may not be appropriate for Matt Wilkie's requirements. It may not be always as easy as with shapefiles but it does not have the limitations. Having said that, in our Oracle environment I do have a problem getting SDE to recognise some spatial datasets created with GDAL but have yet to prove what is happening to cause this and so cannot point a finger of blame in any direction ... Best wishes, Peter Duarte Carreira wrote: Matt, the only reason I have seen presented for some reluctance in pushing spatialite as a de facto standard following shapefile's success, is not having a foothold in the closed source sector. That's the only thing ESRI's fgdb could potentially offer, since the extra data types supported will not be available outside ESRI's software (Terrain, Topology, Networks, etc.). (As for interoperability with ESRI, its users can always export to shapefile. Ofcourse I would prefer to directly read fgdb data but if not possible it's ok too.) So the question is: is it true that for a new universal spatial format to be born it has to have at least read support in the closed source world? Duarte -Mensagem original- De: Matt Wilkie [mailto:map...@gmail.commailto:map...@gmail.com] Enviada: terça-feira, 15 de Junho de 2010 22:52 Para: gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org Assunto: [gdal-dev] Re: ESRI file geodatabase support I think discussing a shapefile successor, or even perhaps a code sprint, is a very good topic for FOSS4G. This same thread that we're weaving now is/has happened on a number mailing lists and usually generated dozens of responses each time. The interest is clear. From my vantage the germinating seed crystal could be spatialite, but there seems to be some general reluctance to jump on board. I'm ignorant of the reasons for that, perhaps that will come out at FOSS4G; wish I could be there! Ivan: I personally welcome and will use a gdal/ogr
RE: [gdal-dev] Re: ESRI file geodatabase support
Matt, the only reason I have seen presented for some reluctance in pushing spatialite as a de facto standard following shapefile's success, is not having a foothold in the closed source sector. That's the only thing ESRI's fgdb could potentially offer, since the extra data types supported will not be available outside ESRI's software (Terrain, Topology, Networks, etc.). (As for interoperability with ESRI, its users can always export to shapefile. Ofcourse I would prefer to directly read fgdb data but if not possible it's ok too.) So the question is: is it true that for a new universal spatial format to be born it has to have at least read support in the closed source world? Duarte -Mensagem original- De: Matt Wilkie [mailto:map...@gmail.com] Enviada: terça-feira, 15 de Junho de 2010 22:52 Para: gdal-dev@lists.osgeo.org Assunto: [gdal-dev] Re: ESRI file geodatabase support I think discussing a shapefile successor, or even perhaps a code sprint, is a very good topic for FOSS4G. This same thread that we're weaving now is/has happened on a number mailing lists and usually generated dozens of responses each time. The interest is clear. From my vantage the germinating seed crystal could be spatialite, but there seems to be some general reluctance to jump on board. I'm ignorant of the reasons for that, perhaps that will come out at FOSS4G; wish I could be there! Ivan: I personally welcome and will use a gdal/ogr that uses the currently installed arcgis libraries however for the health of the industry I'd like to see unencumbered access. Thanks for letting me know at least part of my ramblings are of interest cheers, -matt -- View this message in context: http://osgeo-org.1803224.n2.nabble.com/gdal-dev-ESRI-file-geodatabase-support-tp5159756p5183957.html Sent from the GDAL - Dev mailing list archive at Nabble.com. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] Open source vector geoprocessing libraries?
Jason, Have you looked at GeoKettle [1]? And recently I found GearScape [2], which seemed very interesting to me. Though neither is based on python... Duarte Carreira [1] - http://sourceforge.net/projects/geokettle/ [2] - http://www.fergonco.es/gearscape/index.php De: Emilio Mayorga [emiliomayo...@gmail.com] Enviado: terça-feira, 12 de Janeiro de 2010 18:25 Para: Jason Roberts Cc: gdal-dev Assunto: Re: [gdal-dev] Open source vector geoprocessing libraries? Hi Jason, This may not be quite what you have in mind, but check out the PySAL (Open Source Python Library for Spatial Analytical Functions) project: http://geodacenter.asu.edu/pysal I've never used it, and have only looked at a recent presentation (http://conference.scipy.org/static/wiki/rey_pysal.pdf). It's not clear that it includes or even aims to include the traditional spatial operators provided by GEOS. I also have no idea if it uses OGR for its vector data access. But the developers have done some terrific work in spatial analysis tools in the past. BTW, I'd love to see your marine spatial ecology tools moved to an open source, platform neutral code base! Cheers, -Emilio Mayorga Applied Physics Laboratory University of Washington Box 355640 Seattle, WA 98105-6698 USA On Mon, Jan 11, 2010 at 2:32 PM, Jason Roberts jason.robe...@duke.edu wrote: Dear geospatial software experts, By integrating with GEOS, OGR can perform various spatial operations on individual geometries, such as buffer, intersection, union, and so on. Is there a library that efficiently performs these kinds of operations on entire OGRLayers? For example, this library would have functions that would buffer all of the features in a layer, or intersect all of the features in one layer with all of those in another. Basically, I am looking for an open source technology that replicates the geoprocessing tools found in ArcGIS and other GIS packages. These tools traditionally operate on one or more layers as input and produce one or more layers as output. If such a library does not exist, does the OGR team envision that they might add such capabilities to OGR in the future? From software design and performance points of view, would it be appropriate to extend OGR to include functions for spatial operations on entire layers, or is this best left to other libraries? I can see rudimentary ways to implement such tools (e.g. for intersecting layers: loop over all features in both layers, calling OGRGeometry::Touches on all combinations, or something similar). But I am not a geometry expert and do not know if OGRLayer's cursor-based design is compatible with such capabilities; I do not know about spatial indexing, for example. I develop open source geoprocessing tools that help with spatial ecology problems. At the moment, my tools depend on heavily on ArcGIS for these operations with vector layers. I would like to remove this dependency, and, if possible, develop a toolbox that exposes the same ecology tools to several GIS packages. Many GIS packages, such as ArcGIS, QGIS, MapWindow, and OpenJump, support plugin extensions. I am wondering whether how difficult it would be to develop a package of tools that does not depend on a specific GIS package but exposes them to several packages via the package-specific plugin mechanisms. For this to work, I'd have to find a library that can do the kind of geoprocessing with layers that ArcGIS can do, or write my own. Writing it myself sounds daunting and am hoping that there are existing projects to draw from. Thank you very much for any comments you can provide. Jason ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] Fastest vector format for combining shapefiles
Hello. I've been following this discussion with much interest. Spatialite really does standout to me as the mostly done format... as to ESRI interoperability, it will be easy to make a small tool/model for ESRI users that runs OGR in the background, so a simple checkout-checkin process can be achieved without much effort. If no one else builds it, I can make time for that. On the other hand it doesn't seem healthy to me that an OS GIS format should be tossed just because ESRI doesn't read it yet. It will if the format takes off the ground. (although implementing news formats in ArcGIS is difficult as can be seen by ziggis' example, and ESRI is not best known for including new formats) Just my 2cents. Duarte Carreira De: doug_newc...@fws.gov [doug_newc...@fws.gov] Enviado: segunda-feira, 19 de Outubro de 2009 13:59 Para: Chris Barker Cc: gdal-dev@lists.osgeo.org; gdal-dev-boun...@lists.osgeo.org Assunto: Re: [gdal-dev] Fastest vector format for combining shapefiles I'm confused, I thought spatialite support was being added to ogr in 1.7 . http://www.gdal.org/ogr/drv_sqlite.html As to ESRI, I asked at the last DOI meeting prior to the ESRI conference this year about spatialite support and they said they had no plans to support it. Of course, they used to say the same thing about postgresql/postgis and if spatialite is implimented in ogr it will probably be included down the road. Doug Doug Newcomb USFWS Raleigh, NC 919-856-4520 ext. 14 doug_newc...@fws.gov - The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life is too short for undocumented, proprietary data formats. [cid:1__=0ABBFCC7DFD59B838f9e8a93df938690@fws.gov]Chris Barker chris.bar...@noaa.gov Chris Barker chris.bar...@noaa.gov Sent by: gdal-dev-boun...@lists.osgeo.org 10/18/2009 11:42 PM To gdal-dev@lists.osgeo.org gdal-dev@lists.osgeo.org cc Subject Re: [gdal-dev] Fastest vector format for combining shapefiles Simon Greener wrote: What else is there to do for its adoption as the new shapefile for the open source community? I heard someone say once that a specification isn't a standard unless there is more than one implementation (preferably more). Is there any other way to read or write a spatialite file than sqlite itself? It's also not a standard without a specification -- is there one (other than the source code, which, of course is the only truly accurate specification). Also -- do ESRI tools read or write it? Like it or not, they are the elephant in the room. My ESRI-using colleagues want to distribute data in the ESRI geodatabase format -- I don't like that, as I don't use ESRI tools, but I can't suggest an alternative that won't work for them. Sigh. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev inline: graycol.gifinline: ecblank.gif___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] OGR + ArcSDE: support for non-spatial tables
Does anyone know if there's support for non-spatial tables in ArcSDE through OGR? For instance to make a join with a spatial table? Thanks, Duarte ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] ogr 1.6 and ArcSDE 9.3sp1: not reading varchar2 fields
Mike, That's a good idea, but not really well accepted in many cases... In my particular case I ended up changing my tables to varchar2... We'll have to wait for unicode support in ogr. Thanks, Duarte De: Michael Smith [michael.sm...@usace.army.mil] Enviado: sexta-feira, 27 de Março de 2009 10:22 Para: Duarte Carreira; gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] ogr 1.6 and ArcSDE 9.3sp1: not reading varchar2 fields Duarte, Why not create a view that has the TRANSLATE and TO_CHAR and then use that in OGR. Mike -- Michael Smith RSGIS Center ERDC - CRREL US Army Corps of Engineers On 3/27/09 5:44 AM, Duarte Carreira dcarre...@edia.pt wrote: I'm correcting the subject of the message - the problem is with NVARCHAR fields (not varchar2). It seems this relates to utf issues/limitations in ogr. If I could use SQL functions this could be solved by the database itself. Like using TO_CHAR(field_name) or TRANSLATE(field_name USING CS_CHAR). But it seems ogr does not support these sql functions/operators... Any additional suggestions? Thanks, Duarte -Mensagem original- De: Duarte Carreira [mailto:dcarre...@edia.pt] Enviada: quinta-feira, 26 de Março de 2009 23:54 Para: gdal-dev@lists.osgeo.org Assunto: [gdal-dev] ogr 1.6 and ArcSDE 9.3sp1: not reading varchar2 fields Hello. As always trying to coerce these two to work together nicely... The situation: exporting ArcSDE feature class to anything, shapefile, gpx... server: Oracle xe (10), ArcSDE 9.3 sp1, Win2003 Std x32 client: Vista x32, GDAL/OGR 1.6 build 1500 from Tamas site, sde dll's from 9.3 sp1 As you can see below, Oracle describes several fields as being number and nvarchar2, and ogrinfo only identifies numeric fields. All text fields come out as unknown. Now, what could this be due to? Anyone facing the same issue? I tried to use SQL to cast a nvarchar to character without success... Thanks, Duarte Feature class description in Oracle: Column Name Data Type NullableDefault Primary Key OBJECTIDNUMBER No - - IDPERC NVARCHAR2(6)Yes - - NOMENVARCHAR2(150) Yes - - DESCRICAO NVARCHAR2(255) Yes - - IDPONTO NVARCHAR2(10) Yes - - IDSIG NUMBER Yes - - XCOORD NUMBER(38,8)Yes - - YCOORD NUMBER(38,8)Yes - - MORADA_COMPLETA NVARCHAR2(255) Yes - - CONCELHONVARCHAR2(50) Yes - - FREGUESIA NVARCHAR2(50) Yes - - CODPOSTAL NVARCHAR2(10) Yes - - TELEFONENVARCHAR2(10) Yes - - DESCRICAOPT NVARCHAR2(255) Yes - - NOMEPT NVARCHAR2(150) Yes - - SHAPE ST_GEOMETRY Yes - - 1 - 16 ogrinfo output: Layer name: user.table Geometry: Unknown (any) ERROR 1: SE_layer_get_statistics: -51/Underlying DBMS error Feature Count: -1 Extent: (6177.862000, -123296.045000) - (63997.354000, -91245.397000) Layer SRS WKT: PROJCS[Datum_73_Hayford_Gauss_IPCC, GEOGCS[GCS_Datum_73, DATUM[Datum_73, SPHEROID[International_1924,6378388.0,297.0]], PRIMEM[Greenwich,0.0], UNIT[Degree,0.0174532925199433]], PROJECTION[Transverse_Mercator], PARAMETER[False_Easting,180.598], PARAMETER[False_Northing,-86.99], PARAMETER[Central_Meridian,-8.1319062], PARAMETER[Scale_Factor,1.0], PARAMETER[Latitude_Of_Origin,39.66], UNIT[Meter,1.0]] OBJECTID: Integer (10.0) IDPERC: (unknown) (6.0) NOME: (unknown) (50.0) DESCRICAO: (unknown) (50.0) IDLINHA: (unknown) (5.0) SHAPE_LENG: Real (38.8) ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] convert 2 shapefiles to a single GPX?
Now this I've got to try! The VRT file idea is intriguing. I didn't know it had this much flexibility... Thanks, Duarte -Mensagem original- De: Even Rouault [mailto:even.roua...@mines-paris.org] Enviada: sábado, 17 de Janeiro de 2009 20:43 Para: gdal-dev@lists.osgeo.org Cc: Duarte Carreira; Stefan Keller; Jean-Claude Repetto Assunto: Re: [gdal-dev] convert 2 shapefiles to a single GPX? Here's one solution : 1) create a directory, for example merge_shp 2) copy in merge_shp the shapefile that is going to be put in the waypoints layer as waypoints.shp (rename the .dbf and .shx accordingly) 3) copy in merge_shp the shapefile that is going to be put in the tracks layer as tracks.shp (rename the .dbf and .shx accordingly) 4) run ogr2ogr -f GPX merged.gpx merge_shp waypoints tracks (the order of the layers is important) A solution based on an OGR VRT file defining the two layers from the 2 shapefiles would also probably work. Le Thursday 15 January 2009 12:03:27 Duarte Carreira, vous avez écrit : Maybe the same approach can be used with 2 gpx files? That would be very good. I cannot merge 2 shapefiles with different geometry types. I could convert line shapegpx1 and point shape-gpx2 and then merge gpx2+gpx2-gpx3... Thanks, Duarte De: Stefan Keller [mailto:sfkel...@gmail.com] Enviada: quarta-feira, 14 de Janeiro de 2009 22:40 Para: Jean-Claude Repetto Cc: Duarte Carreira; gdal-dev@lists.osgeo.org Assunto: Re: [gdal-dev] convert 2 shapefiles to a single GPX? A merge of two shapefiles 'file1.shp' and 'file2.shp' into a new file 'file_merged.shp' is performed like this (See http://www.gdal.org/ogr/drv_shapefile.html): % ogr2ogr file_merged.shp file1.shp % ogr2ogr -update -append file_merged.shp file2.shp -nln file_merged Then convert to GPX: % ogr2ogr -f GPX out.gpx file_merged.shp Stefan http://geoconverter.hsr.chhttp://geoconverter.hsr.ch/ 2009/1/14 Jean-Claude Repetto jrepe...@free.frmailto:jrepe...@free.fr Duarte Carreira a écrit : Can OGR convert 2 shapefiles to a single GPX? The objective is to add tracks from a line shapefile and waypoints from a point shapefile... Hello, I don't know the answer to your question, but since GPX files are text files, it is very easy to merge two GPX files with a text editor. HTH ! Jean-Claude ___ gdal-dev mailing list gdal-dev@lists.osgeo.orgmailto:gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] convert 2 shapefiles to a single GPX?
Can OGR convert 2 shapefiles to a single GPX? The objective is to add tracks from a line shapefile and waypoints from a point shapefile... Thanks, Duarte Carreira ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] arcsde plugins for 1.6
Hi. I'm looking for 1.6 binaries for gdal and ogr plugins for ArcSDE and can't find any in FWTools nor OSGeo distributions... is there anywhere I could get these? Thanks. Duarte Carreira ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
RE: [gdal-dev] is there a way to rename fields with gdal/ogr 1.5.x?
Tamas, Thanks for your reply. The problem to using the latest FWTools is the missing SDE plugin... compiling it myself is behond my knowledge. That's why I was hoping for a workaround with 1.5.x. Do you suppose I could get a compiled copy?? Regards, Duarte De: Duarte Carreira [mailto:[EMAIL PROTECTED] Enviada: sexta-feira, 3 de Outubro de 2008 18:21 Para: gdal-dev@lists.osgeo.org Assunto: [gdal-dev] is there a way to rename fields with gdal/ogr 1.5.x? Hi all. I was trying to convert rename a shapefile and getting syntax errors with the -sql string... So I'm wondering if this is working in 1.5.x or only in 1.6dev? If it's not working in 1.5 is there any other way to accomplish this in this version? Thanks, Duarte Carreira ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
