RE: [OSGeo-Discuss] OGR and Open Design Alliance DWG-DXF
Thanks Frank, I'll take it from here. If I get stuck i'll holler. Roger Bedell, President Sylvan Ascent Inc. ro...@sylvanascent.com +34 626 855 662 -Original Message- From: Frank Warmerdam warmer...@pobox.com Sent: 17 February 2009 18:56 To: Roger Bedell ro...@sylvanascent.com; OSGeo Discussions discuss@lists.osgeo.org Subject: Re: [OSGeo-Discuss] OGR and Open Design Alliance DWG-DXF Roger Bedell wrote: Hi Frank, I found this: http://svn.osgeo.org/gdal/spike/dxfdwg/ which seems to be an OGR to DWG/DXF exporter using the ODA libs. Can you tell me what the status of this is? If it isn't active, can I pick it up and work with it a bit? Roger, This code was removed from GDAL trunk because a bit of it was derived directly from header files of the ODA libraries and so it was not properly licensed as open source. As of a year or two ago I believe it worked within the limited scope of what it was supposed to do - which was to export simple features DWG files from OGR datasources. Feel free to use the code, but keep in mind that the license provenance is flawed, though not necessarily insurmountable so. I'm not sure that this forum is the best place for additional discussion on technical details. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: Raster data on RDBMS
Well, now we get to the crux of the matter, what are the benefits? Let's analyze this a bit more to see if anything seems important. You mention: 1) Spatial Extension - not sure what this is, but maybe you can build image operations into the database. 2) Schemas - Schemas can be queried, put into views, massaged 3) Metadata - well that's always handy, and likely much easier to maintain and query in a database. 4) Georeferences definition - Right, like simple features georeferencing, built into the schema 5) Spatial Indexation - Could make tiles faster to retrieve, coupled with pyramiding, should be decent performance I'll add this: 6) As newer data comes in you can more easily upgrade a raster coverage, as the metadata (like the date) can be queried for the latest and greatest, while retaining the older stuff. This might be trickier in a file based system. and how about the usual rdbms stuff like 7) Replication - might be useful once in a while, esp in big systems 8) Scalability (not sure exactly if this is the word, but db vendors/OS projects have put a lot of effort into scaling over lots of users) 9) Backup 10) Transactions, possibly, if you are bringing raster data in from a satellite and something goes wrong? Unlikely to be to much of a benefit though. 11) Potentially more robust than using a file system More anyone? How about disadvantages, like 1) You have to import the raster data into the database. 2) Have to decide what format/projection/datum to use to store the data. 3) Possibly more storage is used, but these days who cares? 4) Tile edge effects (with most compression schemes, there is often a noticeable joint when joining two tiles) 5) Partial tiles - when you split up an image, it rarely fits perfectly into your chosen tile size. What do you do with the leftovers? More? Another 2 cents, Roger From: [EMAIL PROTECTED] on behalf of Lucena, Ivan Sent: Fri 10/31/2008 1:20 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Re: Raster data on RDBMS Paul, That is not the answer your are waiting for but... IMHO, once you overcome the mythical concept that a database server will always perform slower than a direct file access then Spatial is not special anymore! [who said that?] and you can think on the benefits just like a banker or an accounting bureau. Database servers in general are capable of making a good use of the available resources. For raster what is needed is a good BLOB support with cursor preferably. Spatial extension and schemas are indispensable accessories, they should provide metadata, georeferences definition, spatial indexation, etc. but they should not drag down the performance. Just my two cents. Ivan ---Original Message--- From: Paul Ramsey [EMAIL PROTECTED] Subject: Re: [OSGeo-Discuss] Re: Raster data on RDBMS Sent: Oct 31 '08 02:11 On Thu, Oct 30, 2008 at 5:25 PM, Gilberto Camara [EMAIL PROTECTED] wrote: but the benefits of having raster data on a DBMS are much more important. And those benefits are? ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss winmail.dat___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: Raster data on RDBMS
Gilberto, Sorry to be so ignorant, but can you give me a quick overview of how Terralib does this? Roger Bedell, President Sylvan Ascent Inc. 800-362-8971 +34 626 855 662 [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] www.sylvanascent.com http://www.sylvanascent.com/ www.topodepot.com http://www.topodepot.com From: [EMAIL PROTECTED] on behalf of Gilberto Camara Sent: Thu 10/30/2008 8:25 PM To: discuss@lists.osgeo.org Subject: [OSGeo-Discuss] Re: Raster data on RDBMS Dear all I would like to take a broader view of the issue of raster data in a DBMS. Issues of performance are relevant, but the benefits of having raster data on a DBMS are much more important. Consider that raster sensors and grid models are by far the dominant source of new geospatial data. If FOOS4G solutions do not include the capability of handling raster data in a DBMS, they would be lacking in functionality compared to commercial solutions from Oracle and ESRI. INPE´s FOSS4G developement of raster data on RDBMS using the TerraLib library is a tangible proof of concept. TerraAmazon (built using TerraLib) is INPE's OS solution for monitoring tropical forests operationally. The application was described in a recent article on the OSGEO journal, and it is arguably one of the biggest geospatial databases built using FOSS4G. Hundreds of images and hundreds of thousands of polygons are used operationally in Brazil´s real-time monitoring of deforestation. We hope our example helps to convince the community that we should not waste time arguing that we shouldn't store raster data in DBMS. FOOS4G needs this capability. Better yet, we already HAVE this capability on a production level. Best Regards Gilberto -- === Dr.Gilberto Camara Director General National Institute for Space Research (INPE) Sao Jose dos Campos, Brazil voice: +55-12-3945-6035 fax: +55-12-3921-6455 web: http://www.dpi.inpe.br/gilberto blog: http://techne-episteme.blogspot.com/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss winmail.dat___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Raster data on RDBMS
Mike and Ivan, I'd like to see them also compared to a caching solution, like GeoWebCache, or TileCache. These effectively create a file-based database of little bitty tiles at certain resolutions, kind of like a tile pyrimid that is created gradually over time as the image data is accessed. One would think the file-based cache system would be faster than a similar database solution, with the database solution giving no real benefits that I can see. I basically agreed with that but after seeing how fragile and complicated even a well defined structure of folders and files could be I would vote in favor of the good and old relational model. Since the cache is maintained by the software in a completely defined way, and never messed with by humans, I wonder what could go wrong? Roger Bedell, Sylvan Ascent Inc. 800-362-8971 +34 626 855 662 [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] From: [EMAIL PROTECTED] on behalf of Smith, Michael ERDC-CRREL-NH Sent: Wed 10/29/2008 10:13 AM To: Lucena, Ivan; OSGeo Discussions; Paul Ramsey Subject: Re: [OSGeo-Discuss] Raster data on RDBMS Ivan, Those numbers look impressive. We are just starting to set up some new hardware here and I plan to do some testing also. Perhaps we can collaborate and come up with a test suite in order to track these numbers across builds. Mike -- Michael Smith RSGIS Center ERDC - CRREL US Army Corps of Engineers On 10/29/08 1:35 AM, Lucena, Ivan [EMAIL PROTECTED] wrote: Paul, Good thought. Let's see. The default blocking used by the GeoRaster driver is (256, 256, 1). That is good because GeoTiffs doesn't tile on band space. So I would imagine that if I tiled the GeoTiff this way: [EMAIL PROTECTED]:~/Data time gdal_translate Barcelona_2007_R2C2.TIF Barcelona_2007_R2C2_tiled.TIF -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 Input file size is 14336, 14336 0...10...20...30...40...50...60...70...80...90...100 - done. real 0m42.991s user 0m20.289s sys 0m2.516s The comparison would be fair: [EMAIL PROTECTED]:~/Data time gdal_translate Barcelona_2007_R2C2_tiled.TIF out2.tif -srcwin 0 0 2000 2000 Input file size is 14336, 14336 0...10...20...30...40...50...60...70...80...90...100 - done. real 0m1.604s user 0m1.156s sys 0m0.444s What do you think? I would imagine that if I run gdaladdo to add Pyramids on the GeoRaster one application could take advantage of it by telling Oracle to cache the BLOB in memory. So the next time a user zoom-in the performance would be even better. I am trying to setup a mapserver experiment on that issue but for now I would like to keep my analysis on that very simple process of extracting a subset. Best regards, Ivan ---Original Message--- From: Paul Ramsey [EMAIL PROTECTED] Subject: Re: [OSGeo-Discuss] Raster data on RDBMS Sent: Oct 29 '08 05:00 The data is chunked in Oracle into tiles, so unless you tile the TIFF as well you aren't really doing a direct comparison. Even if you end up with the same numbers for both processes, I'll still be impressed, since I assumed Oracle would have a higher overhead. P. On Tue, Oct 28, 2008 at 9:54 PM, Lucena, Ivan [EMAIL PROTECTED] wrote: Hi There, I would like to return to a discussion that we had months ago about raster on RDBMS. But this time I would like to present some number. As long as I could recall there was basically two major arguments contrary to storing raster on RDBMS. One very pragmatical: Why waste precious process time with the overhead of dealing with queries, tables, client-sever back and forth just to get the data from BLOB fields on a database when you can get it directly from the file system?. The other argument was semantical: Why store raster on RDBMS if in general we are not expecting to have a transactions on that data? I cannot argue against the second one. I basically agreed with that but after seeing how fragile and complicated even a well defined structure of folders and files could be I would vote in favor of the good and old relational model. That is my experiment. I downloaded two free data samples from Naveteq website. Two geotiff files with the same size and number of bands (14336, 14336, 3): [EMAIL PROTECTED]:~/Data du -k Barcelona_2007_R2C2.TIF 602828 Barcelona_2007_R2C2.TIF [EMAIL PROTECTED]:~/Data du -k San_Francisco_2006_R1C2.TIF 602828 San_Francisco_2006_R1C2.TIF Then I loaded those images to Oracle Spatial GeoRaster using GDAL. The loading process is comparable than some commercial ETL products on the market. It took about 2 minutes to load each image. [EMAIL PROTECTED]:~/Data time gdal_translate -of georaster Barcelona_2007_R2C2.TIF georaster:scott,tiger,orcl,RDT_2$,2 Input file size is 14336, 14336 0...10...20...30...40...50...60...70...80...90...100 - done. Ouput dataset: (georaster:scott,tiger,orcl,RDT_2$,2) on GDAL_IMPORT,RASTER real 1m54.973s user 0m4.368s sys
RE: [OSGeo-Discuss] Raster data on RDBMS
Hi Frank, I'm not sure I completely agree about your apples/oranges, here's why: We are in the process of putting together a big raster server, so I'm evaluating the best way to proceed. I'm quite familiar with putting raster tiles in a database, did that way back in the last century. I've decided to use Geoserver, since we need (want) WFS-T and WCS, and the latest speed race between MapServer and GeoServer seems to be a tie more or less. As I see it we have several options: 1) Put the big original raster images in back of Geoserver, access using their Mosaic and the GDAL based ImageIO-ext. Advantage - easy, but kind of slow. 2) Take the originals and build a file-based pyramid Advantage - faster, but a lot of work, plus duplication and tricky to keep updated as new data comes in. 3) Take the originals and build a PostGIS based pyramid. Likely, about the same as 3 in speed and work and duplication. 4) Do 1, but put a pyramiding tileserver in front. It builds the pyrimid in 2 and 3 over time, and is likely the fastest if you hit the cache, and is no harder to do than #1. I think, that since the goal of all this storage of pyramids and the like is just to get speed, that they aren't apples/oranges, but apples apples, since they are both pyramid schemes, just in different places, either in front, or in back of the server. Roger Bedell, President Sylvan Ascent Inc. 800-362-8971 +34 626 855 662 [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] www.sylvanascent.com http://www.sylvanascent.com/ www.topodepot.com http://www.topodepot.com/ From: [EMAIL PROTECTED] on behalf of Frank Warmerdam Sent: Wed 10/29/2008 10:43 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Raster data on RDBMS Sylvan Ascent Inc. wrote: Mike and Ivan, I'd like to see them also compared to a caching solution, like GeoWebCache, or TileCache. These effectively create a file-based database of little bitty tiles at certain resolutions, kind of like a tile pyrimid that is created gradually over time as the image data is accessed. One would think the file-based cache system would be faster than a similar database solution, with the database solution giving no real benefits that I can see. Roger, While it might be educational to compare to a tilecache solution it is really comparing apples and oranges. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, [EMAIL PROTECTED] light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss winmail.dat___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] OS Raster server advice?
Hello everyone, We are looking at putting together a big, fast raster server for a Gis warehouse, and prefer open source. What would you use if you had to store 20TB data, serve any of it quickly to WMS or Google Earth, as well as being able to create any sort of output image in most any format/projection. I've used GDAL extensively in the past, but not sure it would be the best way to proceed, or how to store the data. I recently saw Rasdaman is going open source, a possibility? Also that GlobeXplorer used PostGIS (but how?) We also want WCS, but I'm not exactly sure what that even means. Any help would be greatly appreciated! Roger Bedell, President Sylvan Ascent Inc. 800-362-8971 +34 626 855 662 [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] www.sylvanascent.com http://www.sylvanascent.com/ www.topodepot.com http://www.topodepot.com/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
