[OSGeo-Discuss] Raster data on a DBMS
Dear OSGEO Jim Gray´s paper and much more on this issue is on his site at MS Research. Storing images on a database gives much more benefits that simple retrieval of metadata. Databases offer concurrency control, data protection, integrity and management features that simple file systems are lacking. If you have hundreds of images scattered around as files, you lack data management. Your metadata may point to a file that could have been deleted. In a multi-user environment, file systems do not prevent different users from updating the same image. The result may be a data which is inconsistent. Allow me to reiterate my earlier argument, which is that FOSS4G should **allow** users the option of storing raster data in a database. Storing images in a database is not recommended in each and every situation. The user should have the option, according to his needs. The current debate on whether images should be stored on an RDBMS reminds me of a similar debate during the early 90s, concerning whether vector data should be stored in an RDBMS. Remember the days of ARC-INFO? In mid 90s, our team at INPE tried to use the Postgres-95 RDBMS to store vector data. The result was a system with a very slow performance. The concept was right, but the implementation was lacking. It was only when PostgreSQL and PostGIS came of age that we could develop a multi-user spatial database with good performance. By the same argument, these are early days of storing raster data in RDBMS. There are missing features on the database and the performance may be slower than file systems. But the concept is fundamentally correct. I predict that five years hence this debate will be solved and we will look at it as a relique of the past. Best Regards Gilberto Christopher Schmidt said: I don't see anything in that paragraph that indicates that storing the *image data* in the database is important. (A link to the paper online or something could change that, of course.) Specifically, I don't think there's any doubt that if you have many-many files, it makes sense to store the *queryable image information* -- things like spatial extent, temporal extent, etc. -- belong in a database. The question is, in the "data" column, do you store a File Path, or the Image Data? Until/Unless databases get/have image manipulation tools directly, I can't see the value of storing the image data itself in the database. The points above argue against file-system based metadata storage/retrieval: sorting files by date, searching through index files, etc., so far as I can tell, but I don't see a compelling argument for image data in the database above. Of course, this is assuming that the image data access pattern is the same "in the database" and "on disk": for example, storing GeoTIFF data, then using GDAL to parse the string from the database as a GeoTIFF file. If the database you're using has a different (faster) Image access algorithm, then of course there can be benefits. However, those same benefits could presumably be realized with sufficiently complete libraries for accessing the image externally: If Oracles' Database product, for example, internally tiles the image, and they had a library to access the image in the same way, presumably you could store those bits on disk as well. However, if that library depends internally on a database, then integration of all points into the same database might help in some ways. In any case, I think there's obvious reasons to store your image metadata in a database -- and *using the same tools for accessing the images*, I don't think we've yet seen a compelling argument for storing image blobs in the database. Of course, all things are not equal :) If your database has built in MrSID support, for example, you could imagine using Database Storage for Images, because you'd get the automatic compression combined with the querying -- but that's not about the Database Specifically, just the image storage/reading library that comes along with it. Regards, -- Christopher Schmidt Web Developer -- === Dr.Gilberto Camara Director General National Institute for Space Research (INPE) Sao Jose dos Campos, Brazil voice: +55-12-3945-6035 fax: +55-12-3921-6455 web: http://www.dpi.inpe.br/gilberto blog: http://techne-episteme.blogspot.com/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] Benefits raster data on RDBMS
Dear all
Concerning the benefits of having raster data
stored together with vector data in a spatial
database, let me first quote from an excellent
paper from the late Jim Gray
("Scientific Data Management in the Coming Decade"):
"What’s wrong with files?
Everything builds from files as a base. HDF uses files.
Database systems use files. But, file systems have no
metadata beyond a hierarchical directory structure and file
names. They encourage a do-it-yourself- data-model that
will not benefit from the growing suite of data analysis
tools. They encourage do-it-yourself-access-methods that
will not do parallel, associative, temporal, or spatial
search. They also lack a high-level query language.
Lastly, most file systems can manage millions of files, but
by the time a file system can deal with billions of files, it
has become a database system."
In other words, if you have substantial amounts of raster
data (as is increasingly the case in geospatial application),
you will need to develop a significant amount of software
to manage your files. Unless... your data is handled by a
raster-enabled spatial database.
Best Regards
Gilberto
--
===
Dr.Gilberto Camara
Director General
National Institute for Space Research (INPE)
Sao Jose dos Campos, Brazil
voice: +55-12-3945-6035
fax: +55-12-3921-6455
web: http://www.dpi.inpe.br/gilberto
blog: http://techne-episteme.blogspot.com/
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] Re: Raster data on RDBMS
Dear all I would like to take a broader view of the issue of raster data in a DBMS. Issues of performance are relevant, but the benefits of having raster data on a DBMS are much more important. Consider that raster sensors and grid models are by far the dominant source of new geospatial data. If FOOS4G solutions do not include the capability of handling raster data in a DBMS, they would be lacking in functionality compared to commercial solutions from Oracle and ESRI. INPE´s FOSS4G developement of raster data on RDBMS using the TerraLib library is a tangible proof of concept. TerraAmazon (built using TerraLib) is INPE's OS solution for monitoring tropical forests operationally. The application was described in a recent article on the OSGEO journal, and it is arguably one of the biggest geospatial databases built using FOSS4G. Hundreds of images and hundreds of thousands of polygons are used operationally in Brazil´s real-time monitoring of deforestation. We hope our example helps to convince the community that we should not waste time arguing that we shouldn't store raster data in DBMS. FOOS4G needs this capability. Better yet, we already HAVE this capability on a production level. Best Regards Gilberto -- === Dr.Gilberto Camara Director General National Institute for Space Research (INPE) Sao Jose dos Campos, Brazil voice: +55-12-3945-6035 fax: +55-12-3921-6455 web: http://www.dpi.inpe.br/gilberto blog: http://techne-episteme.blogspot.com/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] Re: scale of FOSS projects
Dear all [EMAIL PROTECTED] stated: (...) > In the past i've heard it suggested that really successful open source > projects now need serious organisational backing. They can't be built > by a network of partly-funded enthusiast contributors alone. (...) > If this is inevitable, why? Is innovation less possible outside the > "enterprise"? Is this even a FOSS problem or a computing-in-the-broad > one? As one of the list members who argued in favour of serious organizational backing for OSS, let me throw my ideas on the issue: (a) True innovation is extremely hard in any field. Companies and governments worldwide aim at promoting and producing innovation, but breakthroughs come slowly and the winners are always a happy few. (b) To have a software development that is at the same time innovative and cooperative is even more difficult. Cooperation requires shared conceptualizations. This is much easier to achieve when the aim is to reproduce an existing design. This is the case of OSGEO projects that aim to have an open source version of OGC specifications. (c) Many innovations are produced at academic institutions. Most of those institutions have no incentive nor mission to support open-source development projects. Taking these innovations out of academia and giving them institutional support (private or public) is a way to ensuring these innovations are exposed to the market. Those with real value will survive. (d) For better or worse, the GIS arena is currently OGC-driven. OGC has levelled the market, by producing a set of common specifications, that both OS and proprietary systems must adhere to. By nature, standards bodies tend to stifle innovation. OGC has helped us make a lot of progress on vector-based GIS and web services. By contrast, OGC has reduced the motivation for innovation in issues such as spatial analysis, raster-based GIS, semantics, visualization, interfaces, and spatio-temporal models. (e) Our current dilemma is that almost all FOSS4G products are focused on OGC-compliance. This reduces the potential for innovation and generates very similar products. Thus, innovation in GIS is likely to come from outside the OGC-compliance focus that pervades our community. We need new interface paradigms, new ways of interacting in with mobile devices, new ways of modelling environmental change. Someone, somewhere, might be working on these innovations. I hope that it evolves it an open source product. Best Regards Gilberto P.S. For those who are interested, may I immodestly suggest some readings on the topic: G. Camara, F. Fonseca, "Information Policies and Open Source Software in Developing Countries." JASIST, vol 58(1):121-132, January 2007. http://www.dpi.inpe.br/gilberto/papers/camara_fonseca_jasist.pdf G.Camara, H. Onsrud, "Open Source GIS Software: Myths and Realities." In: Julie M. Esanu and Paul F. Uhlir, Eds, Open Access and the Public Domain in Digital Data and Information for Science: Proceedings of an International Symposium. Washington, The National Academies Press, 2004. http://www.dpi.inpe.br/gilberto/papers/camara_open_source_myths.pdf G. Câmara et al., “TerraLib: An open-source GIS library for large-scale environmental and socio-economic applications”. In: Brent Hall (ed), “Open Source Approaches to Spatial Data Handling”. Berlin, Springer, 2008. http://www.terralib.org/docs/papers/TerraLib-OSBook-versionJanuary2008.pdf -- === Dr.Gilberto Camara Director General National Institute for Space Research (INPE) Sao Jose dos Campos, Brazil voice: +55-12-3945-6035 fax: +55-12-3921-6455 web: http://www.dpi.inpe.br/gilberto blog: http://techne-episteme.blogspot.com/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Image Handling in RDBMS
Dear Ivan A GDAL driver for a TerraLib raster is a very welcome development. This driver will allow FOSS4G folks to experiment with their image datasets and reach their own conclusions about the issue. Additionally, we are discussing internally on how to make our code available inside PostGIS. What happens currently is that our priority for 2008 is to make TerraLib OGC-compliant at all levels. This is a task we announced at the Victoria FOSS4G conference and that we hope to be finished before the end of the year. After this priority is carried out, we will consider other matters. Best Regards Gilberto Lucena, Ivan escreveu: Dear Gilberto, Unfortunately I could not follow the discussion since my last reply to Bruce. I spent more time on air than on land in the last three days. I believe that your message is a reply to Paul Ramsey message: http://lists.osgeo.org/pipermail/discuss/2008-February/003202.html. Am I correct? What I would like to reply to both is that there is a new open source initiative on http://trac.osgeo.org/*gdal*/browser/trunk/gdal/frmts/*terralib*, a GDAL driver for Terralib *raster*. It is in a very preliminary stage. So far it only returns gdalinfo queries but it will be available for all gdal tools and API soon. I hope that it will help to dismistify the raster-on-db issue at least a little bit. Best regards, Ivan Lucena -- === Dr.Gilberto Camara Director General National Institute for Space Research (INPE) Sao Jose dos Campos, Brazil voice: +55-12-3945-6035 fax: +55-12-3921-6455 web: http://www.dpi.inpe.br/gilberto blog: http://techne-episteme.blogspot.com/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] Image Handling in RDBMS
Dear OSGEO I also beg to differ with Paul about the number of cases where image data handling in RDBMS is useful or necessary. In corporate applications where vectors and raster live together, and data processing and editing operations are used, handling different types of data together in a database is convenient. Control is more important than performance. Our TerraAmazon system (presented in the latest FOSS4G conference) does this, and we handle large files without problems. For one-way image servers, on a write once, read mostly basis, then the case for handling raster data inside the DBMS is less clear. Since the output is an image delivered to the user, either as a file or as a visualization, DBMS features such as concurrency control and long transactions are not needed. Performance is more important than control in this case. Best regards, Gilberto -- === Dr.Gilberto Camara Director General National Institute for Space Research (INPE) Sao Jose dos Campos, Brazil ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] RE: Open Geo Database
Dear all Regarding the FOSS Geodatabase discussion, I would like to bring a different issue to the discussion. It concerns whether we should be concerned with broadening the number of FOSS databases that offer support for spatial data, or whether we should focus on deepening and increasing the support for different types of spatio-temporal data on PostGIS. Consider a situation where PostGIS would demonstrably handle more spatial data types than Oracle and would do so efficiently. This would provide a tremendous leverage for all FOSS4G applications and developers. To me, PostGIS is to FOSS4G what Linux is for FOSS as a whole. It's the enabling platform for succesfull FOSS4G enterprise applications. By concentrating on PostGIS and associated tools, we will build a reliable and consistent set of tools. Best regards, Gilberto -- === Dr.Gilberto Camara Director General National Institute for Space Research (INPE) Sao Jose dos Campos, Brazil -Original Message- From: [EMAIL PROTECTED] Date: Wed, 23 Jan 2008 17:15:27 To:discuss@lists.osgeo.org Subject: Discuss Digest, Vol 13, Issue 25 Send Discuss mailing list submissions to discuss@lists.osgeo.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.osgeo.org/mailman/listinfo/discuss or, via email, send a message with subject or body 'help' to [EMAIL PROTECTED] You can reach the person managing the list at [EMAIL PROTECTED] When replying, please edit your Subject line so it is more specific than "Re: Contents of Discuss digest..." Today's Topics: 1. Re: FOSS GIS Podcasts (andrea giacomelli) 2. RE: Open Geo Database (Sampson, David) 3. Finding Data Sets (Erik Uzureau) 4. Re: Finding Data Sets ([EMAIL PROTECTED]) 5. Re: Finding Data Sets (Chris Holmes) 6. Re: Finding Data Sets ([EMAIL PROTECTED]) -- Message: 1 Date: Wed, 23 Jan 2008 19:23:52 +0100 From: "andrea giacomelli" <[EMAIL PROTECTED]> Subject: Re: [OSGeo-Discuss] FOSS GIS Podcasts To: "OSGeo Discussions" Message-ID: <[EMAIL PROTECTED]> Content-Type: text/plain; charset=ISO-8859-1 ...Hi Landon, a) It's not a podcast (but still a ...cast, and it's fresh...subcribers of the Italian mailing lists, my apologies for re-posting): http://www.youtube.com/watch?v=_ZOQSn5tyqQ This is a sort of "teaser" video on GIS and the role that awareness raising organizations may play in promoting FOSS GIS (eventually talking about GFOSS.it for Italy). The message is directed to people who are not typical OSGeo discussion readers ;) , but I'd still appreciate feedback from the list. Maybe you can find some ideas of use for an introduction... b) I'd like to have more "episodes" gradually delving into more substantial topics, such as applications etc...time and resources allowing I'll be keeping an eye on material you may develop -and other links which are being flagged out- as they may be a starting point for new stuff. Regards! andrea, aka pibinko http://pibinko.altervista.org http://www.gfoss.it 2008/1/22, Nick Black <[EMAIL PROTECTED]>: > http://www.opengeodata.org/?cat=13 - have some interesting podcasts,.. > > > > > > Are there other existing FOSS GIS podcasts? > > > > > > > > > > > > Landon > > > -- Message: 2 Date: Wed, 23 Jan 2008 16:12:14 -0500 From: "Sampson, David" <[EMAIL PROTECTED]> Subject: RE: [OSGeo-Discuss] Open Geo Database To: "OSGeo Discussions" Message-ID: <[EMAIL PROTECTED]> Content-Type: text/plain; charset="iso-8859-1" just as another note, as of GRASS 7 it looks like they will be moving their default DB format to SQLITE instead of the traditional DBF format http://grass.gdf-hannover.de/wiki/GRASS_7_ideas_collection#Database it might make sense to have a community effort contribute to this as a whole. The positive side is that there is already adoption of an OSGEO product as GRASS is moving in this direction. Determine a comon schema that can be shared around between projects and voila. Or we could ignore this and then revist this same issue again in the future with no movement forward. just thought I'd share. Cheers From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Bob Basques Sent: Wednesday, January 16, 2008 09:40 To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open Geo Database All, Don't know the specifics about the engine being used, but we discussed using the Plone storage mechanism, a few times, for spatial data. Just didn't seem like a quick enough payoff to try anything out. At least not on the surface. There was a thought that there might be something to start with though. bobb Miguel Montesinos wrote: Hi, ¿Does anyone know about similar projects using object oriented databases, such as db4objects[1], wh
[OSGeo-Discuss] Re: FOSS4GIS business models
Dear Markus, Frank, and all I will try to dwell a bit further on some issues raised by Frank and Markus. Sorry for the long message. Certain issues deserve it... Message from Frank Warmerdam Comments from Gilberto (Frank) (...) But, I am left with the impression that the same model applied widely by many national or state governments would result in a lot of duplication. I'd like to explore models where governments at different levels cooperate and contribute to joint development. Good point. Governments have a propensity to fund local development, since they are creating jobs. The risk, as you point out, is duplication. To avoid this trend, we need a coordination board at the international level, that can assure national governments that their interests are taken into account. The ideal would be a UN-supported effort. However, developed nations have blocked attempts for United Nations agencies to coordinate widespread adoption of FOSS (e.g., at the World Summit on the Information Society). Currently, OSGEO and OSC play an important international role. OSGEO allows us to connect (as we´re doing right now). OSC gives users hope of avoiding the lock-in effect and thus reduces some of the FUD effect. But we should recognize that national government institutions are not present at either OSGEO or OSC. Thus, I´m hoping that GSDI might be able to increase its role as a place where FOSS can be promoted and presented to an international audience. GSDI´s president, Harlan Onsrud, is extremely serious, open-minded and supportive of FOSS. I will be at GSDI 2008 at Trindad and promise to give you a report from the trenches. (Frank) ...Perhaps due to the relentless propaganda of the "anti government right" in North America, I have some concerns about governments throwing large amounts of money into FOSS development without clear thinking about how to make that money work efficiently. It is easy to imagine boondoggles that could suck up lots of money with little in the way of useful products. Relax... Lord Keynes, Roosevelt´s New Deal, the Marshall Plan, NASA´s Apollo Program and other examples show that efficiency is not a prerequisite for successful use of public money. Efficient use of public money is not about achieving feasible products in the most cost-effective way. It is about achieving goals that would be impossible for the private sector. Putting a man on the moon, developing health care solutions for the poor, and saving the planet from disastrous climate change are tasks for governments. They cannot be measured by how much money you put into them, but by their results. Message from Markus Lupp > Responses by Gilberto > (Markus) I would like to give some comments on this from the perspective of a GIS company with an Open Source business model. > (...) Now to the question of government intervention. > After reading Gilberto's mail I asked myself what is meant by this term? > (...) In Germany (...)there is a growing support from people in governmental agencies who decided by themselves that they want to use more open source software > (Gilberto - is this what you mean by indirect support?). Government intervention can be direct or indirect. In the first case, public funds directly support FOSS4G projects (as in the case of gvSIG). In the second, there is a consensus that it is in the public interest to consider FOSS as a serious alternative to proprietary software and thus public managers are not afraid of FUD tactics (as it happens in Europe currently). Europe is taking FOSS very seriously. For those who haven´t seen it yet, take a look at the EC´s Open Source Observatory (http://ec.europa.eu/idabc/en/chapter/452). This is a very good example of indirect support for FOSS. > (Markus) So from my point of view it is possible to compete in the GIS market using an open source business model without any high-level government intervention (although it surely helps) I respectfully disagree. I doubt you could achieve the same success in the USA, where there is no active public policy in support of FOSS. I stand by my earlier assessment that successful long-term FOSS needs government support (direct or indirect). Best Gilberto -- === Dr.Gilberto Camara Director General National Institute for Space Research (INPE) Sao Jose dos Campos, Brazil voice: +55-12-3945-6035 fax: +55-12-3921-6455 web: http://www.dpi.inpe.br/gilberto blog: http://techne-episteme.blogspot.com/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
[OSGeo-Discuss] Re: FOSS4GIS business models
Dear OSGEO Discussion List members: Paul Ramsey´s remarks are right on target. First, GIS is a large arena and there are different motivations for developers, that prevent them from joining a single project such as uDIG. Second, it is very difficult for a private company to develop a world-class FOSS4G product and survive based only on consulting fees for the commercial sector. Third, to overcome these limitations there is a need for governmental intervention, which may be direct, as in the case of Catalonian government´s support for gvSIG, or indirect, as in the decision of Germany to support open source software. In Brazil, the National Institute for Space Research (INPE) has been supporting local GIS development for 25 years, with a lot of success in our national user community. Without official support, there would be no local FOOS4G development in Brazil. In 2003, I did a F00S4G market survey and published the results as a chapter of a US National Academy of Sciences book: "Open Source GIS Software: Myths and Realities" . We analysed 70 FOSS4G software projects taken from the FreeGIS list, and divided them into three categories: networked products (e.g. GRASS), corporate products (e.g., PostGIS) and individual products (e.g., CAVOR). From each product, we assessed its maturity, level of support and functionality. Our main conclusions at the time were: (a) Only 6% of the products were developed by networked teams. Thus, the “Linux paradigm” is the exception rather than the rule. (b) Corporations (private or public) are the main developers of successful open source products. Corporations account for 41% of all products. (e) Individual-led software (a small team of 1-3 people) have less quality and more mortality than the above. These results show that the impetus behind successful open source software was not coming from altruistic individuals working in the midnight hour, but from professional programmers. I consider that a similar result would be obtained today, should the assessment be repeated. This analysis was further elaborated in a JASIST paper: "Information Policies and Open Source Software in Developing Countries" . For the FOSS4G effort to be fruitful and sustainable, we need a very informed and candid assessment of our business model. My personal view, based on 25 years of experience, is that government intervention is essential for the open source model to survive beyond a handful of examples. Best regards Gilberto -- === Dr.Gilberto Camara Director General National Institute for Space Research (INPE) Sao Jose dos Campos, Brazil web: http://www.dpi.inpe.br/gilberto [EMAIL PROTECTED] escreveu: Send Discuss mailing list submissions to discuss@lists.osgeo.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.osgeo.org/mailman/listinfo/discuss or, via email, send a message with subject or body 'help' to [EMAIL PROTECTED] You can reach the person managing the list at [EMAIL PROTECTED] When replying, please edit your Subject line so it is more specific than "Re: Contents of Discuss digest..." Today's Topics: 1. Re: [postgis-users] A bit off topic, but FOSS GIS clients... (Paul Ramsey) 2. RE: Re: [postgis-users] A bit off topic, but FOSS GIS clients... (Landon Blake) 3. Re: Re: [postgis-users] A bit off topic, but FOSSGIS clients... (Andrea Aime) 4. Re: Re: [postgis-users] A bit off topic, but FOSS GIS clients... (Howard Butler) 5. Re: Re: [postgis-users] A bit off topic, but FOSSGIS clients... (Dr. Markus Lupp) 6. FOSS4Geo session at the Geosciences World Congress (33rd IGC 2008 in Oslo) (Henning Lorenz) 7. Re: Re: [postgis-users] A bit off topic, but FOSS GIS clients... (Jacolin Yves) 8. RE: Re: [postgis-users] A bit off topic, but FOSSGIS clients... (Miguel Montesinos) -- Message: 1 Date: Wed, 2 Jan 2008 09:41:19 -0800 From: Paul Ramsey <[EMAIL PROTECTED]> Subject: [OSGeo-Discuss] Re: [postgis-users] A bit off topic, but FOSS GIS clients... To: PostGIS Users Discussion <[EMAIL PROTECTED]> Cc: OSGeo Discussions Message-ID: <[EMAIL PROTECTED]> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed I realize it ain't easy. But could consolidation (future effort) make it easier? The only thing that can be consolidated is developer effort, and even where there are no programming language barriers (such as in the Java world) there are lots of countervailing reasons that make mergers impractical. "Everyone should drop their projects and work on uDig." But all the gvSIG developers are supported by funding from Spanish government that requires all the work be GPL; and they also prefer a pure Java implementation to the
