Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
These items are also supported in SDF today: - SRS is a must, complete projection parameters (stored as WKT). - full set of attibute types (c, int2, int4, int8, float, double, txt (i18l), date and timestamps (with gmt precise offset), blobs, ... - indexing (unique and non-unique) on key attributes The only exception is that I do not believe SDF supports the blob data type. There is also support for range constraints. Bob - Original Message - From: Paul Spencer [EMAIL PROTECTED] To: OSGeo Discussions discuss@lists.osgeo.org Sent: Wednesday, November 14, 2007 5:24 AM Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format On 14-Nov-07, at 7:20 AM, Christopher Schmidt wrote: - optional coloring and styles, break values, rendering and scale limits, persistent joins or relates, color ramp, ... are things which are provided by SLD and the like, which means that you really want SDF + WMC -- I don't think that this is SDF's job, and I don't think that it should be the job of any geodata format. mapinfo files come with embedded styling so there is at least one format out there that combines the two. KML arguably does the same thing. I think it would be worth exploring the merits of providing the ability to embed styling information. I'm not sure if it is useful to place styling 'hints' in metadata, assuming that the format supports arbitrary metadata. Cheers Paul +-+ |Paul Spencer [EMAIL PROTECTED]| +-+ |Chief Technology Officer | |DM Solutions Group Inchttp://www.dmsolutions.ca/ | +-+ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
On 11/14/07, Christopher Schmidt [EMAIL PROTECTED] wrote: On Wed, Nov 14, 2007 at 04:12:21AM -0600, [EMAIL PROTECTED] wrote: I would just love to click open and see something nice, specially if someone has already taken the time to make it beautiful. Think of it as the output of a word processor instead of an editor. Excel vs. VisiCalc; and the list may go on forever. All this is to say: not only the data needs a better container, the maps as well. A data format is not a 'container' for maps, it's a container for data. The container for your maps should be something like WMC (though I don't know if it makes sense off the web): grouping together datasources and SLD and the like. The first half of the things you identified: - optional space for metadata, - optional thumnails (in 2 sizes: thumb and browse) - optional embeded symbols (truetype/svg glyphs) and prefered layout - optional coloring and styles, break values, rendering and scale limits, persistent joins or relates, color ramp, ... are things which are provided by SLD and the like, which means that you really want SDF + WMC -- I don't think that this is SDF's job, and I don't think that it should be the job of any geodata format. From a pure data approach, I agree that how it looks does not belong in what it is. Data should be completely separate from the presentation. There is a part of me, though, that believes that embedding some default styling in the data container would be very useful. Apple's Quicklook is a great technology that allows one to quickly look at the data before deciding what to do with it. So, at there is some precedence for this. This is how it could work -- if any styling is provided from outside, the internal styling would be ignored. However, if nothing is provided from outside, then the default styling as determined by the creator would be used. Being able to click on the data package and see a preview would be fantastic. Built in containers for metadata and SRS would be absolutely essential. Manolo, could you please add these to the wiki page that David set up, else I will add it but not until tomorrow or so. Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss -- Puneet Kishor http://punkish.eidesis.org/ Nelson Institute for Environmental Studies http://www.nelson.wisc.edu/ Open Source Geospatial Foundation (OSGeo) http://www.osgeo.org/ Summer 2007 ST Policy Fellow, The National Academies http://www.nas.edu/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
disclaimer: I speak from a position of complete ignorance about SDF, so everything I might say could be wrong. On 11/14/07, Robert Bray [EMAIL PROTECTED] wrote: A little clarification on SDF, since it comes up once or twice in this thread. Jason's earlier descriptions of it's capabilities are pretty good. It supports multiple Feature Classes / Tables per file, strongly typed properties, multiple geometry properties per class, with seperate R-Trees for each geometry property. All of this is stored in a single file that is heavily optimized for spatial reads. The SDF FDO Provider suppports a multiple reader / single writer model. The geometries themselves include simple features + circular arcs in 2D, 2D with Z, 2D with M, or 2D with Z M. The R-Trees are currently 2D. SDF does sound the cat's meow. Is it an open format? ABSOLUTELY (we just never wrote a spec, but I am willing to get it done) What is the licensing of this format? What is the plan for licensing of this format? Is the format going to be put out in public domain ever? This is an important discussion, but it probably doesn't belong in this thread. I bring it up here just to make sure it is not forgotten. Another little known fact is that in the process of creating SDF we (Autodesk) literally wrote the code three times. The first time we built SDF on BerkleyDB, a great open source project but it has some fairly significant license fees for using it in a proprietary product. The second time we wrote it on SQLite, however the performance penalaty of the Relational layer was significant (read orders of magnitude). The third time we chose to strip away the SQLite relational layer and built directly on the SQLite Backend components (B-Tree, Pager, and OS Interface). In the end the third implementation actually turned out to be faster than BDB and is the one we use today. By not having the relational layer we lose a lot in terms of attribute data handling, or is that still somehow preserved in SDF? Keep in mind, SQLite has gone through major revisions, some of them bringing huge speed bumps in the process. That said, different folks have different evaluation critieria -- I believe it was Steve W who said that pretty much all he cared about was speed. For him it seems SDF might work very well. For me, I don't mind sacrificing some speed for ease and flexibility of related data storage, querying, and retrieval. How can we achieve both? All this said, I'd really like to understand everyones requirements for this new format. If SDF fits thats great, if not thats ok too. We are always happy to contribute what we can to the community. Thanks Bob. As you can see, we are still articulating the requirements, and while we might be accused of requirement wanking, design by committee might be better than no design at all. I find two problems with Shapefiles -- one, that it is not in public domain (I am not even sure of what licensing there is on it), and while ESRI is not likely to pull a Unisys on us, it just is philosophically better to free if possible. The second, more major problem is Shapefile's antiquated data technology. DBF is a royal pain in the derierre with its 10 char column name, no relational tables, no indexing of data constraints. The geometry part of Shapefiles seems to be pretty good and adequate; it is the data part that is the problem. Bob - Original Message - From: Michael P. Gerlek [EMAIL PROTECTED] To: [EMAIL PROTECTED]; OSGeo Discussions discuss@lists.osgeo.org; [EMAIL PROTECTED] Sent: Tuesday, November 13, 2007 12:23 PM Subject: RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format Regarding the suggestion that MapServer takes on this new format as the primary format: I think this is way beyond the scope of what OSGeo should be doing. I agree with bitnerd. If the MapServer team thinks this is a valuable and worthwhile format, they will adopt it at some point. It would not be unreasonable for them to step back and see how thing progresses before deciding to spend their valuable ergs on it. The burden is on the OpenShape people to show the idea is worthwhile and meritorious. (My two cents on the standards question: OSGeo is not a standards organization, but / however / on the other hand / nonetheless one of the reasons OSGeo exists is to foster such collaborations. If some people want to try and develop something new like this, I'm all in favor of OSGeo offering mailing list and wiki space to help out. Declaring this to be a standard effort, however, is probably premature in any case -- more useful at this point to see the idea sketched out further, see who's interested, etc.) -mpg ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
P Kishor wrote: I find two problems with Shapefiles -- one, that it is not in public domain (I am not even sure of what licensing there is on it), and while ESRI is not likely to pull a Unisys on us, it just is philosophically better to free if possible. Puneet, I think this is a red herring. A file format is not normally considered copyrightable. Generally speaking the only way to control a file format is through patents (for instance on compression methods or some fancy spatial indexing) and through trademarks on the format name. There is no apparent such issues with shapefiles. While the Shapefile format may not have been developed by an open process, nor is it (as far as I know) a dejure standard, but it is as open as any format. I think we need to focus on the technical failings of existing formats, and ensure a new format is widely usable (no strings attached). There were also questions about SDF format. As I see it, one failing of the SDF format is that there isn't a published specification (that I'm aware of) and it is a complex enough format that it would be challenging to build a complete distinct library for the format (in Java for instance). Also, the format is really an evolving format. But, I still think SDF is a good candidate. I think, so far, there isn't a strong enough motivation in the community to develop a new format that does everything I want. So (I think) a green field design is unlikely to get past the standards wanking stage that SteveC always goes out of his way to deride. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, [EMAIL PROTECTED] light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| President OSGeo, http://osgeo.org ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
On 11/14/07, David William Bitner [EMAIL PROTECTED] wrote: I never (I think I never did) argued that Shapefile is not open. I argued that it is not Free. I could be wrong. Here's the open published specification: http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf yes, I am aware of this, and have used it on and off over the past decade. Do what you will with it. I don't know in this case what you imply by Free. Might ESRI make changes in the future that they don't publish? Maybe, but at this version of shapefiles -- that are pervasive throughout the industry, I would doubt they would forgo backwards compatibility and this version of the specification is out there and free and open as far as I can tell. ok, so there may or may not be an issue. I did bring it up, and so I risk the discussion focusing on this aspect too much. Let's forget about this aspect for now. Shall we focus on the technical limitations of Shapefiles in order to keep moving forward? Many thanks, Puneet. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
I find two problems with Shapefiles -- one, that it is not in public domain (I am not even sure of what licensing there is on it), and while ESRI is not likely to pull a Unisys on us, it just is philosophically better to free if possible. I don't see this as an issue at all -- legally speaking I don't think ESRI has any grounds to attack anyone for using or implementing the Shapefile format. (and from a market standpoint too, it'd be a bad move for them...) The second, more major problem is Shapefile's antiquated data technology. DBF is a royal pain in the derierre with its 10 char column name, no relational tables, no indexing of data constraints. The geometry part of Shapefiles seems to be pretty good and adequate; it is the data part that is the problem. This, though, is more interesting. There are some things in the shapefile format that limit me today, and so I'm interested in alternatives. -mpg ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Landon Blake wrote: P.S. - This is probably a crazy idea, but has anyone ever considered talking to ESRI about cooperating on an update Shapefile spec? Landon, I believe ESRI sees the file based geodatabase as filling roughly the role that the Shapefile played in the ArcView 3.x days. Of course it isn't clear that it will be very open and interoperable with other software packages. I can't imagine a upgrade to shapefiles that would satisfy all our requirements that has anything but the name in common with the existing shapefiles. So why drag ESRI into it? Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, [EMAIL PROTECTED] light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| President OSGeo, http://osgeo.org ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Puneet wrote: Shall we focus on the technical limitations of Shapefiles in order to keep moving forward? I was going to add a couple of limitations in addition to the limitations of attribute data brought on by use of a DBF file as a storage container: [1] No way to store simple topology. [2] No way to store features with different schema and/or different geometry representations in the same file. [3] No way to store labeling or annotation. I'm not sure if you would want to do all of this with a new file format, but I know its limitations of the Shapefile format that I have run into in the past. Landon P.S. - This is probably a crazy idea, but has anyone ever considered talking to ESRI about cooperating on an update Shapefile spec? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of P Kishor Sent: Wednesday, November 14, 2007 9:06 AM To: [EMAIL PROTECTED] Cc: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format On 11/14/07, David William Bitner [EMAIL PROTECTED] wrote: I never (I think I never did) argued that Shapefile is not open. I argued that it is not Free. I could be wrong. Here's the open published specification: http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf yes, I am aware of this, and have used it on and off over the past decade. Do what you will with it. I don't know in this case what you imply by Free. Might ESRI make changes in the future that they don't publish? Maybe, but at this version of shapefiles -- that are pervasive throughout the industry, I would doubt they would forgo backwards compatibility and this version of the specification is out there and free and open as far as I can tell. ok, so there may or may not be an issue. I did bring it up, and so I risk the discussion focusing on this aspect too much. Let's forget about this aspect for now. Shall we focus on the technical limitations of Shapefiles in order to keep moving forward? Many thanks, Puneet. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Frank, I was watching this PyTables video [http://www.carabos.com/videos/pytables-1-intro] and one thought came to my mind: HDF5 can easily be used to store and retrieve vector, raster and attribute tables. We would need to standardize a schema tough. Best regards, Ivan PS. I am not that Ivan on that video. Frank Warmerdam wrote: Landon Blake wrote: P.S. - This is probably a crazy idea, but has anyone ever considered talking to ESRI about cooperating on an update Shapefile spec? Landon, I believe ESRI sees the file based geodatabase as filling roughly the role that the Shapefile played in the ArcView 3.x days. Of course it isn't clear that it will be very open and interoperable with other software packages. I can't imagine a upgrade to shapefiles that would satisfy all our requirements that has anything but the name in common with the existing shapefiles. So why drag ESRI into it? Best regards, ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
