Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
On Wed, Nov 21, 2007 at 11:52:06PM -0700, Robert Bray wrote: Chris, I agree and will see what I can do to make it happen. If we want wider adoption it may also be beneficial to see some kind of C/C++ access library created for the format. In the past I always felt the FDO Provider was that library, but the masses seem to be telling me I am wrong about that :) /me sends a '30 days no news' nag, continues to wait patiently. Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Chris, I agree and will see what I can do to make it happen. If we want wider adoption it may also be beneficial to see some kind of C/C++ access library created for the format. In the past I always felt the FDO Provider was that library, but the masses seem to be telling me I am wrong about that :) Bob - Original Message - From: Christopher Schmidt [EMAIL PROTECTED] To: OSGeo Discussions discuss@lists.osgeo.org Sent: Wednesday, November 21, 2007 9:04 PM Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format On Tue, Nov 13, 2007 at 11:18:42PM -0800, Robert Bray wrote: Is it an open format? ABSOLUTELY (we just never wrote a spec, but I am willing to get it done) All this said, I'd really like to understand everyones requirements for this new format. If SDF fits thats great, if not thats ok too. We are always happy to contribute what we can to the community. I have an interest in seeing wider implementation of SDF. I have a user who is interested in using FeatureServer against SDF data -- but I don't have an environment in which I can build/use the FDO Python bindings (being a mac user), so I can't help the user out. It seems more likely that an alternative implementation to FDO's could be created if documentation on the format was available. Obviously, there's no guarenteee (and unfortunately, I'm not a very good coder, so I can't offer much), but I'd at least like to be able to investigate how difficult SDF might be to implement: not just for my own personal reasons, but because implementation-difficulty can affect uptake of a format as much as the technical benefits can (if not more). so in response to 'I am willing to get it done', I would strongly encourage you to do so! Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Robert Bray wrote: Chris, I agree and will see what I can do to make it happen. If we want wider adoption it may also be beneficial to see some kind of C/C++ access library created for the format. In the past I always felt the FDO Provider was that library, but the masses seem to be telling me I am wrong about that :) Bob, Sounds like a very good idea to setup a libsdf project. This way, it could be used by FDO, OGR and others. Cheers -- Mateusz Loskot http://mateusz.loskot.net ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
How about an OGR driver for SDF? No need to invent a new API when one already exists. Traian From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Mateusz Loskot [EMAIL PROTECTED] Sent: Thursday, November 22, 2007 2:36 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format Robert Bray wrote: Chris, I agree and will see what I can do to make it happen. If we want wider adoption it may also be beneficial to see some kind of C/C++ access library created for the format. In the past I always felt the FDO Provider was that library, but the masses seem to be telling me I am wrong about that :) Bob, Sounds like a very good idea to setup a libsdf project. This way, it could be used by FDO, OGR and others. Cheers -- Mateusz Loskot http://mateusz.loskot.net ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
On Thu, Nov 22, 2007 at 05:17:46PM -0800, Traian Stanev wrote: How about an OGR driver for SDF? No need to invent a new API when one already exists. I think that was exactly Bob's point: There is already an FDO driver for SDF. If OGR is sufficient, I'm not entirely sure why FDO wouldn't be by the same argument. I'm with Mat, in feeling that it's not the path to the widest usage in only FDO *or* OGR, and that a reference implementation outside of either of those would result in possibly higher OEM-style integration into non OGR/FDO products, and that a single shared implementation is the best way for all developers to share a single effort when moving forward. Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Traian Stanev wrote: How about an OGR driver for SDF? No need to invent a new API when one already exists. Traian, AFAIKU, that's the idea that triggered the debate but it would be good if there is non-API specific library for SDF. The the same library could be used by OGR, FDO, etc. Cheers -- Mateusz Loskot http://mateusz.loskot.net ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
I think many people have it backwards. SDF is basically a reference implementation of the FDO API (or a significant subset thereof). It's not an independent file format that happens to have an FDO driver. Any C++ API for SDF would likely have to use FDO data structures and types underneath to get SDF data out of the file. This is because FDO data structures are directly serialized into SDF binary tables. Traian From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Christopher Schmidt [EMAIL PROTECTED] Sent: Thursday, November 22, 2007 9:23 PM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format On Thu, Nov 22, 2007 at 05:17:46PM -0800, Traian Stanev wrote: How about an OGR driver for SDF? No need to invent a new API when one already exists. I think that was exactly Bob's point: There is already an FDO driver for SDF. If OGR is sufficient, I'm not entirely sure why FDO wouldn't be by the same argument. I'm with Mat, in feeling that it's not the path to the widest usage in only FDO *or* OGR, and that a reference implementation outside of either of those would result in possibly higher OEM-style integration into non OGR/FDO products, and that a single shared implementation is the best way for all developers to share a single effort when moving forward. Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
On Tue, Nov 13, 2007 at 11:18:42PM -0800, Robert Bray wrote: Is it an open format? ABSOLUTELY (we just never wrote a spec, but I am willing to get it done) All this said, I'd really like to understand everyones requirements for this new format. If SDF fits thats great, if not thats ok too. We are always happy to contribute what we can to the community. I have an interest in seeing wider implementation of SDF. I have a user who is interested in using FeatureServer against SDF data -- but I don't have an environment in which I can build/use the FDO Python bindings (being a mac user), so I can't help the user out. It seems more likely that an alternative implementation to FDO's could be created if documentation on the format was available. Obviously, there's no guarenteee (and unfortunately, I'm not a very good coder, so I can't offer much), but I'd at least like to be able to investigate how difficult SDF might be to implement: not just for my own personal reasons, but because implementation-difficulty can affect uptake of a format as much as the technical benefits can (if not more). so in response to 'I am willing to get it done', I would strongly encourage you to do so! Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
From: [EMAIL PROTECTED] [mailto:discuss- [EMAIL PROTECTED] On Behalf Of Jo Walsh Sent: 18 November 2007 6:26 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format dear Steve, all, On Tue, Nov 13, 2007 at 05:24:55PM +, Steve Coast wrote: Real artists ship. For everyone else there's standards wanking. As the origins of the word 'yardstick' suggest, size is relative, and standards and wanking have always been intimately connected. http://www.etymonline.com/index.php?search=yardstick this: We should have got a committee to design a standard, then we could think about a committee to design an ontology... and choose a As http://wiki.openstreetmap.org/index.php/Map_Features states, there is benefit in agreeing a recommended set of features and corresponding tags. This page is a beautiful work and has many interesting properties. But it's an ontology designed by a committee, the work of core contributors needing more structure to achieve their aims, than an completely open key-value tag system. I'll point out that the original map features list was created by me and not by committee. It was put together so that those with no experience of what might be mapped would have some idea of where to start, remember that OSM is not a project driven by the traditional cartography or GIS industries, for most people joining the project it all needs to be learnt from the ground up. Map Features was never intended to be a standard and hopefully it never will be. Since its introduction in March 06 there have been plenty who have advocated standardising it and a process of voting on proposed new tags to join the Map features list still goes on by a minority, however in practice if a tag is proposed and mappers find it useful they simply use it and that's the way it should be. Voting only sees a small handful of responses anyway. Principally it's a good way for discouraging poor tag design rather than setting a standard. I'm impressed to see how many different language communities have worked to translate this page and explain its contents. Yet, italians, swedes and spaniards alike must use english-language key and value 'tags' in order to have their work show up on the map. Many annotations made before the core recommended feature set was fixed have been lost to perception; though it's technically possible to use a different system, adapt the rendering and annotation clients, then that work would be lost to re-use. A bit more structure would afford a lot more future adaptation and translation. Map features was always intended to be an English orientated tagging list. I always envisaged that other languages would develop their own tags and use those for their local areas. I also expected (and still expect) to see other tagging formats for other peculiar uses. Map features never included tagging ideas for transport routing for instance. These haven't really happened much yet, perhaps through a lack of confidence in breaking with tradition more than anything else. The immediate gratification of rendered mapping (which uses the longest established and most prolifically used tags) is also a big driver. Hopefully though with time a lot more language specific tagging will turn up in the data set. OSM is a brilliant project, borne of real need and social momentum. Meanwhile some corners of the standards industry really have gone off the rails and appear to be acting against common sense and user benefit. About the most depressing thing i heard at FOSS4G was, in the middle of an interesting talk, we were going to implement cool feature X, but we're *waiting for the standard*. As i think you've pointed out in the past, standards like the core Map Features ought to emerge from common practise, from comparing different things that are shipped and running and being used. Standards that *do* work, like WMS and RSS, will get picked up. The core difference in approach seems to be a question of process. As Bitner pointed out, sometimes it makes sense to slow down a bit in order to let others catch up, so we can all go faster. Bitner may have a point but I'm not so sure it holds true for OSM, which has demonstrated that you just need to facilitate the creativity for great progress to be made. Standards have not been the driver for making it a success in the short time to date and each week we see something new and exciting within the project without any reference to an agreed standard or committee decision. If we stop and wait for everyone to catch up we will surely stifle the cutting edge creativity from which the project benefits immensely. Like all the exploration of the past there will always be a those forging ahead, setting the path of the project, and opening up new territories. Without them there will be nowhere for the masses to settle and make use of this great data OSM is creating. Cheers Andy Andy Robinson
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
dear Steve, all, On Tue, Nov 13, 2007 at 05:24:55PM +, Steve Coast wrote: Real artists ship. For everyone else there's standards wanking. As the origins of the word 'yardstick' suggest, size is relative, and standards and wanking have always been intimately connected. http://www.etymonline.com/index.php?search=yardstick this: We should have got a committee to design a standard, then we could think about a committee to design an ontology... and choose a As http://wiki.openstreetmap.org/index.php/Map_Features states, there is benefit in agreeing a recommended set of features and corresponding tags. This page is a beautiful work and has many interesting properties. But it's an ontology designed by a committee, the work of core contributors needing more structure to achieve their aims, than an completely open key-value tag system. I'm impressed to see how many different language communities have worked to translate this page and explain its contents. Yet, italians, swedes and spaniards alike must use english-language key and value 'tags' in order to have their work show up on the map. Many annotations made before the core recommended feature set was fixed have been lost to perception; though it's technically possible to use a different system, adapt the rendering and annotation clients, then that work would be lost to re-use. A bit more structure would afford a lot more future adaptation and translation. OSM is a brilliant project, borne of real need and social momentum. Meanwhile some corners of the standards industry really have gone off the rails and appear to be acting against common sense and user benefit. About the most depressing thing i heard at FOSS4G was, in the middle of an interesting talk, we were going to implement cool feature X, but we're *waiting for the standard*. As i think you've pointed out in the past, standards like the core Map Features ought to emerge from common practise, from comparing different things that are shipped and running and being used. Standards that *do* work, like WMS and RSS, will get picked up. The core difference in approach seems to be a question of process. As Bitner pointed out, sometimes it makes sense to slow down a bit in order to let others catch up, so we can all go faster. jo -- ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
These items are also supported in SDF today: - SRS is a must, complete projection parameters (stored as WKT). - full set of attibute types (c, int2, int4, int8, float, double, txt (i18l), date and timestamps (with gmt precise offset), blobs, ... - indexing (unique and non-unique) on key attributes The only exception is that I do not believe SDF supports the blob data type. There is also support for range constraints. Bob - Original Message - From: Paul Spencer [EMAIL PROTECTED] To: OSGeo Discussions discuss@lists.osgeo.org Sent: Wednesday, November 14, 2007 5:24 AM Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format On 14-Nov-07, at 7:20 AM, Christopher Schmidt wrote: - optional coloring and styles, break values, rendering and scale limits, persistent joins or relates, color ramp, ... are things which are provided by SLD and the like, which means that you really want SDF + WMC -- I don't think that this is SDF's job, and I don't think that it should be the job of any geodata format. mapinfo files come with embedded styling so there is at least one format out there that combines the two. KML arguably does the same thing. I think it would be worth exploring the merits of providing the ability to embed styling information. I'm not sure if it is useful to place styling 'hints' in metadata, assuming that the format supports arbitrary metadata. Cheers Paul +-+ |Paul Spencer [EMAIL PROTECTED]| +-+ |Chief Technology Officer | |DM Solutions Group Inchttp://www.dmsolutions.ca/ | +-+ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
On 11/14/07, Christopher Schmidt [EMAIL PROTECTED] wrote: On Wed, Nov 14, 2007 at 04:12:21AM -0600, [EMAIL PROTECTED] wrote: I would just love to click open and see something nice, specially if someone has already taken the time to make it beautiful. Think of it as the output of a word processor instead of an editor. Excel vs. VisiCalc; and the list may go on forever. All this is to say: not only the data needs a better container, the maps as well. A data format is not a 'container' for maps, it's a container for data. The container for your maps should be something like WMC (though I don't know if it makes sense off the web): grouping together datasources and SLD and the like. The first half of the things you identified: - optional space for metadata, - optional thumnails (in 2 sizes: thumb and browse) - optional embeded symbols (truetype/svg glyphs) and prefered layout - optional coloring and styles, break values, rendering and scale limits, persistent joins or relates, color ramp, ... are things which are provided by SLD and the like, which means that you really want SDF + WMC -- I don't think that this is SDF's job, and I don't think that it should be the job of any geodata format. From a pure data approach, I agree that how it looks does not belong in what it is. Data should be completely separate from the presentation. There is a part of me, though, that believes that embedding some default styling in the data container would be very useful. Apple's Quicklook is a great technology that allows one to quickly look at the data before deciding what to do with it. So, at there is some precedence for this. This is how it could work -- if any styling is provided from outside, the internal styling would be ignored. However, if nothing is provided from outside, then the default styling as determined by the creator would be used. Being able to click on the data package and see a preview would be fantastic. Built in containers for metadata and SRS would be absolutely essential. Manolo, could you please add these to the wiki page that David set up, else I will add it but not until tomorrow or so. Regards, -- Christopher Schmidt Web Developer ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss -- Puneet Kishor http://punkish.eidesis.org/ Nelson Institute for Environmental Studies http://www.nelson.wisc.edu/ Open Source Geospatial Foundation (OSGeo) http://www.osgeo.org/ Summer 2007 ST Policy Fellow, The National Academies http://www.nas.edu/ ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
disclaimer: I speak from a position of complete ignorance about SDF, so everything I might say could be wrong. On 11/14/07, Robert Bray [EMAIL PROTECTED] wrote: A little clarification on SDF, since it comes up once or twice in this thread. Jason's earlier descriptions of it's capabilities are pretty good. It supports multiple Feature Classes / Tables per file, strongly typed properties, multiple geometry properties per class, with seperate R-Trees for each geometry property. All of this is stored in a single file that is heavily optimized for spatial reads. The SDF FDO Provider suppports a multiple reader / single writer model. The geometries themselves include simple features + circular arcs in 2D, 2D with Z, 2D with M, or 2D with Z M. The R-Trees are currently 2D. SDF does sound the cat's meow. Is it an open format? ABSOLUTELY (we just never wrote a spec, but I am willing to get it done) What is the licensing of this format? What is the plan for licensing of this format? Is the format going to be put out in public domain ever? This is an important discussion, but it probably doesn't belong in this thread. I bring it up here just to make sure it is not forgotten. Another little known fact is that in the process of creating SDF we (Autodesk) literally wrote the code three times. The first time we built SDF on BerkleyDB, a great open source project but it has some fairly significant license fees for using it in a proprietary product. The second time we wrote it on SQLite, however the performance penalaty of the Relational layer was significant (read orders of magnitude). The third time we chose to strip away the SQLite relational layer and built directly on the SQLite Backend components (B-Tree, Pager, and OS Interface). In the end the third implementation actually turned out to be faster than BDB and is the one we use today. By not having the relational layer we lose a lot in terms of attribute data handling, or is that still somehow preserved in SDF? Keep in mind, SQLite has gone through major revisions, some of them bringing huge speed bumps in the process. That said, different folks have different evaluation critieria -- I believe it was Steve W who said that pretty much all he cared about was speed. For him it seems SDF might work very well. For me, I don't mind sacrificing some speed for ease and flexibility of related data storage, querying, and retrieval. How can we achieve both? All this said, I'd really like to understand everyones requirements for this new format. If SDF fits thats great, if not thats ok too. We are always happy to contribute what we can to the community. Thanks Bob. As you can see, we are still articulating the requirements, and while we might be accused of requirement wanking, design by committee might be better than no design at all. I find two problems with Shapefiles -- one, that it is not in public domain (I am not even sure of what licensing there is on it), and while ESRI is not likely to pull a Unisys on us, it just is philosophically better to free if possible. The second, more major problem is Shapefile's antiquated data technology. DBF is a royal pain in the derierre with its 10 char column name, no relational tables, no indexing of data constraints. The geometry part of Shapefiles seems to be pretty good and adequate; it is the data part that is the problem. Bob - Original Message - From: Michael P. Gerlek [EMAIL PROTECTED] To: [EMAIL PROTECTED]; OSGeo Discussions discuss@lists.osgeo.org; [EMAIL PROTECTED] Sent: Tuesday, November 13, 2007 12:23 PM Subject: RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format Regarding the suggestion that MapServer takes on this new format as the primary format: I think this is way beyond the scope of what OSGeo should be doing. I agree with bitnerd. If the MapServer team thinks this is a valuable and worthwhile format, they will adopt it at some point. It would not be unreasonable for them to step back and see how thing progresses before deciding to spend their valuable ergs on it. The burden is on the OpenShape people to show the idea is worthwhile and meritorious. (My two cents on the standards question: OSGeo is not a standards organization, but / however / on the other hand / nonetheless one of the reasons OSGeo exists is to foster such collaborations. If some people want to try and develop something new like this, I'm all in favor of OSGeo offering mailing list and wiki space to help out. Declaring this to be a standard effort, however, is probably premature in any case -- more useful at this point to see the idea sketched out further, see who's interested, etc.) -mpg ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
P Kishor wrote: I find two problems with Shapefiles -- one, that it is not in public domain (I am not even sure of what licensing there is on it), and while ESRI is not likely to pull a Unisys on us, it just is philosophically better to free if possible. Puneet, I think this is a red herring. A file format is not normally considered copyrightable. Generally speaking the only way to control a file format is through patents (for instance on compression methods or some fancy spatial indexing) and through trademarks on the format name. There is no apparent such issues with shapefiles. While the Shapefile format may not have been developed by an open process, nor is it (as far as I know) a dejure standard, but it is as open as any format. I think we need to focus on the technical failings of existing formats, and ensure a new format is widely usable (no strings attached). There were also questions about SDF format. As I see it, one failing of the SDF format is that there isn't a published specification (that I'm aware of) and it is a complex enough format that it would be challenging to build a complete distinct library for the format (in Java for instance). Also, the format is really an evolving format. But, I still think SDF is a good candidate. I think, so far, there isn't a strong enough motivation in the community to develop a new format that does everything I want. So (I think) a green field design is unlikely to get past the standards wanking stage that SteveC always goes out of his way to deride. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, [EMAIL PROTECTED] light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| President OSGeo, http://osgeo.org ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
On 11/14/07, David William Bitner [EMAIL PROTECTED] wrote: I never (I think I never did) argued that Shapefile is not open. I argued that it is not Free. I could be wrong. Here's the open published specification: http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf yes, I am aware of this, and have used it on and off over the past decade. Do what you will with it. I don't know in this case what you imply by Free. Might ESRI make changes in the future that they don't publish? Maybe, but at this version of shapefiles -- that are pervasive throughout the industry, I would doubt they would forgo backwards compatibility and this version of the specification is out there and free and open as far as I can tell. ok, so there may or may not be an issue. I did bring it up, and so I risk the discussion focusing on this aspect too much. Let's forget about this aspect for now. Shall we focus on the technical limitations of Shapefiles in order to keep moving forward? Many thanks, Puneet. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
I find two problems with Shapefiles -- one, that it is not in public domain (I am not even sure of what licensing there is on it), and while ESRI is not likely to pull a Unisys on us, it just is philosophically better to free if possible. I don't see this as an issue at all -- legally speaking I don't think ESRI has any grounds to attack anyone for using or implementing the Shapefile format. (and from a market standpoint too, it'd be a bad move for them...) The second, more major problem is Shapefile's antiquated data technology. DBF is a royal pain in the derierre with its 10 char column name, no relational tables, no indexing of data constraints. The geometry part of Shapefiles seems to be pretty good and adequate; it is the data part that is the problem. This, though, is more interesting. There are some things in the shapefile format that limit me today, and so I'm interested in alternatives. -mpg ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Landon Blake wrote: P.S. - This is probably a crazy idea, but has anyone ever considered talking to ESRI about cooperating on an update Shapefile spec? Landon, I believe ESRI sees the file based geodatabase as filling roughly the role that the Shapefile played in the ArcView 3.x days. Of course it isn't clear that it will be very open and interoperable with other software packages. I can't imagine a upgrade to shapefiles that would satisfy all our requirements that has anything but the name in common with the existing shapefiles. So why drag ESRI into it? Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, [EMAIL PROTECTED] light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| President OSGeo, http://osgeo.org ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Puneet wrote: Shall we focus on the technical limitations of Shapefiles in order to keep moving forward? I was going to add a couple of limitations in addition to the limitations of attribute data brought on by use of a DBF file as a storage container: [1] No way to store simple topology. [2] No way to store features with different schema and/or different geometry representations in the same file. [3] No way to store labeling or annotation. I'm not sure if you would want to do all of this with a new file format, but I know its limitations of the Shapefile format that I have run into in the past. Landon P.S. - This is probably a crazy idea, but has anyone ever considered talking to ESRI about cooperating on an update Shapefile spec? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of P Kishor Sent: Wednesday, November 14, 2007 9:06 AM To: [EMAIL PROTECTED] Cc: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format On 11/14/07, David William Bitner [EMAIL PROTECTED] wrote: I never (I think I never did) argued that Shapefile is not open. I argued that it is not Free. I could be wrong. Here's the open published specification: http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf yes, I am aware of this, and have used it on and off over the past decade. Do what you will with it. I don't know in this case what you imply by Free. Might ESRI make changes in the future that they don't publish? Maybe, but at this version of shapefiles -- that are pervasive throughout the industry, I would doubt they would forgo backwards compatibility and this version of the specification is out there and free and open as far as I can tell. ok, so there may or may not be an issue. I did bring it up, and so I risk the discussion focusing on this aspect too much. Let's forget about this aspect for now. Shall we focus on the technical limitations of Shapefiles in order to keep moving forward? Many thanks, Puneet. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Frank, I was watching this PyTables video [http://www.carabos.com/videos/pytables-1-intro] and one thought came to my mind: HDF5 can easily be used to store and retrieve vector, raster and attribute tables. We would need to standardize a schema tough. Best regards, Ivan PS. I am not that Ivan on that video. Frank Warmerdam wrote: Landon Blake wrote: P.S. - This is probably a crazy idea, but has anyone ever considered talking to ESRI about cooperating on an update Shapefile spec? Landon, I believe ESRI sees the file based geodatabase as filling roughly the role that the Shapefile played in the ArcView 3.x days. Of course it isn't clear that it will be very open and interoperable with other software packages. I can't imagine a upgrade to shapefiles that would satisfy all our requirements that has anything but the name in common with the existing shapefiles. So why drag ESRI into it? Best regards, ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Puneet, You wrote: Should be easy to transition to. By building the new format on the structure of the Shapefile format, and *in fact*, calling it open shapefiles or some such thing, we indicate from its name that the transition is not that revolutionary but is evolutionary. This, hopefully, will bring some name-familiarity, and make the transition less scary. I really think you are going to run into problems using the Shapefile as part of the trademark or name for any product not sold by ESRI. I strongly recommend against this move. Let people adopt the implementation of your idea for its merits, not for name recognition that comes from another product line. You wrote: ANSI standard C is still that magic common denominator that compiles and works predictably on most number of systems. I have a lot against Java, but those who love Java should definitely work on tools for accessing and working with this new format as it would only make the format more widely used and adopted. It sounds to me like you are really describing a tool. File formats are written in a binary encoding or text, not in a programming language. If you are designing a tool you can choose the programming language of your choice, but be aware that this will limit the developers that adopt the tool. This will be the case no matter what language you choose to use, whether it is C, Java, or something else. If, in contrast, you are creating a file format, then programming languages shouldn't really matter. Binary and text data can be accessed by almost all programming languages. I think you need to decide if you want a tool or a data format. It sounds like you are shooting more for a spatial database written in the C programming language that uses some form of the ESRI Shapefile as its underlying data storage mechanism. To me that is a tool or piece of software, not a format. But maybe I don't completely understand your goal. Landon -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of P Kishor Sent: Tuesday, November 13, 2007 8:35 AM To: OSGeo Discussions Subject: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format Thanks everyone, for responding. Here is my groundwork. The new format -- - Should be fast. SQLite is plenty fast, and anything that simply extends the Shapefile format to inject relational capabilities should be pretty fast. It should definitely be faster than a geodatabase format (such as PostGIS/ArcSDE) and perhaps even faster than Shapefiles especially while accessing attribute data. DBF is sequential, and searching for textual information is particularly expensive. SQLite has been tuned to excellence. I have been working with it for a few years now, and it really is an amazing product, development community, support, and capabilities. That it is in public domain makes for a transfat-free icing on the cake. - Should be unencumbered by licenses and copyrights. Ideally, the new format could also be put back into public domain. We want to remove all encumbrances to encourage rapid and wide adoption. - Should be a single file. Well, some like multiple files and some like single files. We can achieve both objectives by using a tar-gzipped packaging such as Apple tends to use for much of its stuff (for example, its Pages wordprocessor uses a tgzipped xml file along with other resources for icons and pictures and stuff). Or, if speed is going to be affected because of gzipping and gunzipping, just a package format (I have no idea if this is a Unix thing or a Mac OS thing -- we, in the Mac world, call them packages... they appear like files in the Finder, and like directories in the shell). - Should be easy to transition to. By building the new format on the structure of the Shapefile format, and *in fact*, calling it open shapefiles or some such thing, we indicate from its name that the transition is not that revolutionary but is evolutionary. This, hopefully, will bring some name-familiarity, and make the transition less scary. - Frank mentions SQLite's lack of datatypes as an issue -- I guess that is a matter of preference. I personally quite like that freedom as it gives me, the application developer, complete control over what goes where. SQLite actually does have now a few datatypes that it respects, but doesn't complain about. Since all users will be accessing the data via an application, as long as the application is well defined, it should be fine. - SQLite excels at one thing that it has been entrusted to do -- retrieve data that it has been entrusted with at extremely fast speeds, and maintain ACID data integrity in case of a programmatic catastrophe. The transactions themselves are worth their price of admission, which, happily, happens to be zero. - Langdon mentions Java support -- well, yes, use/work on SQLite JDBC. I have been using it for a few days now and find it to be a pretty competent conduit. Extend it, spatialize it. ANSI standard C
RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
Landon wrote: --- I really think you are going to run into problems using the Shapefile as part of the trademark or name for any product not sold by ESRI. I strongly recommend against this move. --- I'm not a lawyer, but I really doubt that shapefile is unique enough to be subject to trademark. Initial searches of TESS don't show up anything either. That said, I don't think that shapefile should be used anywhere in an open format name _because_ it's too generic and will cause confusion. As my initial blog entry mentioned, I had (have?) pretty high hopes for SDF. It's extremely fast (native rtree), contains multiple tables in a single file, supports complex geometries (arcs, aggregates, etc), includes native schema definition, etc... However... while some initial marketing materials marked SDF as an open format, I have been unable to obtain any clarification from Autodesk on the file specification or openness of SDF. As well, there are some limitations; in the quest for speed, SDF is only using the SQLite table access mechanisms, not the entire relational layer (or something like that). For reference, most of the interesting parts of the SDF Provider, including the Rtree implementation, are available here: http://tinyurl.com/2b5en2 Jason ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
on tools for accessing and working with this new format as it would only make the format more widely used and adopted. It sounds to me like you are really describing a tool. File formats are written in a binary encoding or text, not in a programming language. If you are designing a tool you can choose the programming language of your choice, but be aware that this will limit the developers that adopt the tool. This will be the case no matter what language you choose to use, whether it is C, Java, or something else. If, in contrast, you are creating a file format, then programming languages shouldn't really matter. Binary and text data can be accessed by almost all programming languages. I think you need to decide if you want a tool or a data format. It sounds like you are shooting more for a spatial database written in the C programming language that uses some form of the ESRI Shapefile as its underlying data storage mechanism. To me that is a tool or piece of software, not a format. But maybe I don't completely understand your goal. well, I am, frankly confused. I was quite convinced I wasn't describing a tool but was describing a format. Of course, to describe the format, I positioned it on the format (the SQLite-compatible format) used and popularized by a tool (SQLite, the library, which happens to be written in C). In my mind, having the data format based on SQLite *format* for its relational attribute handling was the real winner. In that sense, perhaps I conflated the format and the tool. I am not well versed in these things to I am probably already walking on thin ice, but that shouldn't stop others. So, forget that I mentioned C and Java... let's just concentrate on a way of laying out data on a disk that is not too dissimilar from how Shapefile data are laid out, except that we utilize the SQLite-compatible binary format for relational data handling, so that SQLite-enabled spatial tools can access this new format. And, put this format into public domain. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of P Kishor Sent: Tuesday, November 13, 2007 8:35 AM To: OSGeo Discussions Subject: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format Thanks everyone, for responding. Here is my groundwork. The new format -- - Should be fast. SQLite is plenty fast, and anything that simply extends the Shapefile format to inject relational capabilities should be pretty fast. It should definitely be faster than a geodatabase format (such as PostGIS/ArcSDE) and perhaps even faster than Shapefiles especially while accessing attribute data. DBF is sequential, and searching for textual information is particularly expensive. SQLite has been tuned to excellence. I have been working with it for a few years now, and it really is an amazing product, development community, support, and capabilities. That it is in public domain makes for a transfat-free icing on the cake. - Should be unencumbered by licenses and copyrights. Ideally, the new format could also be put back into public domain. We want to remove all encumbrances to encourage rapid and wide adoption. - Should be a single file. Well, some like multiple files and some like single files. We can achieve both objectives by using a tar-gzipped packaging such as Apple tends to use for much of its stuff (for example, its Pages wordprocessor uses a tgzipped xml file along with other resources for icons and pictures and stuff). Or, if speed is going to be affected because of gzipping and gunzipping, just a package format (I have no idea if this is a Unix thing or a Mac OS thing -- we, in the Mac world, call them packages... they appear like files in the Finder, and like directories in the shell). - Should be easy to transition to. By building the new format on the structure of the Shapefile format, and *in fact*, calling it open shapefiles or some such thing, we indicate from its name that the transition is not that revolutionary but is evolutionary. This, hopefully, will bring some name-familiarity, and make the transition less scary. - Frank mentions SQLite's lack of datatypes as an issue -- I guess that is a matter of preference. I personally quite like that freedom as it gives me, the application developer, complete control over what goes where. SQLite actually does have now a few datatypes that it respects, but doesn't complain about. Since all users will be accessing the data via an application, as long as the application is well defined, it should be fine. - SQLite excels at one thing that it has been entrusted to do -- retrieve data that it has been entrusted with at extremely fast
Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format
of the trademark or name for any product not sold by ESRI. I strongly recommend against this move. Let people adopt the implementation of your idea for its merits, not for name recognition that comes from another product line. Good enough point to keep in mind, but not to get hung up over enough to entangle us. Suggestions for names of the data format can be a project in itself. open spatial data format or its variations could be chosen. Still, point taken. You wrote: ANSI standard C is still that magic common denominator that compiles and works predictably on most number of systems. I have a lot against Java, but those who love Java should definitely work on tools for accessing and working with this new format as it would only make the format more widely used and adopted. It sounds to me like you are really describing a tool. File formats are written in a binary encoding or text, not in a programming language. If you are designing a tool you can choose the programming language of your choice, but be aware that this will limit the developers that adopt the tool. This will be the case no matter what language you choose to use, whether it is C, Java, or something else. If, in contrast, you are creating a file format, then programming languages shouldn't really matter. Binary and text data can be accessed by almost all programming languages. I think you need to decide if you want a tool or a data format. It sounds like you are shooting more for a spatial database written in the C programming language that uses some form of the ESRI Shapefile as its underlying data storage mechanism. To me that is a tool or piece of software, not a format. But maybe I don't completely understand your goal. well, I am, frankly confused. I was quite convinced I wasn't describing a tool but was describing a format. Of course, to describe the format, I positioned it on the format (the SQLite-compatible format) used and popularized by a tool (SQLite, the library, which happens to be written in C). In my mind, having the data format based on SQLite *format* for its relational attribute handling was the real winner. In that sense, perhaps I conflated the format and the tool. I am not well versed in these things to I am probably already walking on thin ice, but that shouldn't stop others. So, forget that I mentioned C and Java... let's just concentrate on a way of laying out data on a disk that is not too dissimilar from how Shapefile data are laid out, except that we utilize the SQLite-compatible binary format for relational data handling, so that SQLite-enabled spatial tools can access this new format. And, put this format into public domain. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of P Kishor Sent: Tuesday, November 13, 2007 8:35 AM To: OSGeo Discussions Subject: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open data format Thanks everyone, for responding. Here is my groundwork. The new format -- - Should be fast. SQLite is plenty fast, and anything that simply extends the Shapefile format to inject relational capabilities should be pretty fast. It should definitely be faster than a geodatabase format (such as PostGIS/ArcSDE) and perhaps even faster than Shapefiles especially while accessing attribute data. DBF is sequential, and searching for textual information is particularly expensive. SQLite has been tuned to excellence. I have been working with it for a few years now, and it really is an amazing product, development community, support, and capabilities. That it is in public domain makes for a transfat-free icing on the cake. - Should be unencumbered by licenses and copyrights. Ideally, the new format could also be put back into public domain. We want to remove all encumbrances to encourage rapid and wide adoption. - Should be a single file. Well, some like multiple files and some like single files. We can achieve both objectives by using a tar-gzipped packaging such as Apple tends to use for much of its stuff (for example, its Pages wordprocessor uses a tgzipped xml file along with other resources for icons and pictures and stuff). Or, if speed is going to be affected because of gzipping and gunzipping, just a package format (I have no idea if this is a Unix thing or a Mac OS thing -- we, in the Mac world, call them packages... they appear like files in the Finder, and like directories in the shell). - Should be easy to transition to. By building the new format on the structure of the Shapefile format, and *in fact
