Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Christopher Schmidt]

On Wed, Nov 21, 2007 at 11:52:06PM -0700, Robert Bray wrote:
> Chris,
> 
> I agree and will see what I can do to make it happen. If we want wider 
> adoption it may also be beneficial to see some kind of C/C++ access library 
> created for the format. In the past I always felt the FDO Provider was that 
> library, but the masses seem to be telling me I am wrong about that :)

/me sends a '30 days no news' nag, continues to wait patiently.

Regards,
-- 
Christopher Schmidt
Web Developer
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Traian Stanev]

I think many people have it backwards. SDF is basically a reference 
implementation of the FDO API (or a significant subset thereof). It's not an 
independent file format that happens to have an FDO driver. Any C++ API for SDF 
would likely have to use FDO data structures and types underneath to get SDF 
data out of the file. This is because FDO data structures are directly 
serialized into SDF binary tables.

Traian


From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Christopher Schmidt 
[EMAIL PROTECTED]
Sent: Thursday, November 22, 2007 9:23 PM
To: OSGeo Discussions
Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,    open 
data format

On Thu, Nov 22, 2007 at 05:17:46PM -0800, Traian Stanev wrote:
> How about an OGR driver for SDF? No need to invent a new API when one already 
> exists.

I think that was exactly Bob's point: There is already an FDO driver for
SDF. If OGR is sufficient, I'm not entirely sure why FDO wouldn't be by
the same argument.

I'm with Mat, in feeling that it's not the path to the widest usage in
only FDO *or* OGR, and that a reference implementation outside of either
of those would result in possibly higher OEM-style integration into non
OGR/FDO products, and that a single shared implementation is the best
way for all developers to share a single effort when moving forward.

Regards,
--
Christopher Schmidt
Web Developer
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Christopher Schmidt]

On Thu, Nov 22, 2007 at 05:17:46PM -0800, Traian Stanev wrote:
> How about an OGR driver for SDF? No need to invent a new API when one already 
> exists.

I think that was exactly Bob's point: There is already an FDO driver for
SDF. If OGR is sufficient, I'm not entirely sure why FDO wouldn't be by
the same argument.

I'm with Mat, in feeling that it's not the path to the widest usage in
only FDO *or* OGR, and that a reference implementation outside of either
of those would result in possibly higher OEM-style integration into non
OGR/FDO products, and that a single shared implementation is the best
way for all developers to share a single effort when moving forward. 

Regards,
-- 
Christopher Schmidt
Web Developer
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Mateusz Loskot]

Traian Stanev wrote:
> How about an OGR driver for SDF? No need to invent a new API
> when one already exists.

Traian,

AFAIKU, that's the idea that triggered the debate
but it would be good if there is non-API specific library for SDF.
The the same library could be used by OGR, FDO, etc.

Cheers
-- 
Mateusz Loskot
http://mateusz.loskot.net
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Traian Stanev]

How about an OGR driver for SDF? No need to invent a new API when one already 
exists.

Traian


From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Mateusz Loskot [EMAIL 
PROTECTED]
Sent: Thursday, November 22, 2007 2:36 PM
To: OSGeo Discussions
Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
data format

Robert Bray wrote:
> Chris,
>
> I agree and will see what I can do to make it happen. If we want wider
> adoption it may also be beneficial to see some kind of C/C++ access
> library created for the format. In the past I always felt the FDO
> Provider was that library, but the masses seem to be telling me I am
> wrong about that :)

Bob,

Sounds like a very good idea to setup a libsdf project.
This way, it could be used by FDO, OGR and others.

Cheers
--
Mateusz Loskot
http://mateusz.loskot.net
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Mateusz Loskot]

Robert Bray wrote:
> Chris,
> 
> I agree and will see what I can do to make it happen. If we want wider
> adoption it may also be beneficial to see some kind of C/C++ access
> library created for the format. In the past I always felt the FDO
> Provider was that library, but the masses seem to be telling me I am
> wrong about that :)

Bob,

Sounds like a very good idea to setup a libsdf project.
This way, it could be used by FDO, OGR and others.

Cheers
-- 
Mateusz Loskot
http://mateusz.loskot.net
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Robert Bray]

Chris,

I agree and will see what I can do to make it happen. If we want wider 
adoption it may also be beneficial to see some kind of C/C++ access library 
created for the format. In the past I always felt the FDO Provider was that 
library, but the masses seem to be telling me I am wrong about that :)


Bob
- Original Message - 
From: "Christopher Schmidt" <[EMAIL PROTECTED]>

To: "OSGeo Discussions" 
Sent: Wednesday, November 21, 2007 9:04 PM
Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open 
data format




On Tue, Nov 13, 2007 at 11:18:42PM -0800, Robert Bray wrote:

Is it an open format?  ABSOLUTELY (we just never wrote a spec, but I am
willing to get it done)

All this said, I'd really like to understand everyones requirements for
this new format. If SDF fits thats great, if not thats ok too. We are
always happy to contribute what we can to the community.


I have an interest in seeing wider implementation of SDF. I have a user
who is interested in using FeatureServer against SDF data -- but I don't
have an environment in which I can build/use the FDO Python bindings
(being a mac user), so I can't help the user out.

It seems more likely that an alternative implementation to FDO's could be
created if documentation on the format was available. Obviously, there's
no guarenteee (and unfortunately, I'm not a very good coder, so I can't
offer much), but I'd at least like to be able to investigate how
difficult SDF might be to implement: not just for my own personal
reasons, but because implementation-difficulty can affect uptake of a
format as much as the technical benefits can (if not more).

so in response to 'I am willing to get it done', I would strongly
encourage you to do so!

Regards,
--
Christopher Schmidt
Web Developer
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss




___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Christopher Schmidt]

On Tue, Nov 13, 2007 at 11:18:42PM -0800, Robert Bray wrote:
> Is it an open format?  ABSOLUTELY (we just never wrote a spec, but I am 
> willing to get it done)
> 
> All this said, I'd really like to understand everyones requirements for 
> this new format. If SDF fits thats great, if not thats ok too. We are 
> always happy to contribute what we can to the community.

I have an interest in seeing wider implementation of SDF. I have a user
who is interested in using FeatureServer against SDF data -- but I don't
have an environment in which I can build/use the FDO Python bindings
(being a mac user), so I can't help the user out.

It seems more likely that an alternative implementation to FDO's could be 
created if documentation on the format was available. Obviously, there's
no guarenteee (and unfortunately, I'm not a very good coder, so I can't
offer much), but I'd at least like to be able to investigate how
difficult SDF might be to implement: not just for my own personal
reasons, but because implementation-difficulty can affect uptake of a
format as much as the technical benefits can (if not more).

so in response to 'I am willing to get it done', I would strongly
encourage you to do so!

Regards,
-- 
Christopher Schmidt
Web Developer
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Andy Robinson]

>From: [EMAIL PROTECTED] [mailto:discuss-
>[EMAIL PROTECTED] On Behalf Of Jo Walsh
>Sent: 18 November 2007 6:26 AM
>To: OSGeo Discussions
>Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
>data format
>
>dear Steve, all,
>On Tue, Nov 13, 2007 at 05:24:55PM +, Steve Coast wrote:
>> Real artists ship. For everyone else there's standards wanking.
>
>As the origins of the word 'yardstick' suggest, size is relative,
>and standards and wanking have always been intimately connected.
>http://www.etymonline.com/index.php?search=yardstick
>
>> this: We should have got a committee to design a standard, then we
>> could think about a committee to design an ontology... and choose a
>
>As http://wiki.openstreetmap.org/index.php/Map_Features states,
>"there is benefit in agreeing a recommended set of features and
>corresponding tags". This page is a beautiful work and has many
>interesting properties. But it's an ontology designed by a committee,
>the work of core contributors needing more structure to achieve their
>aims, than an completely open key-value tag system.
>

I'll point out that the original map features list was created by me and not
by committee. It was put together so that those with no experience of what
might be mapped would have some idea of where to start, remember that OSM is
not a project driven by the traditional cartography or GIS industries, for
most people joining the project it all needs to be learnt from the ground
up. Map Features was never intended to be a standard and hopefully it never
will be. Since its introduction in March 06 there have been plenty who have
advocated "standardising" it and a process of voting on proposed new tags to
join the Map features list still goes on by a minority, however in practice
if a tag is proposed and mappers find it useful they simply use it and
that's the way it should be. Voting only sees a small handful of responses
anyway. Principally it's a good way for discouraging poor tag design rather
than setting a standard.

>I'm impressed to see how many different language communities have
>worked to translate this page and explain its contents.
>Yet, italians, swedes and spaniards alike must use english-language
>key and value 'tags' in order to have their work show up on the map.
>Many annotations made before the "core recommended feature set" was
>fixed have been lost to perception; though it's technically possible
>to use a different system, adapt the rendering and annotation clients,
>then that work would be lost to re-use. A bit more structure would
>afford a lot more future adaptation and translation.

Map features was always intended to be an English orientated tagging list. I
always envisaged that other languages would develop their own tags and use
those for their local areas. I also expected (and still expect) to see other
tagging formats for other peculiar uses. Map features never included tagging
ideas for transport routing for instance. These haven't really happened much
yet, perhaps through a lack of confidence in breaking with tradition more
than anything else. The immediate gratification of rendered mapping (which
uses the longest established and most prolifically used tags) is also a big
driver. Hopefully though with time a lot more language specific tagging will
turn up in the data set.

>
>OSM is a brilliant project, borne of real need and social momentum.
>Meanwhile some corners of the standards industry really have gone off
>the rails and appear to be acting against common sense and user benefit.
>About the most depressing thing i heard at FOSS4G was, in the middle
>of an interesting talk, "we were going to implement cool feature X,
>but we're *waiting for the standard*".
>
>As i think you've pointed out in the past, "standards" like the core
>Map Features ought to emerge from common practise, from comparing
>different things that are shipped and running and being used.
>"Standards" that *do* work, like WMS and RSS, will get picked up.
>The core difference in approach seems to be a question of process.
>As Bitner pointed out, sometimes it makes sense to slow down a bit in
>order to let others catch up, so we can all go faster.
>

Bitner may have a point but I'm not so sure it holds true for OSM, which has
demonstrated that you just need to facilitate the creativity for great
progress to be made. Standards have not been the driver for making it a
success in the short time to date and each week we see something new and
exciting within the project without any reference to an agreed standard or
committee decision. If we stop and wait for everyone to catch up we will
surely stifle the cutting edge creativity from which the pr

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Jo Walsh]

dear Steve, all,
On Tue, Nov 13, 2007 at 05:24:55PM +, Steve Coast wrote:
> Real artists ship. For everyone else there's standards wanking.

As the origins of the word 'yardstick' suggest, size is relative,
and standards and wanking have always been intimately connected.
http://www.etymonline.com/index.php?search=yardstick
 
> this: We should have got a committee to design a standard, then we  
> could think about a committee to design an ontology... and choose a  

As http://wiki.openstreetmap.org/index.php/Map_Features states, 
"there is benefit in agreeing a recommended set of features and
corresponding tags". This page is a beautiful work and has many
interesting properties. But it's an ontology designed by a committee,
the work of core contributors needing more structure to achieve their
aims, than an completely open key-value tag system.

I'm impressed to see how many different language communities have
worked to translate this page and explain its contents. 
Yet, italians, swedes and spaniards alike must use english-language 
key and value 'tags' in order to have their work show up on the map.
Many annotations made before the "core recommended feature set" was
fixed have been lost to perception; though it's technically possible
to use a different system, adapt the rendering and annotation clients,
then that work would be lost to re-use. A bit more structure would
afford a lot more future adaptation and translation.

OSM is a brilliant project, borne of real need and social momentum.
Meanwhile some corners of the standards industry really have gone off
the rails and appear to be acting against common sense and user benefit.
About the most depressing thing i heard at FOSS4G was, in the middle
of an interesting talk, "we were going to implement cool feature X,
but we're *waiting for the standard*". 

As i think you've pointed out in the past, "standards" like the core
Map Features ought to emerge from common practise, from comparing
different things that are shipped and running and being used. 
"Standards" that *do* work, like WMS and RSS, will get picked up.
The core difference in approach seems to be a question of process. 
As Bitner pointed out, sometimes it makes sense to slow down a bit in
order to let others catch up, so we can all go faster.


jo
--
 
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Lucena, Ivan]

Frank,

I was watching this PyTables video 
[http://www.carabos.com/videos/pytables-1-intro] and one thought came to 
my mind: HDF5 can easily be used to store and retrieve vector, raster 
and attribute tables. We would need to standardize a schema tough.


Best regards,

Ivan

PS. I am not that Ivan on that video.

Frank Warmerdam wrote:

Landon Blake wrote:

P.S. - This is probably a crazy idea, but has anyone ever considered
talking to ESRI about cooperating on an update Shapefile spec?


Landon,

I believe ESRI sees the "file based geodatabase" as filling roughly the
role that the Shapefile played in the ArcView 3.x days.  Of course it isn't
clear that it will be very open and interoperable with other software 
packages.


I can't imagine a "upgrade" to shapefiles that would satisfy all our
requirements that has anything but the name in common with the existing
shapefiles.  So why drag ESRI into it?

Best regards,

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Frank Warmerdam]

Landon Blake wrote:

P.S. - This is probably a crazy idea, but has anyone ever considered
talking to ESRI about cooperating on an update Shapefile spec?


Landon,

I believe ESRI sees the "file based geodatabase" as filling roughly the
role that the Shapefile played in the ArcView 3.x days.  Of course it isn't
clear that it will be very open and interoperable with other software packages.

I can't imagine a "upgrade" to shapefiles that would satisfy all our
requirements that has anything but the name in common with the existing
shapefiles.  So why drag ESRI into it?

Best regards,
--
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, [EMAIL PROTECTED]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| President OSGeo, http://osgeo.org

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Landon Blake]

Puneet wrote: "Shall we focus on the technical limitations of Shapefiles
in order to keep moving forward?"

I was going to add a couple of limitations in addition to the
limitations of attribute data brought on by use of a DBF file as a
storage container:

[1] No way to store simple topology.
[2] No way to store features with different schema and/or different
geometry representations in the same file.
[3] No way to store labeling or annotation.

I'm not sure if you would want to do all of this with a new file format,
but I know its limitations of the Shapefile format that I have run into
in the past.

Landon

P.S. - This is probably a crazy idea, but has anyone ever considered
talking to ESRI about cooperating on an update Shapefile spec?

-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of P Kishor
Sent: Wednesday, November 14, 2007 9:06 AM
To: [EMAIL PROTECTED]
Cc: OSGeo Discussions
Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
data format

On 11/14/07, David William Bitner <[EMAIL PROTECTED]> wrote:
>
>
> >
> >
> >
> > I never (I think I never did) argued that Shapefile is not open. I
> > argued that it is not Free. I could be wrong.
> >
> >
> > >
> Here's the open published specification:
> http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf

yes, I am aware of this, and have used it on and off over the past
decade.

>
> Do what you will with it.  I don't know in this case what you imply by
Free.
>  Might ESRI make changes in the future that they don't publish?
Maybe, but
> at this version of shapefiles -- that are pervasive throughout the
industry,
> I would doubt they would forgo backwards compatibility and this
version of
> the specification is out there and free and open as far as I can tell.
>

ok, so there may or may not be an issue. I did bring it up, and so I
risk the discussion focusing on this aspect too much. Let's forget
about this aspect for now.

Shall we focus on the technical limitations of Shapefiles in order to
keep moving forward?

Many thanks,

Puneet.
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss


Warning:
Information provided via electronic media is not guaranteed against defects 
including translation and transmission errors. If the reader is not the 
intended recipient, you are hereby notified that any dissemination, 
distribution or copying of this communication is strictly prohibited. If you 
have received this information in error, please notify the sender immediately.
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[P Kishor]

On 11/14/07, David William Bitner <[EMAIL PROTECTED]> wrote:
>
>
> >
> >
> >
> > I never (I think I never did) argued that Shapefile is not open. I
> > argued that it is not Free. I could be wrong.
> >
> >
> > >
> Here's the open published specification:
> http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf

yes, I am aware of this, and have used it on and off over the past decade.

>
> Do what you will with it.  I don't know in this case what you imply by Free.
>  Might ESRI make changes in the future that they don't publish?  Maybe, but
> at this version of shapefiles -- that are pervasive throughout the industry,
> I would doubt they would forgo backwards compatibility and this version of
> the specification is out there and free and open as far as I can tell.
>

ok, so there may or may not be an issue. I did bring it up, and so I
risk the discussion focusing on this aspect too much. Let's forget
about this aspect for now.

Shall we focus on the technical limitations of Shapefiles in order to
keep moving forward?

Many thanks,

Puneet.
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Michael P. Gerlek]

> I find two problems with Shapefiles -- one, that it is not in public
> domain (I am not even sure of what licensing there is on it), and
> while ESRI is not likely to pull a Unisys on us, it just is
> philosophically better to free if possible. 

I don't see this as an issue at all -- legally speaking I don't think
ESRI has any grounds to attack anyone for using or implementing the
Shapefile format.  (and from a market standpoint too, it'd be a bad move
for them...)

> The second, more major
> problem is Shapefile's antiquated data technology. DBF is a royal pain
> in the derierre with its 10 char column name, no relational tables, no
> indexing of data constraints. The geometry part of Shapefiles seems to
> be pretty good and adequate; it is the data part that is the problem.

This, though, is more interesting.  There are some things in the
shapefile format that limit me today, and so I'm interested in
alternatives.

-mpg
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[David William Bitner]

>
>
> I never (I think I never did) argued that Shapefile is not open. I
> argued that it is not Free. I could be wrong.
>
> >
>
Here's the open published specification:
http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf
Do what you will with it.  I don't know in this case what you imply by
Free.  Might ESRI make changes in the future that they don't publish?
Maybe, but at this version of shapefiles -- that are pervasive throughout
the industry, I would doubt they would forgo backwards compatibility and
this version of the specification is out there and free and open as far as I
can tell.
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[P Kishor]

On 11/14/07, Frank Warmerdam <[EMAIL PROTECTED]> wrote:
> P Kishor wrote:
> > I find two problems with Shapefiles -- one, that it is not in public
> > domain (I am not even sure of what licensing there is on it), and
> > while ESRI is not likely to pull a Unisys on us, it just is
> > philosophically better to free if possible.
>
> Puneet,
>
> I think this is a red herring.  A file format is not normally considered
> copyrightable.  Generally speaking the only way to control a file format
> is through patents (for instance on compression methods or some fancy
> spatial indexing) and through trademarks on the format name.  There is
> no apparent such issues with shapefiles.  While the Shapefile format may
> not have been developed by an open process, nor is it (as far as I know)
> a dejure standard, but it is as open as any format.

I never (I think I never did) argued that Shapefile is not open. I
argued that it is not Free. I could be wrong.

>
> I think we need to focus on the technical failings of existing formats,
> and ensure a new format is widely usable (no strings attached).

Very rightly so. In my view Shapefile lacks robust attribute data
storage and retrieval capabilities. Others have come up with different
deficiencies as well.

>
> There were also questions about SDF format.  As I see it, one failing of the
> SDF format is that there isn't a published specification (that I'm aware of)
> and it is a complex enough format that it would be challenging to build a
> complete distinct library for the format (in Java for instance).  Also, the
> format is really an evolving format.  But, I still think SDF is a good
> candidate.
>
> I think, so far, there isn't a strong enough motivation in the community
> to develop a new format that does everything I want.  So (I think) a
> "green field" design is unlikely to get past the "standards wanking"
> stage that SteveC always goes out of his way to deride.
>


And, if the "talk" about creating a new format dies, or doesn't get
past beyond the talk stage, because there isn't a strong motivation
for it then so be it. At least we got the discussion going, and at
least more than a few chimed up with what they would like to see in a
new open AND free data format.

Perhaps, as someone suggested on the list, maybe someone will get it
in his/her head to work on one, and when it gets to the
"showing-to-the-public" stage, they will show it. Maybe it will
matter, maybe it won't.

At the very least we ended up with a wiki page full of desired specs. :-)


-- 
Puneet Kishor
http://punkish.eidesis.org/
Nelson Institute for Environmental Studies
http://www.nelson.wisc.edu/
Open Source Geospatial Foundation (OSGeo)
http://www.osgeo.org/
Summer 2007 S&T Policy Fellow, The National Academies
http://www.nas.edu/
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Frank Warmerdam]

P Kishor wrote:

I find two problems with Shapefiles -- one, that it is not in public
domain (I am not even sure of what licensing there is on it), and
while ESRI is not likely to pull a Unisys on us, it just is
philosophically better to free if possible. 


Puneet,

I think this is a red herring.  A file format is not normally considered
copyrightable.  Generally speaking the only way to control a file format
is through patents (for instance on compression methods or some fancy
spatial indexing) and through trademarks on the format name.  There is
no apparent such issues with shapefiles.  While the Shapefile format may
not have been developed by an open process, nor is it (as far as I know)
a dejure standard, but it is as open as any format.

I think we need to focus on the technical failings of existing formats,
and ensure a new format is widely usable (no strings attached).

There were also questions about SDF format.  As I see it, one failing of the
SDF format is that there isn't a published specification (that I'm aware of)
and it is a complex enough format that it would be challenging to build a
complete distinct library for the format (in Java for instance).  Also, the
format is really an evolving format.  But, I still think SDF is a good
candidate.

I think, so far, there isn't a strong enough motivation in the community
to develop a new format that does everything I want.  So (I think) a
"green field" design is unlikely to get past the "standards wanking"
stage that SteveC always goes out of his way to deride.

Best regards,
--
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, [EMAIL PROTECTED]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| President OSGeo, http://osgeo.org

___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[P Kishor]

disclaimer: I speak from a position of complete ignorance about SDF,
so everything I might say could be wrong.

On 11/14/07, Robert Bray <[EMAIL PROTECTED]> wrote:
> A little clarification on SDF, since it comes up once or twice in this
> thread. Jason's earlier descriptions of it's capabilities are pretty good.
> It supports multiple Feature Classes / Tables per file, strongly typed
> properties, multiple geometry properties per class, with seperate R-Trees
> for each geometry property. All of this is stored in a single file that is
> heavily optimized for spatial reads. The SDF FDO Provider suppports a
> multiple reader / single writer model. The geometries themselves include
> simple features + circular arcs in 2D, 2D with Z, 2D with M, or 2D with Z &
> M. The R-Trees are currently 2D.
>

SDF does sound the cat's meow.

> Is it an open format?  ABSOLUTELY (we just never wrote a spec, but I am
> willing to get it done)
>

What is the licensing of this format? What is the plan for licensing
of this format? Is the format going to be put out in public domain
ever? This is an important discussion, but it probably doesn't belong
in this thread. I bring it up here just to make sure it is not
forgotten.

> Another little known fact is that in the process of creating SDF we
> (Autodesk) literally wrote the code three times. The first time we built SDF
> on BerkleyDB, a great open source project but it has some fairly significant
> license fees for using it in a proprietary product. The second time we wrote
> it on SQLite, however the performance penalaty of the Relational layer was
> significant (read orders of magnitude). The third time we chose to strip
> away the SQLite relational layer and built directly on the SQLite Backend
> components (B-Tree, Pager, and OS Interface). In the end the third
> implementation actually turned out to be faster than BDB and is the one we
> use today.
>

By not having the relational layer we lose a lot in terms of attribute
data handling, or is that still somehow preserved in SDF?

Keep in mind, SQLite has gone through major revisions, some of them
bringing huge speed bumps in the process. That said, different folks
have different evaluation critieria -- I believe it was Steve W who
said that pretty much all he cared about was speed. For him it seems
SDF might work very well. For me, I don't mind sacrificing some speed
for ease and flexibility of related data storage, querying, and
retrieval. How can we achieve both?

> All this said, I'd really like to understand everyones requirements for this
> new format. If SDF fits thats great, if not thats ok too. We are always
> happy to contribute what we can to the community.

Thanks Bob. As you can see, we are still articulating the
requirements, and while we might be accused of requirement wanking,
design by committee might be better than no design at all.

I find two problems with Shapefiles -- one, that it is not in public
domain (I am not even sure of what licensing there is on it), and
while ESRI is not likely to pull a Unisys on us, it just is
philosophically better to free if possible. The second, more major
problem is Shapefile's antiquated data technology. DBF is a royal pain
in the derierre with its 10 char column name, no relational tables, no
indexing of data constraints. The geometry part of Shapefiles seems to
be pretty good and adequate; it is the data part that is the problem.

>
> Bob
>
> - Original Message -
> From: "Michael P. Gerlek" <[EMAIL PROTECTED]>
> To: <[EMAIL PROTECTED]>; "OSGeo Discussions" ;
> <[EMAIL PROTECTED]>
> Sent: Tuesday, November 13, 2007 12:23 PM
> Subject: RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
> data format
>
>
> >> > Regarding the suggestion that MapServer takes on this new format as
> > the
> >> > primary format:  I think this is way beyond the scope of what OSGeo
> > should
> >> > be doing.
> >
> > I agree with bitnerd.  If the MapServer team thinks this is a valuable
> > and worthwhile format, they will adopt it at some point.  It would not
> > be unreasonable for them to step back and see how thing progresses
> > before deciding to spend their valuable ergs on it.  The burden is on
> > the "OpenShape" people to show the idea is worthwhile and meritorious.
> >
> > (My two cents on the "standards" question: OSGeo is not a standards
> > organization, but / however / on the other hand / nonetheless one of the
> > reasons OSGeo exists is to foster such collaborations.  If some people
> > want to try and develop something new like this, I'm all in favor of
> > OSGeo offering mailing list and wiki space to help out. 

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Robert Bray]

These items are also supported in SDF today:

- SRS is a must, complete projection parameters (stored as WKT).
- full set of attibute types (c, int2, int4, int8, float, double,
 txt (i18l), date and timestamps (with gmt precise offset), blobs, ...
- indexing (unique and non-unique) on key attributes

The only exception is that I do not believe SDF supports the blob data type. 
There is also support for range constraints.


Bob

- Original Message - 
From: "Paul Spencer" <[EMAIL PROTECTED]>

To: "OSGeo Discussions" 
Sent: Wednesday, November 14, 2007 5:24 AM
Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open 
data format





On 14-Nov-07, at 7:20 AM, Christopher Schmidt wrote:


- optional coloring and styles, break values, rendering and
 scale limits, persistent joins or relates, color ramp, ...


are things which are provided by SLD and the like, which means that  you
really want SDF + WMC -- I don't think that this is SDF's job, and I
don't think that it should be the job of any geodata format.


mapinfo files come with embedded styling so there is at least one  format 
out there that combines the two.  KML arguably does the same  thing.


I think it would be worth exploring the merits of providing the  ability 
to embed styling information.  I'm not sure if it is useful to  place 
styling 'hints' in metadata, assuming that the format supports  arbitrary 
metadata.


Cheers

Paul

+-+
|Paul Spencer  [EMAIL PROTECTED]|
+-+
|Chief Technology Officer |
|DM Solutions Group Inchttp://www.dmsolutions.ca/ |
+-+





___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss




___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[P Kishor]

On 11/14/07, Christopher Schmidt <[EMAIL PROTECTED]> wrote:
> On Wed, Nov 14, 2007 at 04:12:21AM -0600, [EMAIL PROTECTED] wrote:
> > I would just love to click open and see something nice, specially if
> > someone has already taken the time to make it beautiful.
> >
> > Think of it as the output of a word processor instead of an editor.
> > Excel vs. VisiCalc; and the list may go on forever.
> >
> > All this is to say: not only the data needs a better container, the maps
> > as well.
>
> A data format is not a 'container' for maps, it's a container for data.
> The container for your maps should be something like WMC (though I
> don't know if it makes sense off the web): grouping together datasources
> and SLD and the like. The first half of the things you identified:
>
> > - optional space for metadata,
> > - optional thumnails (in 2 sizes: thumb and browse)
> > - optional embeded symbols (truetype/svg glyphs) and prefered layout
> > - optional coloring and styles, break values, rendering and
> >   scale limits, persistent joins or relates, color ramp, ...
>
> are things which are provided by SLD and the like, which means that you
> really want SDF + WMC -- I don't think that this is SDF's job, and I
> don't think that it should be the job of any geodata format.

>From a pure data approach, I agree that "how it looks" does not belong
in "what it is." Data should be completely separate from the
presentation.

There is a part of me, though, that believes that embedding some
default styling in the data container would be very useful. Apple's
Quicklook is a great technology that allows one to "quickly look" at
the data before deciding what to do with it. So, at there is some
precedence for this.

This is how it could work -- if any styling is provided from outside,
the internal styling would be ignored. However, if nothing is provided
from outside, then the default styling as determined by the creator
would be used.

Being able to click on the data package and see a preview would be fantastic.

Built in containers for metadata and SRS would be absolutely essential.

Manolo, could you please add these to the wiki page that David set up,
else I will add it but not until tomorrow or so.

>
> Regards,
> --
> Christopher Schmidt
> Web Developer
> ___
> Discuss mailing list
> Discuss@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
>


-- 
Puneet Kishor
http://punkish.eidesis.org/
Nelson Institute for Environmental Studies
http://www.nelson.wisc.edu/
Open Source Geospatial Foundation (OSGeo)
http://www.osgeo.org/
Summer 2007 S&T Policy Fellow, The National Academies
http://www.nas.edu/
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Paul Spencer]

On 14-Nov-07, at 7:20 AM, Christopher Schmidt wrote:


- optional coloring and styles, break values, rendering and
 scale limits, persistent joins or relates, color ramp, ...


are things which are provided by SLD and the like, which means that  
you

really want SDF + WMC -- I don't think that this is SDF's job, and I
don't think that it should be the job of any geodata format.


mapinfo files come with embedded styling so there is at least one  
format out there that combines the two.  KML arguably does the same  
thing.


I think it would be worth exploring the merits of providing the  
ability to embed styling information.  I'm not sure if it is useful to  
place styling 'hints' in metadata, assuming that the format supports  
arbitrary metadata.


Cheers

Paul

+-+
|Paul Spencer  [EMAIL PROTECTED]|
+-+
|Chief Technology Officer |
|DM Solutions Group Inchttp://www.dmsolutions.ca/ |
+-+





___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Bart van den Eijnden (OSGIS)]

I agree that most of these things don't belong in a data format.

But for metadata it would really make sense to be in there IMHO, and
information about the projection/CRS as well. Data without metadata is
worthless anyway (in an ideal world ...).

For shapefiles one can compare this to the .prj projection file and the .xml
metadata file.

Best regards,
Bart

--
Bart van den Eijnden
OSGIS, Open Source GIS
http://www.osgis.nl


- Oorspronkelijk bericht 
Van: OSGeo Discussions 
Naar: OSGeo Discussions 
Onderwerp: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open
data format
Datum: 14/11/07 10:21

> On Wed, Nov 14, 2007 at 04:12:21AM -0600, [EMAIL PROTECTED]
wrote:
> > I would just love to click open and see something nice, specially if
> > someone has already taken the time to make it beautiful.
> > 
> > Think of it as the output of a word processor instead of an editor.
> > Excel vs. VisiCalc; and the list may go on forever.
> > 
> > All this is to say: not only the data needs a better container, the
maps
> > as well.
> 
> A data format is not a 'container' for maps, it's a container for data.
> The container for your maps should be something like WMC (though I
> don't know if it makes sense off the web): grouping together datasources
> and SLD and the like. The first half of the things you identified:
> 
> > - optional space for metadata,
> > - optional thumnails (in 2 sizes: thumb and browse)
> > - optional embeded symbols (truetype/svg glyphs) and prefered layout
> > - optional coloring and styles, break values, rendering and
> >   scale limits, persistent joins or relates, color ramp, ...
> 
> are things which are provided by SLD and the like, which means that you
> really want SDF + WMC -- I don't think that this is SDF's job, and I
> don't think that it should be the job of any geodata format.
> 
> Regards,
> -- 
> Christopher Schmidt
> Web Developer
> ___
> Discuss mailing list
> Discuss@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
> 
> 



___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Christopher Schmidt]

On Wed, Nov 14, 2007 at 04:12:21AM -0600, [EMAIL PROTECTED] wrote:
> I would just love to click open and see something nice, specially if
> someone has already taken the time to make it beautiful.
> 
> Think of it as the output of a word processor instead of an editor.
> Excel vs. VisiCalc; and the list may go on forever.
> 
> All this is to say: not only the data needs a better container, the maps
> as well.

A data format is not a 'container' for maps, it's a container for data.
The container for your maps should be something like WMC (though I
don't know if it makes sense off the web): grouping together datasources
and SLD and the like. The first half of the things you identified:

> - optional space for metadata,
> - optional thumnails (in 2 sizes: thumb and browse)
> - optional embeded symbols (truetype/svg glyphs) and prefered layout
> - optional coloring and styles, break values, rendering and
>   scale limits, persistent joins or relates, color ramp, ...

are things which are provided by SLD and the like, which means that you
really want SDF + WMC -- I don't think that this is SDF's job, and I
don't think that it should be the job of any geodata format.

Regards,
-- 
Christopher Schmidt
Web Developer
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[mdragon]

SDF faster than dbd: wow!

I think the SDF very much solves the issue as Puneet put it, but I will
add to the wish list a few things which may be optional but certainly
usefull and valuable as time saver:

- optional space for metadata,
- optional thumnails (in 2 sizes: thumb and browse)
- optional embeded symbols (truetype/svg glyphs) and prefered layout
- optional coloring and styles, break values, rendering and
  scale limits, persistent joins or relates, color ramp, ...
- SRS is a must, complete projection parameters.
- full set of attibute types (c, int2, int4, int8, float, double,
  txt (i18l), date and timestamps (with gmt precise offset), blobs, ...
- Data range limits, enforced coherence, and presentation format/templates
- indexing (unique and non-unique) on key attributes
- could use integers or floats for coordinates
- uuid (unique id, directory/name independent)

I would just love to click open and see something nice, specially if
someone has already taken the time to make it beautiful.

Think of it as the output of a word processor instead of an editor.
Excel vs. VisiCalc; and the list may go on forever.

All this is to say: not only the data needs a better container, the maps
as well.

Manolo.


> A little clarification on SDF, since it comes up once or twice in this
> thread. Jason's earlier descriptions of it's capabilities are pretty good.
> It supports multiple Feature Classes / Tables per file, strongly typed
> properties, multiple geometry properties per class, with seperate R-Trees
> for each geometry property. All of this is stored in a single file that is
> heavily optimized for spatial reads. The SDF FDO Provider suppports a
> multiple reader / single writer model. The geometries themselves include
> simple features + circular arcs in 2D, 2D with Z, 2D with M, or 2D with Z
> &
> M. The R-Trees are currently 2D.
>
> Is it an open format?  ABSOLUTELY (we just never wrote a spec, but I am
> willing to get it done)
>
> Another little known fact is that in the process of creating SDF we
> (Autodesk) literally wrote the code three times. The first time we built
> SDF
> on BerkleyDB, a great open source project but it has some fairly
> significant
> license fees for using it in a proprietary product. The second time we
> wrote
> it on SQLite, however the performance penalaty of the Relational layer was
> significant (read orders of magnitude). The third time we chose to strip
> away the SQLite relational layer and built directly on the SQLite Backend
> components (B-Tree, Pager, and OS Interface). In the end the third
> implementation actually turned out to be faster than BDB and is the one we
> use today.
>
> All this said, I'd really like to understand everyones requirements for
> this
> new format. If SDF fits thats great, if not thats ok too. We are always
> happy to contribute what we can to the community.
>
> Bob
>


___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Robert Bray]

A little clarification on SDF, since it comes up once or twice in this 
thread. Jason's earlier descriptions of it's capabilities are pretty good. 
It supports multiple Feature Classes / Tables per file, strongly typed 
properties, multiple geometry properties per class, with seperate R-Trees 
for each geometry property. All of this is stored in a single file that is 
heavily optimized for spatial reads. The SDF FDO Provider suppports a 
multiple reader / single writer model. The geometries themselves include 
simple features + circular arcs in 2D, 2D with Z, 2D with M, or 2D with Z & 
M. The R-Trees are currently 2D.


Is it an open format?  ABSOLUTELY (we just never wrote a spec, but I am 
willing to get it done)


Another little known fact is that in the process of creating SDF we 
(Autodesk) literally wrote the code three times. The first time we built SDF 
on BerkleyDB, a great open source project but it has some fairly significant 
license fees for using it in a proprietary product. The second time we wrote 
it on SQLite, however the performance penalaty of the Relational layer was 
significant (read orders of magnitude). The third time we chose to strip 
away the SQLite relational layer and built directly on the SQLite Backend 
components (B-Tree, Pager, and OS Interface). In the end the third 
implementation actually turned out to be faster than BDB and is the one we 
use today.


All this said, I'd really like to understand everyones requirements for this 
new format. If SDF fits thats great, if not thats ok too. We are always 
happy to contribute what we can to the community.


Bob

- Original Message - 
From: "Michael P. Gerlek" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>; "OSGeo Discussions" ; 
<[EMAIL PROTECTED]>

Sent: Tuesday, November 13, 2007 12:23 PM
Subject: RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open 
data format




> Regarding the suggestion that MapServer takes on this new format as

the

> primary format:  I think this is way beyond the scope of what OSGeo

should

> be doing.


I agree with bitnerd.  If the MapServer team thinks this is a valuable
and worthwhile format, they will adopt it at some point.  It would not
be unreasonable for them to step back and see how thing progresses
before deciding to spend their valuable ergs on it.  The burden is on
the "OpenShape" people to show the idea is worthwhile and meritorious.

(My two cents on the "standards" question: OSGeo is not a standards
organization, but / however / on the other hand / nonetheless one of the
reasons OSGeo exists is to foster such collaborations.  If some people
want to try and develop something new like this, I'm all in favor of
OSGeo offering mailing list and wiki space to help out.  Declaring this
to be a "standard" effort, however, is probably premature in any case --
more useful at this point to see the idea sketched out further, see
who's interested, etc.)

-mpg
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss




___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Michael P. Gerlek]

> > Regarding the suggestion that MapServer takes on this new format as
the
> > primary format:  I think this is way beyond the scope of what OSGeo
should
> > be doing.

I agree with bitnerd.  If the MapServer team thinks this is a valuable
and worthwhile format, they will adopt it at some point.  It would not
be unreasonable for them to step back and see how thing progresses
before deciding to spend their valuable ergs on it.  The burden is on
the "OpenShape" people to show the idea is worthwhile and meritorious.

(My two cents on the "standards" question: OSGeo is not a standards
organization, but / however / on the other hand / nonetheless one of the
reasons OSGeo exists is to foster such collaborations.  If some people
want to try and develop something new like this, I'm all in favor of
OSGeo offering mailing list and wiki space to help out.  Declaring this
to be a "standard" effort, however, is probably premature in any case --
more useful at this point to see the idea sketched out further, see
who's interested, etc.)

-mpg
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[David William Bitner]

I have created a (now empty) space on the OSGeo wiki to start to fill in
concrete details that come out of this discussion at
http://wiki.osgeo.org/index.php/Geodata_formats.  Please use the wiki to put
your wishlists for a new open data format, lists of existing data formats
with links to their specifications etc in the wiki.  Please join the Geodata
Mailing list (http://www.osgeo.org/geodata) and continue this thread with
debate and discussion relating to a new format on that list as I believe it
is a more appropriate venue.

David

On Nov 13, 2007 12:55 PM, P Kishor <[EMAIL PROTECTED]> wrote:

> David,
>
>
> On 11/13/07, David William Bitner <[EMAIL PROTECTED]> wrote:
> > Part of the mission of the OSGeo Geodata committee
> > (http://www.osgeo.org/geodata) is to "promote the use of open geospatial
> > formats".  If there is a group that wants to continue pursuing the
> creation
> > of a new open geodata format, I would like to encourage the use of the
> > geodata mailing list. That being said, I think part of the discussion
> that
> > needs to be had is whether or not OSGeo should be creating standards in
> the
> > first place.
> >
> > A couple comments that I have on some of the discussion that has taken
> place
> > in this thread:
> >
> > Regarding the suggestion that MapServer takes on this new format as the
> > primary format:  I think this is way beyond the scope of what OSGeo
> should
> > be doing.  Even if we spec a new standard, we (OSGeo) have no teeth to
> be
> > able to make any of our projects do any kind of implementation of that
> > standard.  The choice of formats that are used by any of our projects is
> > driven by the needs of the users and developers and the resources (time,
> > money) that have been dedicated towards implementing them.  If someone
> takes
> > OpenShape or whatever and decides they have a business need that they
> can
> > spend the time or money to get it implemented then it will be
> implemented.
> > Shapefile has and will continue to be an important format for many
> projects
> > as it is one of, if not the most distributed formats in the GIS world.
> >
>
> I respectfully disagree. I think OSGeo has plenty teeth for those who
> want to believe in it. In the end, yes, just like any real project, it
> needs a core of committed developer and plenty of time (or money --
> usually they are synonymous). This is not something that can happen
> overnight, but if good, it deserves a start and support. That the
> long, long-term effects of a solid, relational, transactional, geodata
> format would be very good is a reasonable assumption for me.
>
> > Regarding the comments on standards wanking:  Standards can get in the
> way
> > of progress along a straight line, but they can also encourage
> > interoperability that can create better progress for everyone.  To get a
> > singular task done, standards often can slow things down, but there
> *are*
> > gains to be had from playing well with everyone else.
>
> Here I totally agree. I am not sure how to interpret the "standards
> wanking" statement. On the one hand it is a reasonably accurate
> assessment of a lot of public hand-wringing and open alliances (for a
> really funny take on this, read Fake Steve's tirade on the open
> handset alliance at
> ).
> But, on the other hand, it is a pretty damning judgment on any attempt
> to do things via collaboration, and thus, on OSGeo and such efforts
> itself.
>
> My take is that if I can't do it alone, I will lay it out in the open
> hoping someone better than me will work on it as well. If I can do it
> alone, I will do it until I think it is ready to benefit from extra
> eyeballs. Sometimes getting started is the biggest hurdle.
>
>
> >
> > David Bitner
> > OSGeo, Public Geospatial Data Project Chair
> >
> > On Nov 13, 2007 11:40 AM, Allan Doyle <[EMAIL PROTECTED]> wrote:
> > >
> > >
> > > On Nov 13, 2007, at 12:24 , Steve Coast wrote:
> > >
> > > > OSM: $0
> > > > CCBYSA: $0
> > > > Donation of entire Netherlands: Priceless
> > > >
> > > > Real artists ship. For everyone else there's standards wanking.
> > >
> > > Perhaps there's an art to wanking standards as well.
> > >
> > >
> > >
> > >
> > > >
> > > >
> > > >
> > > >
> > > > Seriously though, this is so kafka-esque. When OSM started it was
> > > > like this: We should have got a committee to design a standard, then
> > > > we could think about a committee to design an ontology... and choose
> > > > a name... and on some sunny distant day make a map.
> > > >
> > > >
> > > >
> > > > On 13 Nov 2007, at 17:09, P Kishor wrote:
> > > >
> > > >> On 11/13/07, Landon Blake <[EMAIL PROTECTED]> wrote:
> > > >>> Puneet,
> > > >>>
> > > >>> You wrote: "Should be easy to transition to. By building the new
> > > >>> format
> > > >>> on the
> > > >>> structure of the Shapefile format, and *in fact*, calling it "open
> > > >>> shapefiles" or some such thing, we indicate from 

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[P Kishor]

David,


On 11/13/07, David William Bitner <[EMAIL PROTECTED]> wrote:
> Part of the mission of the OSGeo Geodata committee
> (http://www.osgeo.org/geodata) is to "promote the use of open geospatial
> formats".  If there is a group that wants to continue pursuing the creation
> of a new open geodata format, I would like to encourage the use of the
> geodata mailing list. That being said, I think part of the discussion that
> needs to be had is whether or not OSGeo should be creating standards in the
> first place.
>
> A couple comments that I have on some of the discussion that has taken place
> in this thread:
>
> Regarding the suggestion that MapServer takes on this new format as the
> primary format:  I think this is way beyond the scope of what OSGeo should
> be doing.  Even if we spec a new standard, we (OSGeo) have no teeth to be
> able to make any of our projects do any kind of implementation of that
> standard.  The choice of formats that are used by any of our projects is
> driven by the needs of the users and developers and the resources (time,
> money) that have been dedicated towards implementing them.  If someone takes
> OpenShape or whatever and decides they have a business need that they can
> spend the time or money to get it implemented then it will be implemented.
> Shapefile has and will continue to be an important format for many projects
> as it is one of, if not the most distributed formats in the GIS world.
>

I respectfully disagree. I think OSGeo has plenty teeth for those who
want to believe in it. In the end, yes, just like any real project, it
needs a core of committed developer and plenty of time (or money --
usually they are synonymous). This is not something that can happen
overnight, but if good, it deserves a start and support. That the
long, long-term effects of a solid, relational, transactional, geodata
format would be very good is a reasonable assumption for me.

> Regarding the comments on standards wanking:  Standards can get in the way
> of progress along a straight line, but they can also encourage
> interoperability that can create better progress for everyone.  To get a
> singular task done, standards often can slow things down, but there *are*
> gains to be had from playing well with everyone else.

Here I totally agree. I am not sure how to interpret the "standards
wanking" statement. On the one hand it is a reasonably accurate
assessment of a lot of public hand-wringing and open alliances (for a
really funny take on this, read Fake Steve's tirade on the open
handset alliance at
).
But, on the other hand, it is a pretty damning judgment on any attempt
to do things via collaboration, and thus, on OSGeo and such efforts
itself.

My take is that if I can't do it alone, I will lay it out in the open
hoping someone better than me will work on it as well. If I can do it
alone, I will do it until I think it is ready to benefit from extra
eyeballs. Sometimes getting started is the biggest hurdle.


>
> David Bitner
> OSGeo, Public Geospatial Data Project Chair
>
> On Nov 13, 2007 11:40 AM, Allan Doyle <[EMAIL PROTECTED]> wrote:
> >
> >
> > On Nov 13, 2007, at 12:24 , Steve Coast wrote:
> >
> > > OSM: $0
> > > CCBYSA: $0
> > > Donation of entire Netherlands: Priceless
> > >
> > > Real artists ship. For everyone else there's standards wanking.
> >
> > Perhaps there's an art to wanking standards as well.
> >
> >
> >
> >
> > >
> > >
> > >
> > >
> > > Seriously though, this is so kafka-esque. When OSM started it was
> > > like this: We should have got a committee to design a standard, then
> > > we could think about a committee to design an ontology... and choose
> > > a name... and on some sunny distant day make a map.
> > >
> > >
> > >
> > > On 13 Nov 2007, at 17:09, P Kishor wrote:
> > >
> > >> On 11/13/07, Landon Blake <[EMAIL PROTECTED]> wrote:
> > >>> Puneet,
> > >>>
> > >>> You wrote: "Should be easy to transition to. By building the new
> > >>> format
> > >>> on the
> > >>> structure of the Shapefile format, and *in fact*, calling it "open
> > >>> shapefiles" or some such thing, we indicate from its name that the
> > >>> transition is not that revolutionary but is evolutionary. This,
> > >>> hopefully, will bring some name-familiarity, and make the transition
> > >>> less scary."
> > >>>
> > >>> I really think you are going to run into problems using the
> > >>> "Shapefile"
> > >>> as part of the trademark or name for any product not sold by ESRI. I
> > >>> strongly recommend against this move. Let people adopt the
> > >>> implementation of your idea for its merits, not for name recognition
> > >>> that comes from another product line.
> > >>
> > >> Good enough point to keep in mind, but not to get hung up over enough
> > >> to entangle us. Suggestions for names of the data format can be a
> > >> project in itself. "open spatial data format" or its variations could
> > >> be chosen. Still, point ta

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[David William Bitner]

Part of the mission of the OSGeo Geodata committee (
http://www.osgeo.org/geodata) is to "promote the use of open geospatial
formats".  If there is a group that wants to continue pursuing the creation
of a new open geodata format, I would like to encourage the use of the
geodata mailing list. That being said, I think part of the discussion that
needs to be had is whether or not OSGeo should be creating standards in the
first place.

A couple comments that I have on some of the discussion that has taken place
in this thread:

Regarding the suggestion that MapServer takes on this new format as the
primary format:  I think this is way beyond the scope of what OSGeo should
be doing.  Even if we spec a new standard, we (OSGeo) have no teeth to be
able to make any of our projects do any kind of implementation of that
standard.  The choice of formats that are used by any of our projects is
driven by the needs of the users and developers and the resources (time,
money) that have been dedicated towards implementing them.  If someone takes
OpenShape or whatever and decides they have a business need that they can
spend the time or money to get it implemented then it will be implemented.
Shapefile has and will continue to be an important format for many projects
as it is one of, if not the most distributed formats in the GIS world.

Regarding the comments on standards wanking:  Standards can get in the way
of progress along a straight line, but they can also encourage
interoperability that can create better progress for everyone.  To get a
singular task done, standards often can slow things down, but there *are*
gains to be had from playing well with everyone else.

David Bitner
OSGeo, Public Geospatial Data Project Chair
On Nov 13, 2007 11:40 AM, Allan Doyle <[EMAIL PROTECTED]> wrote:

>
> On Nov 13, 2007, at 12:24 , Steve Coast wrote:
>
> > OSM: $0
> > CCBYSA: $0
> > Donation of entire Netherlands: Priceless
> >
> > Real artists ship. For everyone else there's standards wanking.
>
> Perhaps there's an art to wanking standards as well.
>
> >
> >
> >
> >
> > Seriously though, this is so kafka-esque. When OSM started it was
> > like this: We should have got a committee to design a standard, then
> > we could think about a committee to design an ontology... and choose
> > a name... and on some sunny distant day make a map.
> >
> >
> >
> > On 13 Nov 2007, at 17:09, P Kishor wrote:
> >
> >> On 11/13/07, Landon Blake <[EMAIL PROTECTED]> wrote:
> >>> Puneet,
> >>>
> >>> You wrote: "Should be easy to transition to. By building the new
> >>> format
> >>> on the
> >>> structure of the Shapefile format, and *in fact*, calling it "open
> >>> shapefiles" or some such thing, we indicate from its name that the
> >>> transition is not that revolutionary but is evolutionary. This,
> >>> hopefully, will bring some name-familiarity, and make the transition
> >>> less scary."
> >>>
> >>> I really think you are going to run into problems using the
> >>> "Shapefile"
> >>> as part of the trademark or name for any product not sold by ESRI. I
> >>> strongly recommend against this move. Let people adopt the
> >>> implementation of your idea for its merits, not for name recognition
> >>> that comes from another product line.
> >>
> >> Good enough point to keep in mind, but not to get hung up over enough
> >> to entangle us. Suggestions for names of the data format can be a
> >> project in itself. "open spatial data format" or its variations could
> >> be chosen. Still, point taken.
> >>
> >>>
> >>> You wrote: "ANSI standard C is still
> >>> that magic common denominator that compiles and works predictably on
> >>> most number of systems. I have a lot against Java, but those who
> >>> love
> >>> Java should definitely work on tools for accessing and working with
> >>> this new format as it would only make the format more widely used
> >>> and
> >>> adopted."
> >>>
> >>> It sounds to me like you are really describing a tool. File
> >>> formats are
> >>> written in a binary encoding or text, not in a programming
> >>> language. If
> >>> you are designing a tool you can choose the programming language
> >>> of your
> >>> choice, but be aware that this will limit the developers that
> >>> adopt the
> >>> tool. This will be the case no matter what language you choose to
> >>> use,
> >>> whether it is C, Java, or something else.
> >>>
> >>> If, in contrast, you are creating a file format, then programming
> >>> languages shouldn't really matter. Binary and text data can be
> >>> accessed
> >>> by almost all programming languages.
> >>>
> >>> I think you need to decide if you want a tool or a data format. It
> >>> sounds like you are shooting more for a spatial database written
> >>> in the
> >>> C programming language that uses some form of the ESRI Shapefile
> >>> as its
> >>> underlying data storage mechanism. To me that is a tool or piece of
> >>> software, not a format. But maybe I don't completely understand your
> >>> goal.
> >>>
> >>
> >> well

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Allan Doyle]

On Nov 13, 2007, at 12:24 , Steve Coast wrote:


OSM: $0
CCBYSA: $0
Donation of entire Netherlands: Priceless

Real artists ship. For everyone else there's standards wanking.


Perhaps there's an art to wanking standards as well.






Seriously though, this is so kafka-esque. When OSM started it was  
like this: We should have got a committee to design a standard, then  
we could think about a committee to design an ontology... and choose  
a name... and on some sunny distant day make a map.




On 13 Nov 2007, at 17:09, P Kishor wrote:


On 11/13/07, Landon Blake <[EMAIL PROTECTED]> wrote:

Puneet,

You wrote: "Should be easy to transition to. By building the new  
format

on the
structure of the Shapefile format, and *in fact*, calling it "open
shapefiles" or some such thing, we indicate from its name that the
transition is not that revolutionary but is evolutionary. This,
hopefully, will bring some name-familiarity, and make the transition
less scary."

I really think you are going to run into problems using the  
"Shapefile"

as part of the trademark or name for any product not sold by ESRI. I
strongly recommend against this move. Let people adopt the
implementation of your idea for its merits, not for name recognition
that comes from another product line.


Good enough point to keep in mind, but not to get hung up over enough
to entangle us. Suggestions for names of the data format can be a
project in itself. "open spatial data format" or its variations could
be chosen. Still, point taken.



You wrote: "ANSI standard C is still
that magic common denominator that compiles and works predictably on
most number of systems. I have a lot against Java, but those who  
love

Java should definitely work on tools for accessing and working with
this new format as it would only make the format more widely used  
and

adopted."

It sounds to me like you are really describing a tool. File  
formats are
written in a binary encoding or text, not in a programming  
language. If
you are designing a tool you can choose the programming language  
of your
choice, but be aware that this will limit the developers that  
adopt the
tool. This will be the case no matter what language you choose to  
use,

whether it is C, Java, or something else.

If, in contrast, you are creating a file format, then programming
languages shouldn't really matter. Binary and text data can be  
accessed

by almost all programming languages.

I think you need to decide if you want a tool or a data format. It
sounds like you are shooting more for a spatial database written  
in the
C programming language that uses some form of the ESRI Shapefile  
as its

underlying data storage mechanism. To me that is a tool or piece of
software, not a format. But maybe I don't completely understand your
goal.



well, I am, frankly confused.

I was quite convinced I wasn't describing a "tool" but was describing
a "format." Of course, to describe the format, I positioned it on the
"format" (the SQLite-compatible format) used and popularized by a
"tool" (SQLite, the library, which happens to be written in C). In my
mind, having the data format based on SQLite *format* for its
relational attribute handling was the real winner. In that sense,
perhaps I conflated the format and the tool. I am not well versed in
these things to I am probably already walking on thin ice, but that
shouldn't stop others.

So, forget that I mentioned C and Java... let's just concentrate on a
way of laying out data on a disk that is not too dissimilar from how
Shapefile data are laid out, except that we utilize the
SQLite-compatible binary format for relational data handling, so that
SQLite-enabled spatial tools can access this new format.

And, put this format into public domain.




-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of P Kishor
Sent: Tuesday, November 13, 2007 8:35 AM
To: OSGeo Discussions
Subject: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
data format

Thanks everyone, for responding. Here is my "groundwork."

The new format --

- Should be fast. SQLite is plenty fast, and anything that simply
"extends" the Shapefile format to inject relational capabilities
should be pretty fast. It should definitely be faster than a
geodatabase format (such as PostGIS/ArcSDE) and perhaps even faster
than Shapefiles especially while accessing attribute data. DBF is
sequential, and searching for textual information is particularly
expensive. SQLite has been tuned to excellence. I have been working
with it for a few years now, and it really is an amazing product,
development community, support, and capabilities. That it is in  
public

domain makes for a transfat-free icing on the cake.

- Should be unencumbered by licenses and copyrights. Ideally, the  
new

format could also be put back into public domain. We want to remove
all encumbrances to encourage rapid and wide adoption.

- Should be a single file. Well, some like multiple

RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Jason Birch]

Landon wrote:
---
I really think you are going to run into problems using the "Shapefile"
as part of the trademark or name for any product not sold by ESRI. I
strongly recommend against this move.
---

I'm not a lawyer, but I really doubt that shapefile is unique enough to
be subject to trademark.  Initial searches of TESS don't show up
anything either.

That said, I don't think that shapefile should be used anywhere in an
open format name _because_ it's too generic and will cause confusion.

As my initial blog entry mentioned, I had (have?) pretty high hopes for
SDF.  It's extremely fast (native rtree), contains multiple tables in a
single file, supports complex geometries (arcs, aggregates, etc),
includes native schema definition, etc...  However... while some initial
marketing materials marked SDF as an open format, I have been unable to
obtain any clarification from Autodesk on the file specification or
openness of SDF.  As well, there are some limitations; in the quest for
speed, SDF is only using the SQLite table access mechanisms, not the
entire relational layer (or something like that).  

For reference, most of the interesting parts of the SDF Provider,
including the Rtree implementation, are available here:

http://tinyurl.com/2b5en2

Jason
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Steve Coast]

OSM: $0
CCBYSA: $0
Donation of entire Netherlands: Priceless

Real artists ship. For everyone else there's standards wanking.



Seriously though, this is so kafka-esque. When OSM started it was like  
this: We should have got a committee to design a standard, then we  
could think about a committee to design an ontology... and choose a  
name... and on some sunny distant day make a map.



On 13 Nov 2007, at 17:09, P Kishor wrote:


On 11/13/07, Landon Blake <[EMAIL PROTECTED]> wrote:

Puneet,

You wrote: "Should be easy to transition to. By building the new  
format

on the
structure of the Shapefile format, and *in fact*, calling it "open
shapefiles" or some such thing, we indicate from its name that the
transition is not that revolutionary but is evolutionary. This,
hopefully, will bring some name-familiarity, and make the transition
less scary."

I really think you are going to run into problems using the  
"Shapefile"

as part of the trademark or name for any product not sold by ESRI. I
strongly recommend against this move. Let people adopt the
implementation of your idea for its merits, not for name recognition
that comes from another product line.


Good enough point to keep in mind, but not to get hung up over enough
to entangle us. Suggestions for names of the data format can be a
project in itself. "open spatial data format" or its variations could
be chosen. Still, point taken.



You wrote: "ANSI standard C is still
that magic common denominator that compiles and works predictably on
most number of systems. I have a lot against Java, but those who love
Java should definitely work on tools for accessing and working with
this new format as it would only make the format more widely used and
adopted."

It sounds to me like you are really describing a tool. File formats  
are
written in a binary encoding or text, not in a programming  
language. If
you are designing a tool you can choose the programming language of  
your
choice, but be aware that this will limit the developers that adopt  
the
tool. This will be the case no matter what language you choose to  
use,

whether it is C, Java, or something else.

If, in contrast, you are creating a file format, then programming
languages shouldn't really matter. Binary and text data can be  
accessed

by almost all programming languages.

I think you need to decide if you want a tool or a data format. It
sounds like you are shooting more for a spatial database written in  
the
C programming language that uses some form of the ESRI Shapefile as  
its

underlying data storage mechanism. To me that is a tool or piece of
software, not a format. But maybe I don't completely understand your
goal.



well, I am, frankly confused.

I was quite convinced I wasn't describing a "tool" but was describing
a "format." Of course, to describe the format, I positioned it on the
"format" (the SQLite-compatible format) used and popularized by a
"tool" (SQLite, the library, which happens to be written in C). In my
mind, having the data format based on SQLite *format* for its
relational attribute handling was the real winner. In that sense,
perhaps I conflated the format and the tool. I am not well versed in
these things to I am probably already walking on thin ice, but that
shouldn't stop others.

So, forget that I mentioned C and Java... let's just concentrate on a
way of laying out data on a disk that is not too dissimilar from how
Shapefile data are laid out, except that we utilize the
SQLite-compatible binary format for relational data handling, so that
SQLite-enabled spatial tools can access this new format.

And, put this format into public domain.




-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of P Kishor
Sent: Tuesday, November 13, 2007 8:35 AM
To: OSGeo Discussions
Subject: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
data format

Thanks everyone, for responding. Here is my "groundwork."

The new format --

- Should be fast. SQLite is plenty fast, and anything that simply
"extends" the Shapefile format to inject relational capabilities
should be pretty fast. It should definitely be faster than a
geodatabase format (such as PostGIS/ArcSDE) and perhaps even faster
than Shapefiles especially while accessing attribute data. DBF is
sequential, and searching for textual information is particularly
expensive. SQLite has been tuned to excellence. I have been working
with it for a few years now, and it really is an amazing product,
development community, support, and capabilities. That it is in  
public

domain makes for a transfat-free icing on the cake.

- Should be unencumbered by licenses and copyrights. Ideally, the new
format could also be put back into public domain. We want to remove
all encumbrances to encourage rapid and wide adoption.

- Should be a single file. Well, some like multiple files and some
like single files. We can achieve both objectives by using a
tar-gzipped packaging such as Apple tends

RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Landon Blake]

Puneet,

Now I understand what you are talking about. You are using the format
that us used by SQLite, not SQLite itself. This makes a lot more sense,
as long as the SQLite format was designed in a logical manner.

Landon

-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of P Kishor
Sent: Tuesday, November 13, 2007 9:09 AM
To: OSGeo Discussions
Subject: Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
data format

On 11/13/07, Landon Blake <[EMAIL PROTECTED]> wrote:
> Puneet,
>
> You wrote: "Should be easy to transition to. By building the new
format
> on the
> structure of the Shapefile format, and *in fact*, calling it "open
> shapefiles" or some such thing, we indicate from its name that the
> transition is not that revolutionary but is evolutionary. This,
> hopefully, will bring some name-familiarity, and make the transition
> less scary."
>
> I really think you are going to run into problems using the
"Shapefile"
> as part of the trademark or name for any product not sold by ESRI. I
> strongly recommend against this move. Let people adopt the
> implementation of your idea for its merits, not for name recognition
> that comes from another product line.

Good enough point to keep in mind, but not to get hung up over enough
to entangle us. Suggestions for names of the data format can be a
project in itself. "open spatial data format" or its variations could
be chosen. Still, point taken.

>
> You wrote: "ANSI standard C is still
> that magic common denominator that compiles and works predictably on
> most number of systems. I have a lot against Java, but those who love
> Java should definitely work on tools for accessing and working with
> this new format as it would only make the format more widely used and
> adopted."
>
> It sounds to me like you are really describing a tool. File formats
are
> written in a binary encoding or text, not in a programming language.
If
> you are designing a tool you can choose the programming language of
your
> choice, but be aware that this will limit the developers that adopt
the
> tool. This will be the case no matter what language you choose to use,
> whether it is C, Java, or something else.
>
> If, in contrast, you are creating a file format, then programming
> languages shouldn't really matter. Binary and text data can be
accessed
> by almost all programming languages.
>
> I think you need to decide if you want a tool or a data format. It
> sounds like you are shooting more for a spatial database written in
the
> C programming language that uses some form of the ESRI Shapefile as
its
> underlying data storage mechanism. To me that is a tool or piece of
> software, not a format. But maybe I don't completely understand your
> goal.
>

well, I am, frankly confused.

I was quite convinced I wasn't describing a "tool" but was describing
a "format." Of course, to describe the format, I positioned it on the
"format" (the SQLite-compatible format) used and popularized by a
"tool" (SQLite, the library, which happens to be written in C). In my
mind, having the data format based on SQLite *format* for its
relational attribute handling was the real winner. In that sense,
perhaps I conflated the format and the tool. I am not well versed in
these things to I am probably already walking on thin ice, but that
shouldn't stop others.

So, forget that I mentioned C and Java... let's just concentrate on a
way of laying out data on a disk that is not too dissimilar from how
Shapefile data are laid out, except that we utilize the
SQLite-compatible binary format for relational data handling, so that
SQLite-enabled spatial tools can access this new format.

And, put this format into public domain.


>
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of P Kishor
> Sent: Tuesday, November 13, 2007 8:35 AM
> To: OSGeo Discussions
> Subject: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
> data format
>
> Thanks everyone, for responding. Here is my "groundwork."
>
> The new format --
>
> - Should be fast. SQLite is plenty fast, and anything that simply
> "extends" the Shapefile format to inject relational capabilities
> should be pretty fast. It should definitely be faster than a
> geodatabase format (such as PostGIS/ArcSDE) and perhaps even faster
> than Shapefiles especially while accessing attribute data. DBF is
> sequential, and searching for textual information is particularly
> expensive. SQLite has been tuned to excellence. I have been working
> with it for a few years now, and it really is an amazing product,
> development community, support, and capabi

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Paolo Cavallini]

Landon Blake ha scritto:

> I really think you are going to run into problems using the "Shapefile"
> as part of the trademark or name for any product not sold by ESRI.

This can be easily be overcome by using "OpenShape".
I think this is a good idea, as it will make transition psychologically
smoother.
pc
-- 
Paolo Cavallini, see: http://www.faunalia.it/pc



signature.asc
Description: OpenPGP digital signature
___
Discuss mailing list
Discuss@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss

Re: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[P Kishor]

On 11/13/07, Landon Blake <[EMAIL PROTECTED]> wrote:
> Puneet,
>
> You wrote: "Should be easy to transition to. By building the new format
> on the
> structure of the Shapefile format, and *in fact*, calling it "open
> shapefiles" or some such thing, we indicate from its name that the
> transition is not that revolutionary but is evolutionary. This,
> hopefully, will bring some name-familiarity, and make the transition
> less scary."
>
> I really think you are going to run into problems using the "Shapefile"
> as part of the trademark or name for any product not sold by ESRI. I
> strongly recommend against this move. Let people adopt the
> implementation of your idea for its merits, not for name recognition
> that comes from another product line.

Good enough point to keep in mind, but not to get hung up over enough
to entangle us. Suggestions for names of the data format can be a
project in itself. "open spatial data format" or its variations could
be chosen. Still, point taken.

>
> You wrote: "ANSI standard C is still
> that magic common denominator that compiles and works predictably on
> most number of systems. I have a lot against Java, but those who love
> Java should definitely work on tools for accessing and working with
> this new format as it would only make the format more widely used and
> adopted."
>
> It sounds to me like you are really describing a tool. File formats are
> written in a binary encoding or text, not in a programming language. If
> you are designing a tool you can choose the programming language of your
> choice, but be aware that this will limit the developers that adopt the
> tool. This will be the case no matter what language you choose to use,
> whether it is C, Java, or something else.
>
> If, in contrast, you are creating a file format, then programming
> languages shouldn't really matter. Binary and text data can be accessed
> by almost all programming languages.
>
> I think you need to decide if you want a tool or a data format. It
> sounds like you are shooting more for a spatial database written in the
> C programming language that uses some form of the ESRI Shapefile as its
> underlying data storage mechanism. To me that is a tool or piece of
> software, not a format. But maybe I don't completely understand your
> goal.
>

well, I am, frankly confused.

I was quite convinced I wasn't describing a "tool" but was describing
a "format." Of course, to describe the format, I positioned it on the
"format" (the SQLite-compatible format) used and popularized by a
"tool" (SQLite, the library, which happens to be written in C). In my
mind, having the data format based on SQLite *format* for its
relational attribute handling was the real winner. In that sense,
perhaps I conflated the format and the tool. I am not well versed in
these things to I am probably already walking on thin ice, but that
shouldn't stop others.

So, forget that I mentioned C and Java... let's just concentrate on a
way of laying out data on a disk that is not too dissimilar from how
Shapefile data are laid out, except that we utilize the
SQLite-compatible binary format for relational data handling, so that
SQLite-enabled spatial tools can access this new format.

And, put this format into public domain.


>
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of P Kishor
> Sent: Tuesday, November 13, 2007 8:35 AM
> To: OSGeo Discussions
> Subject: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
> data format
>
> Thanks everyone, for responding. Here is my "groundwork."
>
> The new format --
>
> - Should be fast. SQLite is plenty fast, and anything that simply
> "extends" the Shapefile format to inject relational capabilities
> should be pretty fast. It should definitely be faster than a
> geodatabase format (such as PostGIS/ArcSDE) and perhaps even faster
> than Shapefiles especially while accessing attribute data. DBF is
> sequential, and searching for textual information is particularly
> expensive. SQLite has been tuned to excellence. I have been working
> with it for a few years now, and it really is an amazing product,
> development community, support, and capabilities. That it is in public
> domain makes for a transfat-free icing on the cake.
>
> - Should be unencumbered by licenses and copyrights. Ideally, the new
> format could also be put back into public domain. We want to remove
> all encumbrances to encourage rapid and wide adoption.
>
> - Should be a single file. Well, some like multiple files and some
> like single files. We can achieve both objectives by using a
> tar-gzipped packaging such as Apple tends to use for much of its stuff
> (for example, its Pages wordprocessor uses a tgzipped xml file along
> with other resources for icons and pictures and stuff). Or, if speed
> is going to be affected because of gzipping and gunzipping, just a
> package format (I have no idea if this is a Unix thing or a Mac OS
> thing -- we, in the Mac world

RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[Landon Blake]

Puneet,

You wrote: "Should be easy to transition to. By building the new format
on the
structure of the Shapefile format, and *in fact*, calling it "open
shapefiles" or some such thing, we indicate from its name that the
transition is not that revolutionary but is evolutionary. This,
hopefully, will bring some name-familiarity, and make the transition
less scary."

I really think you are going to run into problems using the "Shapefile"
as part of the trademark or name for any product not sold by ESRI. I
strongly recommend against this move. Let people adopt the
implementation of your idea for its merits, not for name recognition
that comes from another product line.

You wrote: "ANSI standard C is still
that magic common denominator that compiles and works predictably on
most number of systems. I have a lot against Java, but those who love
Java should definitely work on tools for accessing and working with
this new format as it would only make the format more widely used and
adopted."

It sounds to me like you are really describing a tool. File formats are
written in a binary encoding or text, not in a programming language. If
you are designing a tool you can choose the programming language of your
choice, but be aware that this will limit the developers that adopt the
tool. This will be the case no matter what language you choose to use,
whether it is C, Java, or something else. 

If, in contrast, you are creating a file format, then programming
languages shouldn't really matter. Binary and text data can be accessed
by almost all programming languages.

I think you need to decide if you want a tool or a data format. It
sounds like you are shooting more for a spatial database written in the
C programming language that uses some form of the ESRI Shapefile as its
underlying data storage mechanism. To me that is a tool or piece of
software, not a format. But maybe I don't completely understand your
goal.

Landon


-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of P Kishor
Sent: Tuesday, November 13, 2007 8:35 AM
To: OSGeo Discussions
Subject: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
data format

Thanks everyone, for responding. Here is my "groundwork."

The new format --

- Should be fast. SQLite is plenty fast, and anything that simply
"extends" the Shapefile format to inject relational capabilities
should be pretty fast. It should definitely be faster than a
geodatabase format (such as PostGIS/ArcSDE) and perhaps even faster
than Shapefiles especially while accessing attribute data. DBF is
sequential, and searching for textual information is particularly
expensive. SQLite has been tuned to excellence. I have been working
with it for a few years now, and it really is an amazing product,
development community, support, and capabilities. That it is in public
domain makes for a transfat-free icing on the cake.

- Should be unencumbered by licenses and copyrights. Ideally, the new
format could also be put back into public domain. We want to remove
all encumbrances to encourage rapid and wide adoption.

- Should be a single file. Well, some like multiple files and some
like single files. We can achieve both objectives by using a
tar-gzipped packaging such as Apple tends to use for much of its stuff
(for example, its Pages wordprocessor uses a tgzipped xml file along
with other resources for icons and pictures and stuff). Or, if speed
is going to be affected because of gzipping and gunzipping, just a
package format (I have no idea if this is a Unix thing or a Mac OS
thing -- we, in the Mac world, call them packages... they appear like
files in the Finder, and like directories in the shell).

- Should be easy to transition to. By building the new format on the
structure of the Shapefile format, and *in fact*, calling it "open
shapefiles" or some such thing, we indicate from its name that the
transition is not that revolutionary but is evolutionary. This,
hopefully, will bring some name-familiarity, and make the transition
less scary.

- Frank mentions SQLite's lack of datatypes as an issue -- I guess
that is a matter of preference. I personally quite like that freedom
as it gives me, the application developer, complete control over what
goes where. SQLite actually does have now a few datatypes that it
respects, but doesn't complain about. Since all users will be
accessing the data via an application, as long as the application is
well defined, it should be fine.

- SQLite excels at one thing that it has been entrusted to do --
retrieve data that it has been entrusted with at extremely fast
speeds, and maintain ACID data integrity in case of a programmatic
catastrophe. The transactions themselves are worth their price of
admission, which, happily, happens to be zero.

- Langdon mentions Java support -- well, yes, use/work on SQLite JDBC.
I have been using it for a few days now and find it to be a pretty
competent conduit. Extend it, spatialize it. ANSI stan