Re: [OSM-dev] [Imports] Inserting OSM data
On 28 March 2010 19:09, Jaak Laineste wrote: > sample has total 52 fields: name and HTML descriptions in 3 languages, > photos, opening hours, contacts (email,web,post, phones), organization > details, tickets,access (wheelchair, public transport details), different > flags etc. Perhaps it is not good idea to add all of it to the OSM? There is already tags for most of the above, and I believe there is a dedicated mailing list for handicap access etc... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
On Sat, Mar 27, 2010 at 2:40 PM, Frederik Ramm wrote: > Jaak Laineste wrote: >> Is there a way to make maintenance >> of only their specific data in OSM easy? > > I think we're having different pictures here. OSM is not a platform > which you can use to host "your" data. The data ceases to become "yours" > once it is in OSM. If that fundamental idea doesn't get across, then > further discourse is fruitless. > > If they are willing to accept the paradigm shift and say: We used to be > the keeper of the official list of museums, but now we see ourselves as > a guardian of community museum data in OSM - checking it, refining it, > and making sure it doesn't get damaged - then there might be common > ground. If only there were some model for these museum enthusiasts to follow. Some model of interested parties curating information for the good of the community. Perhaps refining the information and presenting it in ways that attract the attention of the general public. Or perhaps carving away some layers of the information to show some interesting underlying structure that would not be immediately apparent to a non-expert. Oh! That model could be a museum curator! Except in OpenStreetMap, the curators neither lock the doors nor place the exhibits behind glass boxes and velvet ropes. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
Hi, Jaak Laineste wrote: > There could be quite good reasons to protect some of the data at least > temporarily. Let's look at them then. > Very technical reason: to avoid accidental deletion of nodes > during bulk import (which takes days sometimes). Happened to me - but to be honest, only because I was running the import in a stupid way. Today I would do a changeset upload which creates nodes and the ways using them - resulting in a transactional update on the server which means that nobody will see my nodes before my ways are in. > Well, maybe bulk import in > general is not really fully compatible the spirit of OSM after all. What is > more important purpose of OSM: is it the biggest outdoor mapping capturing > tool, or does it want to be the world largest and best community-created map > database? I wanted to discuss "private/locked data in OSM", not whether or not bulk imports are any good. I have an outspoken opinion on this but will save it for another thread. > My implicit > assumption was that OSM wants to be as good database as possible, but I > could also have totally missed the point of OSM. Your message is an example of good rhetorics, but not one of stringent logic. Imports may help OSM to become a better database, or they may be the ruin of the community - whatever one's opinion, it has nothing to do with locking data. > There are good datasources (from public sector) who have 80% of their data > open and in principle well compatible with OSM, but 20% of them should have > some protection. Technically splitting the data could be so complicated that > their only option now is not to share anything, i.e. just not to use OSM. The example you are about to give is an example of someone wanting protection for 100% of his data. > I have a particular example: a friend just called me, and he is in board of > national assiocation of museums. They have and maintain kind of official > database of all museums in the country. They wanted to have them on web map, > and I suggested to use OpenStreetMap, and not only as background image, but > also insert their data as points to the OSM. This bought me several > questions: > - is the only legitimate way to have one-time bulk import, and then just > hope that community will only improve it? Or could they have a bit more > special control (external IDs, notifications, soft locking of some tags etc) > over the data, at least to make their data maintenance easier. To enable > more automatic sync with their in-house data maintenance systems and > procedures. My view on this is very clear: I do not want data in OSM that I cannot edit. Un-editable data in OSM (basically a static copy of a data set maintained by someone else) increases the bulk of our data but doesn't improve the quality of OSM. It is dead easy to take such a data set and mix it in at rendering time, so if they want their official museums on a map, they need only configure Mapnik to load museum locations from their shape file and that's it. In fact, if *anyone* wants the "official" rather than the community maintained set of museums on the map, and assuming that the official list is available as a shape or GPX, it is absolutely no problem for anyone to mix them in from such a source at render time, and we should indeed strive to make this even easier so that people don't get the idea that everything they want to display on a map has to be imported to OSM first! If we allow such pseudo-imports to go ahead (i.e. where we import a copy of the data but we do not become the master), then we will end up with semi-maintained "shadows" of every Geodatabase in existence - we'll be the Geodata thrash heap for the world. For that specific case you mention I'd say either they keep their data out of OSM, or they set up something that helps them monitor changes to "their" data. If you remember, you initially raised four points, of which I discounted I-III as being un-OSM because they would protect the data from edits; IV was the idea of being able to better monitor things, which I said was a desire shared by many. My message to anyone contemplating imports into OSM is clear - either accept that others change the data (and you may monitor and you may change back if you have good reason), or don't import. > - Today the only way for them is anyway double maintenance: they maintain > their internal/primary database, and maybe they care to copy their > day-to-day updates manually also to OSM. Or of course just publish their data and let the OSM community do the rest. > Is there a way to make maintenance > of only their specific data in OSM easy? I think we're having different pictures here. OSM is not a platform which you can use to host "your" data. The data ceases to become "yours" once it is in OSM. If that fundamental idea doesn't get across, then further discourse is fruitless. If they are willing to accept the paradigm shift and say: We used to be th
Re: [OSM-dev] [Imports] Inserting OSM data
On 27 March 2010 10:28, Jaak Laineste wrote: > > > I have a particular example: a friend just called me, and he is in board > of > national assiocation of museums. They have and maintain kind of official > database of all museums in the country. They wanted to have them on web > map, > and I suggested to use OpenStreetMap, and not only as background image, but > also insert their data as points to the OSM. This bought me several > questions: > - is the only legitimate way to have one-time bulk import, and then just > hope that community will only improve it? Or could they have a bit more > special control (external IDs, notifications, soft locking of some tags > etc) > over the data, at least to make their data maintenance easier. To enable > more automatic sync with their in-house data maintenance systems and > procedures. > Do they not trust OSM to keep their museums in place. You could very easily write a script to query the xapi and return all museum nodes in an area/country. You could then save that data and have your script flag nodes that had moved (or moved more than so much). - Today the only way for them is anyway double maintenance: they maintain > their internal/primary database, and maybe they care to copy their > day-to-day updates manually also to OSM. Is there a way to make maintenance > of only their specific data in OSM easy? How often does the location of a museum change! If they were really worried about vandalism of OSM to vandalise their online map, then they could use OSM as the base image and put some marker icons on top. But that doesn't mean the museum nodes can't be imported to OSM one time anyway. > Also I cannot imagine this type of "once a month" users actually using > hard-core mapping beasts like JOSM JOSM is a mapping beast? Some people with their own data will find JOSM far easier despite a lack of OSM experience. I for one hate the learning curve on Potlatch and so remain near the bottom of it. , maximum what they could care to learn > would be somewhere in Potlatch (but without the roads!) / Mapzen level. > Having the roads scares them? Who are these people! When they drive a car, do they block out the side windows so they're not distracted by the pavement/sidewalk? -- Gregory o...@livingwithdragons.com http://www.livingwithdragons.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
Hi, Im just tagging on to this thread for the archives. As it was just sent to the imports@ list, but not dev@ list, It attempts to solve the ideas that jaak noted below. http://lists.openstreetmap.org/pipermail/talk-gb/2010-March/005544.html (I was going to respond last week on this thread) ... with 'OpenImportsMap' . Perhaps the wiki can be expanded, i added my notes tot he talk page of it, and will attempt to incorporate the ideas from this thread also. http://wiki.openstreetmap.org/wiki/Importing_Government_Data It was in response to: from steveC original post about Announce: openOS (Ordinance survey releasing data) http://lists.openstreetmap.org/pipermail/talk-gb/2010-March/005543.html Cheers, Sam On Sat, Mar 27, 2010 at 10:28 AM, Jaak Laineste wrote: > > > some kind of priority over the others. Like possibility to > > > lock/protect some parts of the data. It could be done in several > levels: > > > > > 1) my inserted data (tag, node, relation, way - any of them) can be > > > defined as private, nobody else can not even see it. > > > 2) my data is protected - you can see, but not modify > > > 3) my data has group privacy/protection under my control: I can give > > > view/ modify / delete permissions to specific users/groups > > > > These three are, in my opinion, not compatible with the spirit of OSM. > > If you want your own data, store it somewhere else ;-) > > There could be quite good reasons to protect some of the data at least > temporarily. Very technical reason: to avoid accidental deletion of nodes > during bulk import (which takes days sometimes). Well, maybe bulk import in > general is not really fully compatible the spirit of OSM after all. What is > more important purpose of OSM: is it the biggest outdoor mapping capturing > tool, or does it want to be the world largest and best community-created > map > database? If outdoor mapping is the primary aim, then right, the corporate > imports are not needed, maybe even need to be banned. If the database > contents and quality the is target, then the imports and database links > must > be as plentifyl, good and made contributor friendly as possible. My > implicit > assumption was that OSM wants to be as good database as possible, but I > could also have totally missed the point of OSM. Anyway, my preference is > that OSM aim is to be as good database as possible, and outdoor mapping is > just one of the great ways to create and update data. > > There are good datasources (from public sector) who have 80% of their data > open and in principle well compatible with OSM, but 20% of them should have > some protection. Technically splitting the data could be so complicated > that > their only option now is not to share anything, i.e. just not to use OSM. > > I have a particular example: a friend just called me, and he is in board > of > national assiocation of museums. They have and maintain kind of official > database of all museums in the country. They wanted to have them on web > map, > and I suggested to use OpenStreetMap, and not only as background image, but > also insert their data as points to the OSM. This bought me several > questions: > - is the only legitimate way to have one-time bulk import, and then just > hope that community will only improve it? Or could they have a bit more > special control (external IDs, notifications, soft locking of some tags > etc) > over the data, at least to make their data maintenance easier. To enable > more automatic sync with their in-house data maintenance systems and > procedures. > - Today the only way for them is anyway double maintenance: they maintain > their internal/primary database, and maybe they care to copy their > day-to-day updates manually also to OSM. Is there a way to make maintenance > of only their specific data in OSM easy? One complicated solution would be > to use JOSM+XAPI to make extracts based their own tags. But this is risky, > you can easily create reduntant data if you do not see the data around each > node. Also I cannot imagine this type of "once a month" users actually > using > hard-core mapping beasts like JOSM, maximum what they could care to learn > would be somewhere in Potlatch (but without the roads!) / Mapzen level. > > > Jaak > > > ___ > Imports mailing list > impo...@openstreetmap.org > http://lists.openstreetmap.org/listinfo/imports > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
At 2010-03-22 12:02, Nic Roets wrote: >... >Unfortunately it has already happened many times. Below is a list of >the third party identifiers that I have found. >... > > >hdop >sat >pdop >fix >vdop To pick a nit, these are likely not ids, but instead useful info provided via GPS. I recognize: hdop = Horizontal Dilution Of Position vdop = Vertical Dilution Of Position I wish my Garmin 60CSx provided these instead of making me record them manually :( -- Alan Mintz ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
I've long considered that it would be good to have a system whereby anyone modifying a specified subset of the database would be presented with a message for them to read before comitting. This would be implemented with a special key that all editing software would be encouraged to support. In this case the message would read "please don't delete my IDs" It would also be useful in instances of unintentional vandalism, or that case where there was an edit war over some change of road layout between a local that knew the ground and a remote user with out of date yahoo data. Just a thought. JR On 23 March 2010 11:57, jamesmikedup...@googlemail.com < jamesmikedup...@googlemail.com> wrote: > On Tue, Mar 23, 2010 at 12:50 PM, Frederik Ramm wrote: > > Jaak Laineste wrote: >> > 1) my inserted data (tag, node, relation, way - any of them) can be >> defined >> > as private, nobody else can not even see it. >> > 2) my data is protected - you can see, but not modify >> > 3) my data has group privacy/protection under my control: I can give >> view/ >> > modify / delete permissions to specific users/groups >> >> These three are, in my opinion, not compatible with the spirit of OSM. >> If you want your own data, store it somewhere else ;-) >> > > Yes, as soon as you license your data under CC-SA you are explicitly > allowing people to modify it. > I think the whole idea of putting the data in osm is not just for making a > pretty picture, but for contributing data for others to build upon. > > mike > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
On Tue, Mar 23, 2010 at 12:50 PM, Frederik Ramm wrote: > Jaak Laineste wrote: > > 1) my inserted data (tag, node, relation, way - any of them) can be > defined > > as private, nobody else can not even see it. > > 2) my data is protected - you can see, but not modify > > 3) my data has group privacy/protection under my control: I can give > view/ > > modify / delete permissions to specific users/groups > > These three are, in my opinion, not compatible with the spirit of OSM. > If you want your own data, store it somewhere else ;-) > Yes, as soon as you license your data under CC-SA you are explicitly allowing people to modify it. I think the whole idea of putting the data in osm is not just for making a pretty picture, but for contributing data for others to build upon. mike ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
Hi, Jaak Laineste wrote: > Just for the > future APIs I suggest to give to original importer (or just editor) some > kind of priority over the others. Like possibility to lock/protect some > parts of the data. It could be done in several levels: > 1) my inserted data (tag, node, relation, way - any of them) can be defined > as private, nobody else can not even see it. > 2) my data is protected - you can see, but not modify > 3) my data has group privacy/protection under my control: I can give view/ > modify / delete permissions to specific users/groups These three are, in my opinion, not compatible with the spirit of OSM. If you want your own data, store it somewhere else ;-) > 4) I can subscribe to notifications for any modifications of my data (this > specific piece of it). So I could easily see and revert changes. This is widely demanded, and already partly implemented by XAPI. It is however not something that necessarily has to be a core service - anyone could implement it. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
On 22 March 2010 12:38, Andy Allan wrote: > On Mon, Mar 22, 2010 at 7:12 PM, Ian Dees wrote: > > > Almost 100% of the time these are imported to allow for the possibility > of > > future updates to the existing imports. > > Except, as you point out, they can't be used in any way for "future > updates" since you've got no idea if the reference stays on the > correct object. So I'd rather spend time explaining to people that > external references are a waste of time than encouraging them any > further. > > > In order to have this possibility, > > there needs to be a consistent/persistent identifier for a particular > > "thing" on the map. Since we don't have support for that in OSM this is > the > > next best thing. > > > > Anyway, I'm not advocating for or against anything, simply explaining > what > > I've experienced as an importer and someone helping others import. > > Has anyone experienced sync working? I know a lot of people have > imported with external IDs but I suspect that it's not going to work > when they actually come to updating one way or the other. > I remember being told it was very important for imports, such as NAPTAN, that can be updated. But I guess that is different, as they are actually the official organiser/referencer of bus stops (and 4th parties might even use that ref). Back to the original poster, Andreas. I am still unclear about some things, are you wanting to bring the data back out of OSM to use in your system/company? If we could maybe look at how we could help you with that then we might be able to understand the best way for you to add to OSM. It would be interesting to know the rough geographic area your data covers. We can then see if the area is near blank or has some dedicated mappers that will be keen to help manually edit your import (e.g. check for imports). Some people here have also requested an extract of the data to look at the quality, or what it contains. Hopefully your not getting overwhelmed by the replies here. Welcome to the OSM community! -- Gregory o...@livingwithdragons.com http://www.livingwithdragons.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
Hi,
Andy Allan wrote:
> No, it's a distraction from the real problem, which is that merging
> datasets is fundamentally hard, and needs to be approached with
> sophistication and fuzzy location-based matching (e.g. feature church,
> near this point, name something close to "all saint(')s").
+1. It gets even more difficult for relations but people are actually
starting to use the /browse/relation/xyz URL as a permalink - lots of
times on the Wiki, but also externally. If I were just a little bit more
evil than I actually am, I'd make it a habit to delete and re-create a
few relations every day just to drive home the message that IDs are not
permanent. But I can offer them no alternative, currently.
> Has anyone experienced sync working? I know a lot of people have
> imported with external IDs but I suspect that it's not going to work
> when they actually come to updating one way or the other.
I think it has been successfully used in re-importing/updating opengeodb
data in Germany and maybe other parts of Europe. But these were just
place nodes. They even had a special tag, openGeoDB:auto_update, that
would list which tags would be eligible for later sync.
Bye
Frederik
--
Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33"
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
On Mon, Mar 22, 2010 at 7:12 PM, Ian Dees wrote:
> Almost 100% of the time these are imported to allow for the possibility of
> future updates to the existing imports.
Except, as you point out, they can't be used in any way for "future
updates" since you've got no idea if the reference stays on the
correct object. So I'd rather spend time explaining to people that
external references are a waste of time than encouraging them any
further.
> In order to have this possibility,
> there needs to be a consistent/persistent identifier for a particular
> "thing" on the map. Since we don't have support for that in OSM this is the
> next best thing.
No, it's a distraction from the real problem, which is that merging
datasets is fundamentally hard, and needs to be approached with
sophistication and fuzzy location-based matching (e.g. feature church,
near this point, name something close to "all saint(')s"). Slapping
external IDs on OSM data and crossing fingers isn't helping.
> Anyway, I'm not advocating for or against anything, simply explaining what
> I've experienced as an importer and someone helping others import.
Has anyone experienced sync working? I know a lot of people have
imported with external IDs but I suspect that it's not going to work
when they actually come to updating one way or the other.
Cheers,
Andy
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
On Mon, Mar 22, 2010 at 2:02 PM, Nic Roets wrote: > On Mon, Mar 22, 2010 at 6:18 PM, Tom Hughes wrote: > > On 22/03/10 15:00, Nic Roets wrote: > > > >> One solution is to add your own tag to the OSM files you generate e.g. > >> smartsoft_id=nnn. And publish the files for review somewhere. Then > >> scan the minutely updates at planet.openstreetmap.de for your id > >> numbers and when they appear you can delete them from your DB. > > > > I'm not sure we really want to encourage the use of our database as a > > storage space for third party identifiers like that. It could get rather > out > > of hand if everybody else wants to attach their own IDs to our data. > > > > Hello Tom, > > Unfortunately it has already happened many times. Below is a list of > the third party identifiers that I have found. > > Instead of banning them outright, I would be happy if we have > guidelines that (a) make it easy for us to identify them as third > party (b) make sure they don't clash with other imports and (c) make > sure they don't take up unnecessary disk space. Almost 100% of the time these are imported to allow for the possibility of future updates to the existing imports. In order to have this possibility, there needs to be a consistent/persistent identifier for a particular "thing" on the map. Since we don't have support for that in OSM this is the next best thing. If, when the import occurred, we were given an ID by the OSM API that always referenced that object into the future, then the entity performing the import could hang on to the link between the OSM "perma-id" and whatever the imported element was and perform updates if a new set of data was made available. Of course, this isn't possible right now because some other mapper could come along and split the imported way into several small pieces, delete it all-together, or rename it. Anyway, I'm not advocating for or against anything, simply explaining what I've experienced as an importer and someone helping others import. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
