Re: [OSM-dev] [Imports] Inserting OSM data
On 28 March 2010 19:09, Jaak Laineste j...@nutiteq.com wrote: sample has total 52 fields: name and HTML descriptions in 3 languages, photos, opening hours, contacts (email,web,post, phones), organization details, tickets,access (wheelchair, public transport details), different flags etc. Perhaps it is not good idea to add all of it to the OSM? There is already tags for most of the above, and I believe there is a dedicated mailing list for handicap access etc... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
Hi, Im just tagging on to this thread for the archives. As it was just sent to the imports@ list, but not dev@ list, It attempts to solve the ideas that jaak noted below. http://lists.openstreetmap.org/pipermail/talk-gb/2010-March/005544.html (I was going to respond last week on this thread) ... with 'OpenImportsMap' . Perhaps the wiki can be expanded, i added my notes tot he talk page of it, and will attempt to incorporate the ideas from this thread also. http://wiki.openstreetmap.org/wiki/Importing_Government_Data It was in response to: from steveC original post about Announce: openOS (Ordinance survey releasing data) http://lists.openstreetmap.org/pipermail/talk-gb/2010-March/005543.html Cheers, Sam On Sat, Mar 27, 2010 at 10:28 AM, Jaak Laineste j...@nutiteq.com wrote: some kind of priority over the others. Like possibility to lock/protect some parts of the data. It could be done in several levels: 1) my inserted data (tag, node, relation, way - any of them) can be defined as private, nobody else can not even see it. 2) my data is protected - you can see, but not modify 3) my data has group privacy/protection under my control: I can give view/ modify / delete permissions to specific users/groups These three are, in my opinion, not compatible with the spirit of OSM. If you want your own data, store it somewhere else ;-) There could be quite good reasons to protect some of the data at least temporarily. Very technical reason: to avoid accidental deletion of nodes during bulk import (which takes days sometimes). Well, maybe bulk import in general is not really fully compatible the spirit of OSM after all. What is more important purpose of OSM: is it the biggest outdoor mapping capturing tool, or does it want to be the world largest and best community-created map database? If outdoor mapping is the primary aim, then right, the corporate imports are not needed, maybe even need to be banned. If the database contents and quality the is target, then the imports and database links must be as plentifyl, good and made contributor friendly as possible. My implicit assumption was that OSM wants to be as good database as possible, but I could also have totally missed the point of OSM. Anyway, my preference is that OSM aim is to be as good database as possible, and outdoor mapping is just one of the great ways to create and update data. There are good datasources (from public sector) who have 80% of their data open and in principle well compatible with OSM, but 20% of them should have some protection. Technically splitting the data could be so complicated that their only option now is not to share anything, i.e. just not to use OSM. I have a particular example: a friend just called me, and he is in board of national assiocation of museums. They have and maintain kind of official database of all museums in the country. They wanted to have them on web map, and I suggested to use OpenStreetMap, and not only as background image, but also insert their data as points to the OSM. This bought me several questions: - is the only legitimate way to have one-time bulk import, and then just hope that community will only improve it? Or could they have a bit more special control (external IDs, notifications, soft locking of some tags etc) over the data, at least to make their data maintenance easier. To enable more automatic sync with their in-house data maintenance systems and procedures. - Today the only way for them is anyway double maintenance: they maintain their internal/primary database, and maybe they care to copy their day-to-day updates manually also to OSM. Is there a way to make maintenance of only their specific data in OSM easy? One complicated solution would be to use JOSM+XAPI to make extracts based their own tags. But this is risky, you can easily create reduntant data if you do not see the data around each node. Also I cannot imagine this type of once a month users actually using hard-core mapping beasts like JOSM, maximum what they could care to learn would be somewhere in Potlatch (but without the roads!) / Mapzen level. Jaak ___ Imports mailing list impo...@openstreetmap.org http://lists.openstreetmap.org/listinfo/imports ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
Hi, Jaak Laineste wrote: There could be quite good reasons to protect some of the data at least temporarily. Let's look at them then. Very technical reason: to avoid accidental deletion of nodes during bulk import (which takes days sometimes). Happened to me - but to be honest, only because I was running the import in a stupid way. Today I would do a changeset upload which creates nodes and the ways using them - resulting in a transactional update on the server which means that nobody will see my nodes before my ways are in. Well, maybe bulk import in general is not really fully compatible the spirit of OSM after all. What is more important purpose of OSM: is it the biggest outdoor mapping capturing tool, or does it want to be the world largest and best community-created map database? I wanted to discuss private/locked data in OSM, not whether or not bulk imports are any good. I have an outspoken opinion on this but will save it for another thread. My implicit assumption was that OSM wants to be as good database as possible, but I could also have totally missed the point of OSM. Your message is an example of good rhetorics, but not one of stringent logic. Imports may help OSM to become a better database, or they may be the ruin of the community - whatever one's opinion, it has nothing to do with locking data. There are good datasources (from public sector) who have 80% of their data open and in principle well compatible with OSM, but 20% of them should have some protection. Technically splitting the data could be so complicated that their only option now is not to share anything, i.e. just not to use OSM. The example you are about to give is an example of someone wanting protection for 100% of his data. I have a particular example: a friend just called me, and he is in board of national assiocation of museums. They have and maintain kind of official database of all museums in the country. They wanted to have them on web map, and I suggested to use OpenStreetMap, and not only as background image, but also insert their data as points to the OSM. This bought me several questions: - is the only legitimate way to have one-time bulk import, and then just hope that community will only improve it? Or could they have a bit more special control (external IDs, notifications, soft locking of some tags etc) over the data, at least to make their data maintenance easier. To enable more automatic sync with their in-house data maintenance systems and procedures. My view on this is very clear: I do not want data in OSM that I cannot edit. Un-editable data in OSM (basically a static copy of a data set maintained by someone else) increases the bulk of our data but doesn't improve the quality of OSM. It is dead easy to take such a data set and mix it in at rendering time, so if they want their official museums on a map, they need only configure Mapnik to load museum locations from their shape file and that's it. In fact, if *anyone* wants the official rather than the community maintained set of museums on the map, and assuming that the official list is available as a shape or GPX, it is absolutely no problem for anyone to mix them in from such a source at render time, and we should indeed strive to make this even easier so that people don't get the idea that everything they want to display on a map has to be imported to OSM first! If we allow such pseudo-imports to go ahead (i.e. where we import a copy of the data but we do not become the master), then we will end up with semi-maintained shadows of every Geodatabase in existence - we'll be the Geodata thrash heap for the world. For that specific case you mention I'd say either they keep their data out of OSM, or they set up something that helps them monitor changes to their data. If you remember, you initially raised four points, of which I discounted I-III as being un-OSM because they would protect the data from edits; IV was the idea of being able to better monitor things, which I said was a desire shared by many. My message to anyone contemplating imports into OSM is clear - either accept that others change the data (and you may monitor and you may change back if you have good reason), or don't import. - Today the only way for them is anyway double maintenance: they maintain their internal/primary database, and maybe they care to copy their day-to-day updates manually also to OSM. Or of course just publish their data and let the OSM community do the rest. Is there a way to make maintenance of only their specific data in OSM easy? I think we're having different pictures here. OSM is not a platform which you can use to host your data. The data ceases to become yours once it is in OSM. If that fundamental idea doesn't get across, then further discourse is fruitless. If they are willing to accept the paradigm shift and say: We used to be the keeper of the official list of museums,
Re: [OSM-dev] [Imports] Inserting OSM data
On Sat, Mar 27, 2010 at 2:40 PM, Frederik Ramm frede...@remote.org wrote: Jaak Laineste wrote: Is there a way to make maintenance of only their specific data in OSM easy? I think we're having different pictures here. OSM is not a platform which you can use to host your data. The data ceases to become yours once it is in OSM. If that fundamental idea doesn't get across, then further discourse is fruitless. If they are willing to accept the paradigm shift and say: We used to be the keeper of the official list of museums, but now we see ourselves as a guardian of community museum data in OSM - checking it, refining it, and making sure it doesn't get damaged - then there might be common ground. If only there were some model for these museum enthusiasts to follow. Some model of interested parties curating information for the good of the community. Perhaps refining the information and presenting it in ways that attract the attention of the general public. Or perhaps carving away some layers of the information to show some interesting underlying structure that would not be immediately apparent to a non-expert. Oh! That model could be a museum curator! Except in OpenStreetMap, the curators neither lock the doors nor place the exhibits behind glass boxes and velvet ropes. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
At 2010-03-22 12:02, Nic Roets wrote: ... Unfortunately it has already happened many times. Below is a list of the third party identifiers that I have found. ... hdop sat pdop fix vdop To pick a nit, these are likely not ids, but instead useful info provided via GPS. I recognize: hdop = Horizontal Dilution Of Position vdop = Vertical Dilution Of Position I wish my Garmin 60CSx provided these instead of making me record them manually :( -- Alan Mintz alan_mintz+...@earthlink.net ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
Hi, Jaak Laineste wrote: Just for the future APIs I suggest to give to original importer (or just editor) some kind of priority over the others. Like possibility to lock/protect some parts of the data. It could be done in several levels: 1) my inserted data (tag, node, relation, way - any of them) can be defined as private, nobody else can not even see it. 2) my data is protected - you can see, but not modify 3) my data has group privacy/protection under my control: I can give view/ modify / delete permissions to specific users/groups These three are, in my opinion, not compatible with the spirit of OSM. If you want your own data, store it somewhere else ;-) 4) I can subscribe to notifications for any modifications of my data (this specific piece of it). So I could easily see and revert changes. This is widely demanded, and already partly implemented by XAPI. It is however not something that necessarily has to be a core service - anyone could implement it. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
On Tue, Mar 23, 2010 at 12:50 PM, Frederik Ramm frede...@remote.org wrote: Jaak Laineste wrote: 1) my inserted data (tag, node, relation, way - any of them) can be defined as private, nobody else can not even see it. 2) my data is protected - you can see, but not modify 3) my data has group privacy/protection under my control: I can give view/ modify / delete permissions to specific users/groups These three are, in my opinion, not compatible with the spirit of OSM. If you want your own data, store it somewhere else ;-) Yes, as soon as you license your data under CC-SA you are explicitly allowing people to modify it. I think the whole idea of putting the data in osm is not just for making a pretty picture, but for contributing data for others to build upon. mike ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
I've long considered that it would be good to have a system whereby anyone modifying a specified subset of the database would be presented with a message for them to read before comitting. This would be implemented with a special key that all editing software would be encouraged to support. In this case the message would read please don't delete my IDs It would also be useful in instances of unintentional vandalism, or that case where there was an edit war over some change of road layout between a local that knew the ground and a remote user with out of date yahoo data. Just a thought. JR On 23 March 2010 11:57, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: On Tue, Mar 23, 2010 at 12:50 PM, Frederik Ramm frede...@remote.orgwrote: Jaak Laineste wrote: 1) my inserted data (tag, node, relation, way - any of them) can be defined as private, nobody else can not even see it. 2) my data is protected - you can see, but not modify 3) my data has group privacy/protection under my control: I can give view/ modify / delete permissions to specific users/groups These three are, in my opinion, not compatible with the spirit of OSM. If you want your own data, store it somewhere else ;-) Yes, as soon as you license your data under CC-SA you are explicitly allowing people to modify it. I think the whole idea of putting the data in osm is not just for making a pretty picture, but for contributing data for others to build upon. mike ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
On Mon, Mar 22, 2010 at 2:02 PM, Nic Roets nro...@gmail.com wrote: On Mon, Mar 22, 2010 at 6:18 PM, Tom Hughes t...@compton.nu wrote: On 22/03/10 15:00, Nic Roets wrote: One solution is to add your own tag to the OSM files you generate e.g. smartsoft_id=nnn. And publish the files for review somewhere. Then scan the minutely updates at planet.openstreetmap.de for your id numbers and when they appear you can delete them from your DB. I'm not sure we really want to encourage the use of our database as a storage space for third party identifiers like that. It could get rather out of hand if everybody else wants to attach their own IDs to our data. Hello Tom, Unfortunately it has already happened many times. Below is a list of the third party identifiers that I have found. Instead of banning them outright, I would be happy if we have guidelines that (a) make it easy for us to identify them as third party (b) make sure they don't clash with other imports and (c) make sure they don't take up unnecessary disk space. Almost 100% of the time these are imported to allow for the possibility of future updates to the existing imports. In order to have this possibility, there needs to be a consistent/persistent identifier for a particular thing on the map. Since we don't have support for that in OSM this is the next best thing. If, when the import occurred, we were given an ID by the OSM API that always referenced that object into the future, then the entity performing the import could hang on to the link between the OSM perma-id and whatever the imported element was and perform updates if a new set of data was made available. Of course, this isn't possible right now because some other mapper could come along and split the imported way into several small pieces, delete it all-together, or rename it. Anyway, I'm not advocating for or against anything, simply explaining what I've experienced as an importer and someone helping others import. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
On Mon, Mar 22, 2010 at 7:12 PM, Ian Dees ian.d...@gmail.com wrote:
Almost 100% of the time these are imported to allow for the possibility of
future updates to the existing imports.
Except, as you point out, they can't be used in any way for future
updates since you've got no idea if the reference stays on the
correct object. So I'd rather spend time explaining to people that
external references are a waste of time than encouraging them any
further.
In order to have this possibility,
there needs to be a consistent/persistent identifier for a particular
thing on the map. Since we don't have support for that in OSM this is the
next best thing.
No, it's a distraction from the real problem, which is that merging
datasets is fundamentally hard, and needs to be approached with
sophistication and fuzzy location-based matching (e.g. feature church,
near this point, name something close to all saint(')s). Slapping
external IDs on OSM data and crossing fingers isn't helping.
Anyway, I'm not advocating for or against anything, simply explaining what
I've experienced as an importer and someone helping others import.
Has anyone experienced sync working? I know a lot of people have
imported with external IDs but I suspect that it's not going to work
when they actually come to updating one way or the other.
Cheers,
Andy
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] Inserting OSM data
On 22 March 2010 12:38, Andy Allan gravityst...@gmail.com wrote: On Mon, Mar 22, 2010 at 7:12 PM, Ian Dees ian.d...@gmail.com wrote: Almost 100% of the time these are imported to allow for the possibility of future updates to the existing imports. Except, as you point out, they can't be used in any way for future updates since you've got no idea if the reference stays on the correct object. So I'd rather spend time explaining to people that external references are a waste of time than encouraging them any further. In order to have this possibility, there needs to be a consistent/persistent identifier for a particular thing on the map. Since we don't have support for that in OSM this is the next best thing. Anyway, I'm not advocating for or against anything, simply explaining what I've experienced as an importer and someone helping others import. Has anyone experienced sync working? I know a lot of people have imported with external IDs but I suspect that it's not going to work when they actually come to updating one way or the other. I remember being told it was very important for imports, such as NAPTAN, that can be updated. But I guess that is different, as they are actually the official organiser/referencer of bus stops (and 4th parties might even use that ref). Back to the original poster, Andreas. I am still unclear about some things, are you wanting to bring the data back out of OSM to use in your system/company? If we could maybe look at how we could help you with that then we might be able to understand the best way for you to add to OSM. It would be interesting to know the rough geographic area your data covers. We can then see if the area is near blank or has some dedicated mappers that will be keen to help manually edit your import (e.g. check for imports). Some people here have also requested an extract of the data to look at the quality, or what it contains. Hopefully your not getting overwhelmed by the replies here. Welcome to the OSM community! -- Gregory o...@livingwithdragons.com http://www.livingwithdragons.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
