Re: [OSM-dev] Chopped of ways. New flag for OSM XML?
On Thu, Aug 14, 2008 at 03:44:57PM +1000, Brett Henderson wrote: Karl Newman wrote: Even still, it's a valid concern and there are other operations (i.e., cutting into tiles or tag transforms) that can manipulate the data into a form that probably shouldn't be uploaded. Karl Yep, agree. If it wasn't the default behaviour it would just be far less likely to occur :-) There was discussion a while ago about creating some form of super bounds element at the top of osm files to indicate which system the data is compatible with (eg. production api versus a dev environment). Perhaps something like that would be more appropriate than resorting to tags on every modified entity? You could get extra funky and generate a unique id for each database upon installation that allows tools to check for compatibility (subversion does this I believe) ... perhaps a simple user entered db name would be more than sufficient. But people *want* to use the data they get from some kind of extraction process to base decisions upon. And they should be able to. Not all the data is invalidated by an extraction, only some of it. It makes sense to mark those parts invalid that are, so that an automated process can decide what it uses and what not. Jochen -- Jochen Topf [EMAIL PROTECTED] http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Chopped of ways. New flag for OSM XML?
Jochen Topf wrote: But people *want* to use the data they get from some kind of extraction process to base decisions upon. And they should be able to. Not all the data is invalidated by an extraction, only some of it. It makes sense to mark those parts invalid that are, so that an automated process can decide what it uses and what not. Jochen Sure. If I stop mangling ways then I can't think of a particular case where this would be useful for osmosis processing ... but that's not to say one doesn't exist. I'll keep quiet and let those more impacted by the topic resume normal discussions ;-) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Chopped of ways. New flag for OSM XML?
On Thu, Aug 14, 2008 at 12:47 AM, Brett Henderson [EMAIL PROTECTED] wrote: Jochen Topf wrote: But people *want* to use the data they get from some kind of extraction process to base decisions upon. And they should be able to. Not all the data is invalidated by an extraction, only some of it. It makes sense to mark those parts invalid that are, so that an automated process can decide what it uses and what not. Jochen Sure. If I stop mangling ways then I can't think of a particular case where this would be useful for osmosis processing ... but that's not to say one doesn't exist. I'll keep quiet and let those more impacted by the topic resume normal discussions ;-) How about Dave Stubb's new tag transform plugin? And the other example I mentioned, tile cutting, where ways are intentionally mangled to end exactly on the tile border? Karl ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Chopped of ways. New flag for OSM XML?
On Fri, Aug 15, 2008 at 12:28:14AM +1000, Brett Henderson wrote: Karl Newman wrote: On Thu, Aug 14, 2008 at 12:47 AM, Brett Henderson [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Jochen Topf wrote: But people *want* to use the data they get from some kind of extraction process to base decisions upon. And they should be able to. Not all the data is invalidated by an extraction, only some of it. It makes sense to mark those parts invalid that are, so that an automated process can decide what it uses and what not. Jochen Sure. If I stop mangling ways then I can't think of a particular case where this would be useful for osmosis processing ... but that's not to say one doesn't exist. I'll keep quiet and let those more impacted by the topic resume normal discussions ;-) How about Dave Stubb's new tag transform plugin? And the other example I mentioned, tile cutting, where ways are intentionally mangled to end exactly on the tile border? I didn't explain myself very well. In the case of these transform tasks you could simply mark the whole file as unsuitable for upload, I couldn't see a case where you'd need to know which entities have been mangled and which haven't. But again, perhaps I just haven't considered a case where it might be useful. I think its better to mark each individual object as complete or incomplete. Most of the file is ok, so if you mark the whole file, people will look at it and say: That looks ok to me, I'll just go ahead. If you mark individual objects its clearer where exactly the problem is. That beeing said, an *additional* marker at the file level saying: Some things in this file might be incomplete, might be good. If you only have the marker at the file level, the file can't be cleaned up, because you don't know which data to trust and which not to trust. If you have markers at all incomplete objects a file is trivial to clean. Jochen -- Jochen Topf [EMAIL PROTECTED] http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Chopped of ways. New flag for OSM XML?
Jochen Topf wrote: Hi! When creating excerpts of OSM data with Osmosis, Osmosis will (depending on flags) chop of ways, i.e. the output will contain ways with only some of the nodes in the way. But looking at the XML there is no way to tell that this has happened. The way still has the same ID and timestamp. There should be a way of recognizing theses chopped of ways, say by adding an incomplete=true attribute or something. This question came up on the German mailing list after somebody had used the excerpt as basis for some automated fixing of data. He (IMHO correctly) checked the Osmosis excerpt with current data from the API by comparing the timestamps which were the same. So he assumed nothing had changed and he could safely edit the data. But because of the excerpt process the data he based his decision on, was not correct. This is part of a larger problem. Say you create an extract with only some of the tags or a relation with only some members. Jochen I made a mistake making way truncation the default behaviour. I intend to fix this but feel free to raise a defect in TRAC. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Chopped of ways. New flag for OSM XML?
On Wed, Aug 13, 2008 at 8:34 PM, Brett Henderson [EMAIL PROTECTED] wrote: Jochen Topf wrote: Hi! When creating excerpts of OSM data with Osmosis, Osmosis will (depending on flags) chop of ways, i.e. the output will contain ways with only some of the nodes in the way. But looking at the XML there is no way to tell that this has happened. The way still has the same ID and timestamp. There should be a way of recognizing theses chopped of ways, say by adding an incomplete=true attribute or something. This question came up on the German mailing list after somebody had used the excerpt as basis for some automated fixing of data. He (IMHO correctly) checked the Osmosis excerpt with current data from the API by comparing the timestamps which were the same. So he assumed nothing had changed and he could safely edit the data. But because of the excerpt process the data he based his decision on, was not correct. This is part of a larger problem. Say you create an extract with only some of the tags or a relation with only some members. Jochen I made a mistake making way truncation the default behaviour. I intend to fix this but feel free to raise a defect in TRAC. Even still, it's a valid concern and there are other operations (i.e., cutting into tiles or tag transforms) that can manipulate the data into a form that probably shouldn't be uploaded. Karl ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Chopped of ways. New flag for OSM XML?
Karl Newman wrote: Even still, it's a valid concern and there are other operations (i.e., cutting into tiles or tag transforms) that can manipulate the data into a form that probably shouldn't be uploaded. Karl Yep, agree. If it wasn't the default behaviour it would just be far less likely to occur :-) There was discussion a while ago about creating some form of super bounds element at the top of osm files to indicate which system the data is compatible with (eg. production api versus a dev environment). Perhaps something like that would be more appropriate than resorting to tags on every modified entity? You could get extra funky and generate a unique id for each database upon installation that allows tools to check for compatibility (subversion does this I believe) ... perhaps a simple user entered db name would be more than sufficient. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Chopped of ways. New flag for OSM XML?
On Tue, Aug 12, 2008 at 6:50 AM, Jochen Topf [EMAIL PROTECTED] wrote: Hi! When creating excerpts of OSM data with Osmosis, Osmosis will (depending on flags) chop of ways, i.e. the output will contain ways with only some of the nodes in the way. But looking at the XML there is no way to tell that this has happened. The way still has the same ID and timestamp. There should be a way of recognizing theses chopped of ways, say by adding an incomplete=true attribute or something. This question came up on the German mailing list after somebody had used the excerpt as basis for some automated fixing of data. He (IMHO correctly) checked the Osmosis excerpt with current data from the API by comparing the timestamps which were the same. So he assumed nothing had changed and he could safely edit the data. But because of the excerpt process the data he based his decision on, was not correct. This is part of a larger problem. Say you create an extract with only some of the tags or a relation with only some members. Jochen How about osmosis:mangled=yes? Or what about touching the timestamp of the entity? Karl ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Chopped of ways. New flag for OSM XML?
On Tue, Aug 12, 2008 at 4:41 PM, Karl Newman [EMAIL PROTECTED] wrote: On Tue, Aug 12, 2008 at 6:50 AM, Jochen Topf [EMAIL PROTECTED] wrote: Hi! When creating excerpts of OSM data with Osmosis, Osmosis will (depending on flags) chop of ways, i.e. the output will contain ways with only some of the nodes in the way. But looking at the XML there is no way to tell that this has happened. The way still has the same ID and timestamp. There should be a way of recognizing theses chopped of ways, say by adding an incomplete=true attribute or something. This question came up on the German mailing list after somebody had used the excerpt as basis for some automated fixing of data. He (IMHO correctly) checked the Osmosis excerpt with current data from the API by comparing the timestamps which were the same. So he assumed nothing had changed and he could safely edit the data. But because of the excerpt process the data he based his decision on, was not correct. This is part of a larger problem. Say you create an extract with only some of the tags or a relation with only some members. Jochen How about osmosis:mangled=yes? Or what about touching the timestamp of the entity? I agree that some kind of attribute would be very useful to indicate that the data is not complete in this case. However, I don't think it should be an osmosis specific one. So mangled='yes' rather than osmosis:mangled='yes'. There could well be other data sources that mangle data, and having to check all the different possible namespaces would be infeasible: osmosis:mangled='yes' osmxapi:truncated='1' someOtherSource:complete='no' etc The core osm xml schema needs a documented attribute that can be used to indicate that the data is incomplete. This can be done immediately as there is no need for any changes to the API, by definition it currently always returns complete data. However it would be nice if some future version of the API were to block any uploads that had a complete='no' attribute on any element. (For the sake of clarity, I'm talking about an XML attribute here, not some additional tag entry like tag k='osmosis:mangled' v='yes'/ which wouldn't help as it would just end up in someone's database :S ). 80n Karl ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
