Re: [Potlatch-dev] P2, snapshot-server, imports, vector layers and more
> From: Andy Allan [mailto:gravityst...@gmail.com] > Subject: Re: [Potlatch-dev] P2, snapshot-server, imports, vector layers > and more > > On 15 November 2012 05:44, Paul Norman wrote: > > > The > > additional attribute prevents the files from working in JOSM > > Oh, really? That sucks. I went for the additional attribute because I > expect XML parsers to ignore it if they didn't understand it. Osmosis, osmconvert and osmfilter all handle it, it appears to be just JOSM that validates it. Of course it'd be pretty easy to strip off the additional attribute, using any of those programs to read the XML then write it out with no changes would likely work. > > but the bigger > > issue is the use of positive IDs. When an object is merged in JOSM it > > carries over its ID if it's a positive ID. This would lead to > > conflicts with existing data in the API. Even if JOSM were patched to > > allow it to work with these files and special-case their positive IDs, > > using positive IDs seems like a recipe for disaster, particularly when > a standardized format exists. > > Well, I wouldn't say that negative ids are standardized, just that p2 > and JOSM use the same convention. I believe Merkaartor uses guids as > placeholder ids, in contrast. It's not the editors, it's the generators. Every converter from shapefiles to .osm that I'm aware of uses negative IDs by default. ogr2osm has hidden options to generate .osm files like osmosis expects (https://github.com/pnorman/ogr2osm/blob/8eb57cfcae743c3b348161f685e483e238288e1d/ogr2osm.py#L125) but shp-to-osm.jar and the old ogr2osm only handle negative IDs afaik. There are also CanVec, cat2osm and numerous other regional efforts that all use negative IDs. They have become the standard interchange format for converted importable data. > Additionally, negative ids for background layers don't seem like the > right solution to me, because that still gives a shared id space. What > happens if you have three background layers, each with a node id="-1"? > Or you create a new way in the main layer, and then pull through a way > with a matching negative id? As far as I can tell JOSM treats negative IDs as meaning these IDs are not assigned by the server so it assigns them to the ID space used for unuploaded objects. You can safely merge multiple files with negative IDs in JOSM. A quick look with show info in JOSM seems to imply that there's a continually decrementing negative ID that persists for as long as JOSM is open, but I haven't read the code involved. > Potlatch approaches this in a "more correct" fashion. Each vector layer > has its own id space. When objects are copied to the main layer their > ids aren't carried over - the main "osm" layer creates corresponding new > entities, copies attributes, and takes care of the placeholder id > assignment. I suspect JOSM will need to handle things in a similar > manner. The "recipe for disaster" of using positive ids in vector layers > is, as I see it, due to JOSM assuming that all the different layers are > in the same id space. As far as I can tell to properly work offline it has to assume positive IDs are in the same ID space or it would be unable to merge two .osm files that are both data from the API. Also, there is no main osm layer in JOSM. It's possible to have multiple files open that are all data from the API. > So does snapshot-server serving negative ids actually solve the problem, > or will it still lead to id space conflicts? It solves the problem. > > Cheers, > Andy > > P.S. I don't think there's actually any osm_id handling code in > snapshot-server anyway, it just uses the same ids as are in the .osm > file in the first place. The problem comes from using osmosis to load the data. Osmosis requires positive IDs, versions and timestamps or the --read-xml task will error. Hence the shoot-self-in-foot options in ogr2osm so I could load it in. I'm not sure how I'm going to be able to load CanVec in - I may have to write some code that turns the negative IDs into positive ones, assigns versions and timestamps and figure out a way to merge it all together. A postgresql query to invert all the IDs might work, but there'd still be the other attributes that aren't expected in an importable file. ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
Re: [Potlatch-dev] P2, snapshot-server, imports, vector layers and more
On 15 November 2012 05:44, Paul Norman wrote: > The > additional attribute prevents the files from working in JOSM Oh, really? That sucks. I went for the additional attribute because I expect XML parsers to ignore it if they didn't understand it. > but the bigger > issue is the use of positive IDs. When an object is merged in JOSM it > carries over its ID if it's a positive ID. This would lead to conflicts with > existing data in the API. Even if JOSM were patched to allow it to work with > these files and special-case their positive IDs, using positive IDs seems > like a recipe for disaster, particularly when a standardized format exists. Well, I wouldn't say that negative ids are standardized, just that p2 and JOSM use the same convention. I believe Merkaartor uses guids as placeholder ids, in contrast. Additionally, negative ids for background layers don't seem like the right solution to me, because that still gives a shared id space. What happens if you have three background layers, each with a node id="-1"? Or you create a new way in the main layer, and then pull through a way with a matching negative id? Potlatch approaches this in a "more correct" fashion. Each vector layer has its own id space. When objects are copied to the main layer their ids aren't carried over - the main "osm" layer creates corresponding new entities, copies attributes, and takes care of the placeholder id assignment. I suspect JOSM will need to handle things in a similar manner. The "recipe for disaster" of using positive ids in vector layers is, as I see it, due to JOSM assuming that all the different layers are in the same id space. So does snapshot-server serving negative ids actually solve the problem, or will it still lead to id space conflicts? Cheers, Andy P.S. I don't think there's actually any osm_id handling code in snapshot-server anyway, it just uses the same ids as are in the .osm file in the first place. ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
Re: [Potlatch-dev] P2, snapshot-server, imports, vector layers and more
On 15 November 2012 08:37, Richard Fairhurst wrote: > For 1; we could continue with vectors.xml, and also provide a way to provide > a URL in the embedding page/query string (much as you can with background > imagery). Recently I've been looking at the tilejson spec, which the Mapbox guys developed for map layers. It's a way of describing what tiles are available, the extent of the map, the attribution and so on in a json document. https://github.com/mapbox/tilejson-spec http://c.tiles.mapbox.com/v3/examples.map-4l7djmvo.json I think we could make imagery.xml more lightweight if we encourage sources (e.g. our ooc maps) to provide a tilejson document - then our imagery list can be just a list of URLs to the tilejson documents. It will also then simplify the experience for a user adding their own imagery, since all the extents, zoom levels etc are figured out automatically. I'm therefore also wondering if we can come up with a similar json doc for vector background layers, that applications (p2/josm) can use to figure out all the information they need. At the moment snapshot-server displays the xml for you to copy and paste into vectors.xml, but Paul has highlighted the need for users to add their own snapshot layers to whichever instance of p2 they are running. Pasting a json-endpoint URL is perhaps much simpler than building a dialog to paste in xml snippets. Thoughts? > (It might be possible to do what JOSM does by putting the sources on a wiki > page, rather than relying on a commit to the P2 repository, but I'm always a > bit wary of people adding Google imagery etc. without us noticing. Any > thoughts?) Eugh, no. Let's stick to keeping things specified in the repository - with github's web-based file editing, it's as easy to contribute to as a wiki, but with enforced review during the pull requests. > For 2; we could add a new menu like the 'Background' one showing data layers > in the current area. I'd like to see something like this. Cheers, Andy ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
Re: [Potlatch-dev] P2, snapshot-server, imports, vector layers and more
On 15/11/2012 09:42, Paul Norman wrote: Deploying the P2 instance is what I'll have problems with - are there any guides on how to deploy P2 with a custom vectors.xml (or imagery.xml) with minimal pain? http://wiki.openstreetmap.org/wiki/Potlatch_2/Deploying_Potlatch_2 http://wiki.openstreetmap.org/wiki/Potlatch_2_merging_tool but, that said, the random.dev.osm.org is (I think) no longer current so you're probably better off pulling down the resources directory from github, and potlatch2.swf from OSM. That would be good, but is one URL enough information or do you also need the crossdomain.xml file location? I suppose you could take multiple fields. Flash Player automatically looks at the root level of the domain for crossdomain.xml - it's not something that P2 specifies. Another issue that impacts on this is the lack of support for polygon bounds. There is a huge overlap between the Canada and lower-48 states US bboxes. I'm not against adding polygon bounds support, but would need someone to point me to some pseudocode for the implementation (geometry always makes my head hurt), and a nice test case. cheers Richard ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
Re: [Potlatch-dev] P2, snapshot-server, imports, vector layers and more
> From: Richard Fairhurst [mailto:rich...@systemed.net] > Subject: Re: [Potlatch-dev] P2, snapshot-server, imports, vector layers > > > [...] > > I must admit I have not verified if this would require changes to P2. > > It's possible that P2 would work fine if snapshot-server returned > negative IDs. > > I've not tested it but I'm 99% sure it would. Happy to test if you set > up an instance somewhere which serves negative ids! I believe I could serve the data from apache easily enough to test. It'd return the entire dataset as a response to a map? Query but for testing with a small dataset that shouldn't be an issue. Deploying the P2 instance is what I'll have problems with - are there any guides on how to deploy P2 with a custom vectors.xml (or imagery.xml) with minimal pain? > > 2. Data layer exposure. To add a snapshot-server data layer to P2 you > > have to deploy your own P2 instance. This severely limits exposure. > > What makes it worse is that if I offer a data layer and deploy P2 and > > someone else does the same, you can't use both at the same time. > > > > A way to add a snapshot-server layer by pointing it at a configuration > > file (e.g. a json file) would be very useful. > > > > Better yet would also be a way to reveal data layers like imagery > > layers are, based on bbox (or better yet, by polygon). > > Ok. You can already specify bboxes in vectors.xml in the same way that > you do in imagery.xml. So I guess the problems to address are: > > 1. Make it easier for 'data providers' to make users aware of data > layers 2. Provide easy UI for users to select data layers > > For 1; we could continue with vectors.xml, and also provide a way to > provide a URL in the embedding page/query string (much as you can with > background imagery). That would be good, but is one URL enough information or do you also need the crossdomain.xml file location? I suppose you could take multiple fields. > (It might be possible to do what JOSM does by putting the sources on a > wiki page, rather than relying on a commit to the P2 repository, but I'm > always a bit wary of people adding Google imagery etc. without us > noticing. Any thoughts?) The JOSM wiki page is moderately painful to edit. I'm not convinced it's a good way. It would be nice if there was one imagery source list XML format shared between JOSM and P2. > For 2; we could add a new menu like the 'Background' one showing data > layers in the current area. I think long-term this is a good idea, but would require more data layers to be available. Are there currently any running besides cyclestreets? Another issue that impacts on this is the lack of support for polygon bounds. There is a huge overlap between the Canada and lower-48 states US bboxes. ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
Re: [Potlatch-dev] P2, snapshot-server, imports, vector layers and more
Paul Norman wrote: Background: P2 has excellent support for data layers but their full power is not exposed to the user by default. I am contemplating using snapshot-server with some extremely large datasets to make the data available to people using P2 and JOSM as vector layers. Great stuff. [...] I must admit I have not verified if this would require changes to P2. It's possible that P2 would work fine if snapshot-server returned negative IDs. I've not tested it but I'm 99% sure it would. Happy to test if you set up an instance somewhere which serves negative ids! 2. Data layer exposure. To add a snapshot-server data layer to P2 you have to deploy your own P2 instance. This severely limits exposure. What makes it worse is that if I offer a data layer and deploy P2 and someone else does the same, you can't use both at the same time. A way to add a snapshot-server layer by pointing it at a configuration file (e.g. a json file) would be very useful. Better yet would also be a way to reveal data layers like imagery layers are, based on bbox (or better yet, by polygon). Ok. You can already specify bboxes in vectors.xml in the same way that you do in imagery.xml. So I guess the problems to address are: 1. Make it easier for 'data providers' to make users aware of data layers 2. Provide easy UI for users to select data layers For 1; we could continue with vectors.xml, and also provide a way to provide a URL in the embedding page/query string (much as you can with background imagery). (It might be possible to do what JOSM does by putting the sources on a wiki page, rather than relying on a commit to the P2 repository, but I'm always a bit wary of people adding Google imagery etc. without us noticing. Any thoughts?) For 2; we could add a new menu like the 'Background' one showing data layers in the current area. cheers Richard ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
