Re: [OSM-talk] Multiple Layers for OSM
On Tue, Sep 25, 2012 at 12:28:56PM -0700, Dave Sutter wrote: It is harder to build because there is added functionality. You have to replace auto ID generation, which isn't too complicated. I'm not sure what you mean about robustness. robustness is about the behaviour of the system in case of failures. In this case all servers (which might be a lot in the future) depend on a single server to give out IDs. If that one doesn't work (or the network to it), the whole system fails. Of if a random server requests huge amounts of ID blocks it could bring down the ID server or we could run out of IDs. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
This is nice, in my manifesto I even have an item on supporting research in to some kind of layer functionality, and now the discussion has started on its own :-). My thoughts - GIS-like layers are a no no, they increase dependencies in the data and we want to achieve the opposite. - any implementation will have to conserve the most scarce resource we have: developer time. This applies equally to core infrastructure as to the myriad of tools and editors. - at least for any layers/databases operated by the OSMF we should not depart from the any user can edit anything principle. - any implementation should a allow conflict free merging of data and synchronizing of data from multiple sources IMHO the easiest, but very hackish, way to achieve this would be to split the (64bit) object ID space up in to sub-spaces (a la CIDR). All tools will have to support 64bit IDs rsn and we are currently throwing away half of the ID space for no real good reason except convenience. The main concern is that 64bits may not be enough ID space long term: for example 63 bits for OSM proper would allow ~18'000 nodes per square meter of the earth's surface (~62'000 without oceans), which may not be enough for detailed multi-storey indoor mapping :-). Such a scheme would not preclude switching to larger IDs down the road (the ID size is essentially just a practical limitation of the databases and tools we are currently using) the individual layers would simply get non-continuous ID spaces allocated if necessary . Simon PS: all calculations errors etc are mine :-) ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
Matt Amos wrote: On Mon, 2012-09-24 at 09:36 +0200, Jochen Topf wrote: It turns out there are many other interesting uses of multiple layers but also many technical and social questions around them. I have written down my thoughts on this subject in a (rather lengthy) blog post: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html what do you think of the Potlatch 2 vector backgrounds [1] and snapshot server [2] as steps in the direction of fixing this? [1]http://wiki.openstreetmap.org/wiki/Potlatch_2_merging_tool [2]http://wiki.openstreetmap.org/wiki/Snapshot_Server Certainly the 'problem' can be broken down into several elements. The editing tools certainly have most of what is needed to display multiple layers, and merge data between layers. JOSM is obviously geared to handling multiple layers and it looks like potlatch2 is heading down the same path? But what is really needed is a viewer that can combine layers from different sources in the same way? Problem here is I think that it does need to be able to handle vector as well as raster layers? But isn't that what http://layers.openstreetmap.fr/ does anyway? So IS there any development work needed here? Other than making the base layers and overlays configurable? I'd like to set up my own 'server' although 'rails' puts me off as it's something else I'd have to learn. Being PHP based, having to cope with java and python simply to repair the tools I'm using is bad enough! I've been running mapserver for years so perhaps I need to look at that, or is there an alternatively? OK LAYERS http://wiki.openstreetmap.org/wiki/OSM_Layers lists 'pluses and minuses' but I'm not sure I agree with many of the points on that. http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html seems more down the right line. Lets split the problem into three. Private secondary layers simply need to be geo-referenced and perhaps provide a 'limit' to the pan function? This is just a rendering problem in the viewer. An example of this would be 'bird-migrations' overlay which would not use ways and nodes from the base layers. Public secondary layers would be such things as 'low water/highwater/marine' or 'contour' which are essentially ways that do not 'naturally' need to be constructed using the SAME way segments that construct the main map data. Actually I don't see the advantage of splitting up even border ways into hundreds of small segments simply because a border segment may consist of numerous small bits of existing ways? A boundary like 'The cotswolds' is to my mind a complete nightmare to edit and consists of hundreds of 'little fixes' to join up elements that don't naturally join themselves. So I think that boundaries fit more naturally at this level with their own ways! The base map is essentially a single layer, and some of the 'Why have layers' points simply require improved filtering of the data. Perhaps only downloading a subset of the raw data. Much as the renderers only display selective subsets. Import datasets are essentially stand alone base map layers from which data is either extracted and merged directly, or just used as a tracing source. Having said that boundaries should have their own ways, I don't see any problem with the likes of JOSM importing from several layers and managing those layers 'in parallel' rather than flattening to a single layer. One can switch layers on and off easily, and perhaps tag between layers where track details should be locked between 'layers' so that editing can handle changes that need to affect all layers using the SECTIONS of the same way. The important thing to bear in mind here is Do you actually ALLOW editing of the details of an import? While we may be able to provide updates in advance of that appearing in a later import, retaining the historic data may be better managed by tagging the relevant elements of an update as 'end_date=xxx' and adding new segments that provide the correction. THIS also fits in better with also maintaining the fine detail underlying a change ... but begs the question 'is this a fix to faulty data' or @an historic change to the object' While I accept that the change log does maintain this data, accessing it can be problematic, while a major API change may be that 'delete' simply mirrors the details to the 'historic' database with the correct end_date flag, and changes to details result in a copy of the original being archived. Having just written that, I am thinking that we need two mechanisms for this. One takes everything flagged with a end_date and moves it to a linked historic database, while the other simply adds the correct end_date flag. SO for border layers you have the option to select a date - past or future - for the rendering we want? -- Lester Caine - G8HFL - Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk
Re: [OSM-talk] Multiple Layers for OSM
On Tue, Sep 25, 2012 at 12:38:20PM +0200, Simon Poole wrote: IMHO the easiest, but very hackish, way to achieve this would be to split the (64bit) object ID space up in to sub-spaces (a la CIDR). All tools will have to support 64bit IDs rsn and we are currently throwing away half of the ID space for no real good reason except convenience. The main concern is that 64bits may not be enough ID space long term: for example 63 bits for OSM proper would allow ~18'000 nodes per square meter of the earth's surface (~62'000 without oceans), which may not be enough for detailed multi-storey indoor mapping :-). Such a scheme would not preclude switching to larger IDs down the road (the ID size is essentially just a practical limitation of the databases and tools we are currently using) the individual layers would simply get non-continuous ID spaces allocated if necessary . Currently the IDs are rather dense in the ID space. (There are holes from deleted objects, but not that many.) This is a very good thing, because it allows efficient implementations, for instance all node locations can be put into an array which needs only 8 bytes * max_node_id. If you give out part of the ID space to different databases this approach does not work any more. Thats not the end of the world, but we have to really start looking into the consequences of these kinds of decisions because we are not a small database any more... Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
Hi! Using GUIDs is problematic because they are so large (compared to a simple ID) and the GUIDs are not dense in the GUID space. This would probably mean another redirection when finding data inside an application. Jochen On Mon, Sep 24, 2012 at 11:31:55PM +0700, Jais Pedersen wrote: Date: Mon, 24 Sep 2012 23:31:55 +0700 From: Jais Pedersen j...@pedersens.net To: talk@openstreetmap.org Subject: Re: [OSM-talk] Multiple Layers for OSM Or we could use GUIDs, but if we use the linking approach, then each layer could have its own IDs and if the links are implemented as tags would it be possible to do it without changes to the API? /Jais On Mon, Sep 24, 2012 at 11:11 PM, Dave Sutter sut...@intransix.com wrote: This is a very interesting thread and I need a little more time to understand what is in it so far. I do want to make one technical comment. It is possible to to have multiple databases and still have unique IDs across them. An ID server can be set up to create the IDs. When one database creates a new object it requests an ID from the ID server rather than creating an ID locally. (And of course, if you are creating a lot of IDs, you would make one request to the ID server in bulk to get all the IDs needed.) Dave On Mon, Sep 24, 2012 at 12:36 AM, Jochen Topf joc...@remote.org wrote: In the recent discussion about the the imports in France and DWG governance the issue of multiple layers in OSM came up again. If we had some kind of layer system we could stage imports through them instead of adding all data to the OSM database directly and this would help finding problems etc. It turns out there are many other interesting uses of multiple layers but also many technical and social questions around them. I have written down my thoughts on this subject in a (rather lengthy) blog post: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html If anybody wants to comment, I think this mailing list is the right place. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
On Mon, Sep 24, 2012 at 09:11:34AM -0700, Dave Sutter wrote: This is a very interesting thread and I need a little more time to understand what is in it so far. I do want to make one technical comment. It is possible to to have multiple databases and still have unique IDs across them. An ID server can be set up to create the IDs. When one database creates a new object it requests an ID from the ID server rather than creating an ID locally. (And of course, if you are creating a lot of IDs, you would make one request to the ID server in bulk to get all the IDs needed.) Yes, but it makes the whole system harder to build and less robust. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
On Mon, Sep 24, 2012 at 03:28:50PM +0100, Matt Amos wrote: On Mon, 2012-09-24 at 09:36 +0200, Jochen Topf wrote: It turns out there are many other interesting uses of multiple layers but also many technical and social questions around them. I have written down my thoughts on this subject in a (rather lengthy) blog post: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html what do you think of the Potlatch 2 vector backgrounds [1] and snapshot server [2] as steps in the direction of fixing this? [...] [1] http://wiki.openstreetmap.org/wiki/Potlatch_2_merging_tool [2] http://wiki.openstreetmap.org/wiki/Snapshot_Server Those are good pieces of the puzzle. I think we need to bring those and other different pieces together into something conceptually hole. 1. First we need a way for everybody to host their own data layers. The Snapshot server is one option, the RailsPort another. Ideally I would like to see one server software that everybody can install and configure according to their needs. 2. Then we need a way of adressing those layers globally. I proposed using URLs with meta data behind them and some registration service in my blog post. 3. Then we need tools to bring those layers together in different ways. All major editors already support some kind of layering. They need to support this adressing. And we need to support extract, copy, merging, etc. of the data. What I am thinking about here is really not that big of a step. It is just a more integrated way of thinking about the pieces and formalizing some interfaces so that the pieces fit well together. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
On Mon, Sep 24, 2012 at 05:04:29PM -0700, Dave Sutter wrote: I liked the Tag Central presentation. I have searched for more information but I can't find much. Has there been any more development on the Tag Central idea? Thats a bit offtopic in this thread, but... Although Tag Central sounds good at first, in my opinion it is the wrong approach. Centralizing control in one point is counter to the way OSM works. Taginfo takes the opposite approach and, unlike Tag Central, it does exist and it is used every day by many mappers. :-) http://wiki.openstreetmap.org/wiki/Taginfo Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
On Mon, Sep 24, 2012 at 11:35:45AM +0200, Martin Koppenhoefer wrote: Another similar concern I have with layers is that of fragmentation of the data which currently is all in the one main layer. In the past there were some people asking for separate thematic layers like landuse (e.g. in order to not show them in their editor), and introducing a layer-system might likely lead to fullfilling this desire. I see this as a problem because landuse is strongly tied to other objects like streets, building lots, and other polygons (e.g. amenity, leisure, place-polygons) and moving or editing only part of this data will also lead to out-of-sync-geometry between layers (won't fit one over the other). To avoid this people would have to look at all layers, which in the end eliminates the benefits of separate layers. Yes, this is the biggest concern I have with this approach, too. We need to develop an idea what data should be in different layers and what data shouldn't. And this has to be something the whole project has to agree on. We don't want a situation where we have to known that to get landcover in France you'll have to add the corinne layer, but in Germany, the landcover is inside the main layer! Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
On Mon, Sep 24, 2012 at 08:54:08PM +0700, Jais Pedersen wrote: The problem with lengthy blog posts is that they result in lengthy replies and probably a long thread :-) Thats not a problem. Thats good! :-) I think most users never look at the edit history and most of us that occasionally do look at it, do it mainly to get some idea about the validity of the data. To use our data to make historical maps, we have to finish making the current snapshot of the world first, then it might be possible to start looking at a way of making different more or less linked snapshots in time. I agree it might not be easy, but I think that just because it is difficult to build a space shuttle, that should not stop us from trying to build the car that most of us actually need first. The problem is that there are people out there that want to make historical maps today. You can't tell them to go away and come back in ten years when we come around to help them. What I want to give them is some minimal tool so that they *can* work on their stuff without disturbing the OSM mainstream. The rendering performance problem for lower zoom levels, is probably better handled during import/update. If apmon keeps improving the performance of osm2pgsql, then it might be realistic to have a normal high zoom database and a separate low zoom database where only the tags need for low zoom levels are imported and the geometries are simplified (it might be as simple as using ST_simplify, but I have no experience with that and currently no access to capable hardware for testing). From Frederics SOTM slides it looks like the sweet spot is around zoom level 8. This is much more difficult than simplifying some geometries. We already have some low zoom data in an extra table in osm2pgsql und many people have already done line simplification in their data. I wrote software myself that can extract simplified motorways etc. from OSM data for use in small scale maps. But that is really hard to do properly and it doesn't always work. Having humans in the loop makes nicer maps more feasible. All I am talking about an option here. If you don't want to use this for your map, thats fine. But I certainly would have needed that option many times already. I do share Martins concerns about how to handle updates to linked data, but maybe that can be solved by having both hard and soft links, so the user that creates the link makes the decision. That will also force you to think about if these two areas are actually linked or did you just reuse the nodes that were conveniently already there? If you have both layers open, you could have a also update soft links mode for those that know what they are doing and in the historic layer we could keep the soft links in an un-linked, but not yet completely destroyed state, where somebody can manually check if the link should be restored or removed. That will not completely solve the problem with historic maps, but if that is the only issue, I don't think that should stop us. Frankly I don't think we can ever solve the linked data problem. Linking structured data to other structured data in a meaningful way is hard to do. Of course we can have some kind of link between the data and I think we should. But the link will only link between generic objects and that doesn't really help us much. Most of the semantic information in OSM is inside the tags and they are not really accessible for linking. What I mean is this: Say you have a node today with an address tag. You then create a building outline and move the address tag to that building. Conventional wisdom is to at least re-use the node inside your new building outline way, to create some kind of connection. But that might not be possible in all cases and, anyway, not everybody does it that way. So there is some connection lost here. But this is a very specialized case anway. How would we design generic tools that allow us to find corresponding data in the same database over time or in different databases? I think a possible approach in this case is similar to what we have been doing for a long time with our semi-automated quality management tools: We use the data itself and especially redundancies in our data to find corresponding data. We tell the mapper about those cases and let them work out the details and fix the data. So if we have several databases and extra tables that link objects ids from one database to the other we can write tools that tell us: Object x changed in this database in some possibly important way but object y in the other database did not change in a correspoding way. Please look into that. The important thing is that those tools can be implemented outside the databases themselves, which makes development much easier. And they can be implemented specially for many different use cases by the interested parties themselves. This approach is by no means perfect, but I fear thats all we can get.
Re: [OSM-talk] Multiple Layers for OSM
It is harder to build because there is added functionality. You have to replace auto ID generation, which isn't too complicated. I'm not sure what you mean about robustness. Another issue this would cause is a bigger time delay on the server. I don't think it would be a good idea unless we really need common ids across servers. If so this has the benefit that no code changes except in the assignment of ids on the server. Sent from my iPhone On Sep 25, 2012, at 11:36 AM, Jochen Topf joc...@remote.org wrote: On Mon, Sep 24, 2012 at 09:11:34AM -0700, Dave Sutter wrote: This is a very interesting thread and I need a little more time to understand what is in it so far. I do want to make one technical comment. It is possible to to have multiple databases and still have unique IDs across them. An ID server can be set up to create the IDs. When one database creates a new object it requests an ID from the ID server rather than creating an ID locally. (And of course, if you are creating a lot of IDs, you would make one request to the ID server in bulk to get all the IDs needed.) Yes, but it makes the whole system harder to build and less robust. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
Jochen Topf wrote: Another similar concern I have with layers is that of fragmentation of the data which currently is all in the one main layer. In the past there were some people asking for separate thematic layers like landuse (e.g. in order to not show them in their editor), and introducing a layer-system might likely lead to fullfilling this desire. I see this as a problem because landuse is strongly tied to other objects like streets, building lots, and other polygons (e.g. amenity, leisure, place-polygons) and moving or editing only part of this data will also lead to out-of-sync-geometry between layers (won't fit one over the other). To avoid this people would have to look at all layers, which in the end eliminates the benefits of separate layers. Yes, this is the biggest concern I have with this approach, too. We need to develop an idea what data should be in different layers and what data shouldn't. And this has to be something the whole project has to agree on. We don't want a situation where we have to known that to get landcover in France you'll have to add the corinne layer, but in Germany, the landcover is inside the main layer! I'm seeing an interface to the raw data that automatically filters which objects are passed to the outgoing data stream. Theoretically there is no reason that the data FEEDING the filter could not be in separate databases. But alternatively separate tables within the one database may allow the faster access to 'layers' of data? Personally I don't see this as a particular reason to be 'layering' data. I've commented elsewhere on the 'out-of-sync' geometry problem between layers, and some potential layers such as contours do lend themselves to being isolated from other data? I'm not too worried about differences as I'm sure it can be managed if essential, but the 'difference' may also be legitimate and pulling that apart at the moment is an even more interesting problem. -- Lester Caine - G8HFL - Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Multiple Layers for OSM
In the recent discussion about the the imports in France and DWG governance the issue of multiple layers in OSM came up again. If we had some kind of layer system we could stage imports through them instead of adding all data to the OSM database directly and this would help finding problems etc. It turns out there are many other interesting uses of multiple layers but also many technical and social questions around them. I have written down my thoughts on this subject in a (rather lengthy) blog post: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html If anybody wants to comment, I think this mailing list is the right place. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
Everyone (including me) has this idea every few years, especially your second idea. Then they start coding and the problems begin to really come to light, especially when it comes to reconsiling editing issues. I think this is one of those it doesn't help to talk about it, problems. Code, find the problems, then take a stab at fixing them, then discuss better ways. - Serge ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
On Mon, Sep 24, 2012 at 04:17:44AM -0400, Serge Wroclawski wrote: Everyone (including me) has this idea every few years, especially your second idea. Then they start coding and the problems begin to really come to light, especially when it comes to reconsiling editing issues. I think this is one of those it doesn't help to talk about it, problems. Code, find the problems, then take a stab at fixing them, then discuss better ways. Yes of course many of those ideas I wrote down are not new. But I disagree with your conclusions. If people already thought about it and started coding and couldn't solve the problems, doesn't that suggest that this is *not* one of those problems where it doesn't help to talk about it? To me this suggests we need more talking, not less. :-) But I think you are partly right. I believe we can't think the whole problem through to the end. Its much too complex for that. But if we can agree on some framework, a rough design, then the details will come. And I think the social issues are more important than the technical issues here. First we have to decide whether we want to go down this road at all. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
The layer rendering thing in first position has nothing to do with what I am suggesting. That can already be done with current technology. But the second one has (People want to host their own layers). :-) Jochen On Mon, Sep 24, 2012 at 10:20:29AM +0200, Mike Dupont wrote: Date: Mon, 24 Sep 2012 10:20:29 +0200 From: Mike Dupont jamesmikedup...@googlemail.com To: Jochen Topf joc...@remote.org Cc: talk@openstreetmap.org Subject: Re: [OSM-talk] Multiple Layers for OSM Well I support this idea and see it as long needed, it is also the top voted item on the ideascale, http://fosm.ideascale.com/ to be able to render as layers. The layers could be used also to split up different licensed data as well. On Mon, Sep 24, 2012 at 9:36 AM, Jochen Topf joc...@remote.org wrote: In the recent discussion about the the imports in France and DWG governance the issue of multiple layers in OSM came up again. If we had some kind of layer system we could stage imports through them instead of adding all data to the OSM database directly and this would help finding problems etc. It turns out there are many other interesting uses of multiple layers but also many technical and social questions around them. I have written down my thoughts on this subject in a (rather lengthy) blog post: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html If anybody wants to comment, I think this mailing list is the right place. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk -- James Michael DuPont Member of Free Libre Open Source Software Kosova http://flossk.org Saving wikipedia(tm) articles from deletion http://SpeedyDeletion.wikia.com Contributor FOSM, the CC-BY-SA map of the world http://fosm.org Mozilla Rep https://reps.mozilla.org/u/h4ck3rm1k3 Free Software Foundation Europe Fellow http://fsfe.org/support/?h4ck3rm1k3 -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
Jochen Topf wrote: In the recent discussion about the the imports in France and DWG governance the issue of multiple layers in OSM came up again. If we had some kind of layer system we could stage imports through them instead of adding all data to the OSM database directly and this would help finding problems etc. It turns out there are many other interesting uses of multiple layers but also many technical and social questions around them. I have written down my thoughts on this subject in a (rather lengthy) blog post: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html If anybody wants to comment, I think this mailing list is the right place. Seems to cover everything although there is only one bit I'd 'dispute' ... latter. Personally I've been happy with potlatch for the sort of editing that I do, and the 'layers' of background imagery have improved greatly over the years. So we do already have some of the tools needed for this? What would be nice is if we could select a background layer when simply displaying - such as the contour data? Over the weekend I'd finally fired up JOSM and needed QGIS to manipulate some of the datasources that I'm currently playing with. Again we have good tools there that already understand layers. In my own case the tool that is missing here is some sort of 'merge' which will take 'ways' from two layers and provides a merged layer with a clean set of unique ways. Actually there may be - and I just haven't found it? The 'problem' is where we need to combine ways from polyline shapes and create way segments where in my case 'boundaries' share a way. I'm sure that the same 'problem' is the basis of MOST imports of raw data? Shape files do have a nice database of objects which we can work against, and the merged data simply adds a 'way' table providing the relations to the original objects. Also processing these tables will allow additional tag information to be included if appropriate. I have no problem making data do what I want, the bit I need help with is processing and comparing the ways :) The bit I have an issue with in your blog is 'different data at different zoom levels'. Although I suspect it is just a difference of terminology. There have been several discussions in the past about micro vs. macro mapping, and adding fine detail needs to be done without affecting the macro view of the data. If this is done on it's own layer but still links to the macro level of the data then it may actually be a sensible way forward? Turning the problem on it's head, many applications USING the data don't need the 'graphics' until it comes to time to display? So FILTERING the data to provide a view more appropriate to the task is probably more important than 'different data'? The conflict that I'm hitting at the moment is that of adding 'footway' details to a map that is 'road centric'. 'converting' details such as pedestrian access across a roundabout to extra tags on a 'data level' where only the roundabout level exists is where the problems start? My own situation is that I HAVE the data sources to provide the overlays ... political boundaries ... and they georeference OSM nicely, but I now need to put the two together. In addition I need to be able to hot spot the areas to go down through the levels. This may be easier in the short term as simple snapshot views as images, but to be able to retain live OSM as a background? -- Lester Caine - G8HFL - Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
2012/9/24 Jochen Topf joc...@remote.org: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html If anybody wants to comment, I think this mailing list is the right place. IMHO there are different requirements for these layers according to what is on them and how it is related to the data on other layers. E.g. the birds routes would not create much problems because they are only roughly linked to current OSM-data, while for the historic data layer I think it would be desirable to have that directly linked (or at least a possibility to link it) to the current data. This is important when there are remains of the historic objects that are still (also partly) present in the current world. These could be either physical but also conceptual (e.g. parcels of a roman castrum which are still valid for todays town, leading the streets (=voids) to be where they used to be). Other examples for this might be city walls, ruins and other remains of historic buildings, historic walls, ...). The problem here is not with static data but arises from the fact that our current OSM data is in continuous motion: as soon as someone moves the current city wall (or refines it) in order to improve it, the historic-layer city-walls should also be refined (or they will get out of sync). We could maybe have something like hardlinks on filesystems for OSM-objects (nodes/way/relations) to solve this? In other cases it might not be desirable that historic objects change when the current objects get modified (i.e. this will also raise complexity a lot for the mapper, as he will have to decide for his edits whether they should also be applied to linked data, which is likely specialist data). Another similar concern I have with layers is that of fragmentation of the data which currently is all in the one main layer. In the past there were some people asking for separate thematic layers like landuse (e.g. in order to not show them in their editor), and introducing a layer-system might likely lead to fullfilling this desire. I see this as a problem because landuse is strongly tied to other objects like streets, building lots, and other polygons (e.g. amenity, leisure, place-polygons) and moving or editing only part of this data will also lead to out-of-sync-geometry between layers (won't fit one over the other). To avoid this people would have to look at all layers, which in the end eliminates the benefits of separate layers. cheers, Martin ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
The problem with lengthy blog posts is that they result in lengthy replies and probably a long thread :-) I think most users never look at the edit history and most of us that occasionally do look at it, do it mainly to get some idea about the validity of the data. To use our data to make historical maps, we have to finish making the current snapshot of the world first, then it might be possible to start looking at a way of making different more or less linked snapshots in time. I agree it might not be easy, but I think that just because it is difficult to build a space shuttle, that should not stop us from trying to build the car that most of us actually need first. The rendering performance problem for lower zoom levels, is probably better handled during import/update. If apmon keeps improving the performance of osm2pgsql, then it might be realistic to have a normal high zoom database and a separate low zoom database where only the tags need for low zoom levels are imported and the geometries are simplified (it might be as simple as using ST_simplify, but I have no experience with that and currently no access to capable hardware for testing). From Frederics SOTM slides it looks like the sweet spot is around zoom level 8. JOSM already handles different layers quite well including copy/cut/paste and basically just needs to keep track of the different data sources and have an option to paste with reference. Other editors that don't want to bother the user with the complications of layers can have a combined view of the layers and silently add and/or change objects in the appropriate layers depending on the type. This of course requires that we split the main database into layers by tags. If we split the layers by tags, the layer repository can be used to keep track of what layer a tag belongs to. That will take away a little of our freedom to tag anyway you like, by requiring that you register your new tag with the repository first. We could use something like the Tag Central [1][2] idea for that and I don't think it would be such bad thing, if you were required to document the tags you invent. It would also make it easier for other bird watchers to discover that there is a bird migration layer and how to use the special tags for it. The layer repository could also have an optional link to a tile server, that you can use as background, ex. you can work on the admin area layer without having to load the entire dataset for a country - I didn't try, but I expect JOSM would not handle loading the France or Germany extracts very well, if at all. I do share Martins concerns about how to handle updates to linked data, but maybe that can be solved by having both hard and soft links, so the user that creates the link makes the decision. That will also force you to think about if these two areas are actually linked or did you just reuse the nodes that were conveniently already there? If you have both layers open, you could have a also update soft links mode for those that know what they are doing and in the historic layer we could keep the soft links in an un-linked, but not yet completely destroyed state, where somebody can manually check if the link should be restored or removed. That will not completely solve the problem with historic maps, but if that is the only issue, I don't think that should stop us. @Lester: Did you try the merge layer feature in JOSM or did I misunderstand your problem? [1] Video: http://vimeo.com/14776099 [2] Slides: http://www.frankieandshadow.com/sotm10/tagcentral.pdf /Jais On Mon, Sep 24, 2012 at 4:35 PM, Martin Koppenhoefer dieterdre...@gmail.com wrote: 2012/9/24 Jochen Topf joc...@remote.org: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html If anybody wants to comment, I think this mailing list is the right place. IMHO there are different requirements for these layers according to what is on them and how it is related to the data on other layers. E.g. the birds routes would not create much problems because they are only roughly linked to current OSM-data, while for the historic data layer I think it would be desirable to have that directly linked (or at least a possibility to link it) to the current data. This is important when there are remains of the historic objects that are still (also partly) present in the current world. These could be either physical but also conceptual (e.g. parcels of a roman castrum which are still valid for todays town, leading the streets (=voids) to be where they used to be). Other examples for this might be city walls, ruins and other remains of historic buildings, historic walls, ...). The problem here is not with static data but arises from the fact that our current OSM data is in continuous motion: as soon as someone moves the current city wall (or refines it) in order to improve it, the historic-layer city-walls should also be refined (or they will get out of sync). We could maybe have
Re: [OSM-talk] Multiple Layers for OSM
On Mon, 2012-09-24 at 09:36 +0200, Jochen Topf wrote: It turns out there are many other interesting uses of multiple layers but also many technical and social questions around them. I have written down my thoughts on this subject in a (rather lengthy) blog post: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html what do you think of the Potlatch 2 vector backgrounds [1] and snapshot server [2] as steps in the direction of fixing this? cheers, matt [1] http://wiki.openstreetmap.org/wiki/Potlatch_2_merging_tool [2] http://wiki.openstreetmap.org/wiki/Snapshot_Server ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
This is a very interesting thread and I need a little more time to understand what is in it so far. I do want to make one technical comment. It is possible to to have multiple databases and still have unique IDs across them. An ID server can be set up to create the IDs. When one database creates a new object it requests an ID from the ID server rather than creating an ID locally. (And of course, if you are creating a lot of IDs, you would make one request to the ID server in bulk to get all the IDs needed.) Dave On Mon, Sep 24, 2012 at 12:36 AM, Jochen Topf joc...@remote.org wrote: In the recent discussion about the the imports in France and DWG governance the issue of multiple layers in OSM came up again. If we had some kind of layer system we could stage imports through them instead of adding all data to the OSM database directly and this would help finding problems etc. It turns out there are many other interesting uses of multiple layers but also many technical and social questions around them. I have written down my thoughts on this subject in a (rather lengthy) blog post: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html If anybody wants to comment, I think this mailing list is the right place. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
Or we could use GUIDs, but if we use the linking approach, then each layer could have its own IDs and if the links are implemented as tags would it be possible to do it without changes to the API? /Jais On Mon, Sep 24, 2012 at 11:11 PM, Dave Sutter sut...@intransix.com wrote: This is a very interesting thread and I need a little more time to understand what is in it so far. I do want to make one technical comment. It is possible to to have multiple databases and still have unique IDs across them. An ID server can be set up to create the IDs. When one database creates a new object it requests an ID from the ID server rather than creating an ID locally. (And of course, if you are creating a lot of IDs, you would make one request to the ID server in bulk to get all the IDs needed.) Dave On Mon, Sep 24, 2012 at 12:36 AM, Jochen Topf joc...@remote.org wrote: In the recent discussion about the the imports in France and DWG governance the issue of multiple layers in OSM came up again. If we had some kind of layer system we could stage imports through them instead of adding all data to the OSM database directly and this would help finding problems etc. It turns out there are many other interesting uses of multiple layers but also many technical and social questions around them. I have written down my thoughts on this subject in a (rather lengthy) blog post: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html If anybody wants to comment, I think this mailing list is the right place. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
Jais Pedersen wrote: @Lester: Did you try the merge layer feature in JOSM or did I misunderstand your problem? There may well be a a tool that does what I'm looking at ;) I have several 'layers' of boundary data, and SOME of the lower level 'ways' are used to define the same boundaries above, so in order to make a single layer, each shape needs to be broken down into a set of ways so we have a unique set of ways and a set of relations that select a 'closed loop' from the available set of ways for each of the original shapes. Building a database of the nine layers of shapes is the easy bit, as I already have that in another application anyway. It IS merging the ways from each layer and working out where to break the original shapes from each layer so they can be recreated simply be selecting the correct combination of way segments. Some segments are used on all 'boundary layers' while others only fill the gaps on perhaps on or two. In my naive way of working, it's looks like just identifying every node that matches from layer to layer and ending up with the biggest set of nodes that match. Merging onto the base OSM map should just be a version of the same process since a small subset of the ways have already been merged, or features already on the map may be usable in the matching process. Where it falls over is where in my case the OS borderline data has a large number of nodes which have been 'stylised' on the map already so do we throw away the more complex import data or keep the simplified view? Manual intervention on the merge process. BUT it does make more sense to me to keep the bulk of this imagery ON a separate database since displaying it or not should be optional at the viewer? -- Lester Caine - G8HFL - Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
I had started this page a few days back to write down the ideas I have had on OSM layers for quite a while: http://wiki.openstreetmap.org/wiki/OSM_Layers It would be a good idea to list out the different approaches to the layer problem on the wiki so that we can compare ideas. -- Arun Ganesh (planemad) http://en.wikipedia.org/wiki/User:Planemad http://j.mp/ArunGanesh ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Multiple Layers for OSM
I liked the Tag Central presentation. I have searched for more information but I can't find much. Has there been any more development on the Tag Central idea? Dave On Mon, Sep 24, 2012 at 6:54 AM, Jais Pedersen j...@pedersens.net wrote: The problem with lengthy blog posts is that they result in lengthy replies and probably a long thread :-) I think most users never look at the edit history and most of us that occasionally do look at it, do it mainly to get some idea about the validity of the data. To use our data to make historical maps, we have to finish making the current snapshot of the world first, then it might be possible to start looking at a way of making different more or less linked snapshots in time. I agree it might not be easy, but I think that just because it is difficult to build a space shuttle, that should not stop us from trying to build the car that most of us actually need first. The rendering performance problem for lower zoom levels, is probably better handled during import/update. If apmon keeps improving the performance of osm2pgsql, then it might be realistic to have a normal high zoom database and a separate low zoom database where only the tags need for low zoom levels are imported and the geometries are simplified (it might be as simple as using ST_simplify, but I have no experience with that and currently no access to capable hardware for testing). From Frederics SOTM slides it looks like the sweet spot is around zoom level 8. JOSM already handles different layers quite well including copy/cut/paste and basically just needs to keep track of the different data sources and have an option to paste with reference. Other editors that don't want to bother the user with the complications of layers can have a combined view of the layers and silently add and/or change objects in the appropriate layers depending on the type. This of course requires that we split the main database into layers by tags. If we split the layers by tags, the layer repository can be used to keep track of what layer a tag belongs to. That will take away a little of our freedom to tag anyway you like, by requiring that you register your new tag with the repository first. We could use something like the Tag Central [1][2] idea for that and I don't think it would be such bad thing, if you were required to document the tags you invent. It would also make it easier for other bird watchers to discover that there is a bird migration layer and how to use the special tags for it. The layer repository could also have an optional link to a tile server, that you can use as background, ex. you can work on the admin area layer without having to load the entire dataset for a country - I didn't try, but I expect JOSM would not handle loading the France or Germany extracts very well, if at all. I do share Martins concerns about how to handle updates to linked data, but maybe that can be solved by having both hard and soft links, so the user that creates the link makes the decision. That will also force you to think about if these two areas are actually linked or did you just reuse the nodes that were conveniently already there? If you have both layers open, you could have a also update soft links mode for those that know what they are doing and in the historic layer we could keep the soft links in an un-linked, but not yet completely destroyed state, where somebody can manually check if the link should be restored or removed. That will not completely solve the problem with historic maps, but if that is the only issue, I don't think that should stop us. @Lester: Did you try the merge layer feature in JOSM or did I misunderstand your problem? [1] Video: http://vimeo.com/14776099 [2] Slides: http://www.frankieandshadow.com/sotm10/tagcentral.pdf /Jais On Mon, Sep 24, 2012 at 4:35 PM, Martin Koppenhoefer dieterdre...@gmail.com wrote: 2012/9/24 Jochen Topf joc...@remote.org: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html If anybody wants to comment, I think this mailing list is the right place. IMHO there are different requirements for these layers according to what is on them and how it is related to the data on other layers. E.g. the birds routes would not create much problems because they are only roughly linked to current OSM-data, while for the historic data layer I think it would be desirable to have that directly linked (or at least a possibility to link it) to the current data. This is important when there are remains of the historic objects that are still (also partly) present in the current world. These could be either physical but also conceptual (e.g. parcels of a roman castrum which are still valid for todays town, leading the streets (=voids) to be where they used to be). Other examples for this might be city walls, ruins and other remains of historic buildings, historic walls, ...).
Re: [OSM-talk] Multiple Layers for OSM
Jochen Topf joc...@remote.org wrote: The layer rendering thing in first position has nothing to do with what I am suggesting. That can already be done with current technology. But the second one has (People want to host their own layers). :-) Jochen On Mon, Sep 24, 2012 at 10:20:29AM +0200, Mike Dupont wrote: Date: Mon, 24 Sep 2012 10:20:29 +0200 From: Mike Dupont jamesmikedup...@googlemail.com To: Jochen Topf joc...@remote.org Cc: talk@openstreetmap.org Subject: Re: [OSM-talk] Multiple Layers for OSM Well I support this idea and see it as long needed, it is also the top voted item on the ideascale, http://fosm.ideascale.com/ to be able to render as layers. The layers could be used also to split up different licensed data as well. On Mon, Sep 24, 2012 at 9:36 AM, Jochen Topf joc...@remote.org wrote: In the recent discussion about the the imports in France and DWG governance the issue of multiple layers in OSM came up again. If we had some kind of layer system we could stage imports through them instead of adding all data to the OSM database directly and this would help finding problems etc. It turns out there are many other interesting uses of multiple layers but also many technical and social questions around them. I have written down my thoughts on this subject in a (rather lengthy) blog post: http://blog.jochentopf.com/2012-09-23-multiple-layers-for-osm.html If anybody wants to comment, I think this mailing list is the right place. If someone is hosting their own layer, doesn't that mean that it isn't being hosted by the OSM database itself? This sounds like more of a rendering issue than a back-end database issue. -- John F. Eldredge -- j...@jfeldredge.com Reserve your right to think, for even to think wrongly is better than not to think at all. -- Hypatia of Alexandria ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
