Re: [OSM-dev] which tool is better for osm data import to postgres
Hi Stefan, On 05.06.2011, at 01:09, Stefan Keller wrote: I'm the project leader of osm2gis. I've tried to do a characterization of the projects mentioned before: http://dev.ifs.hsr.ch/redmine/projects/osminabox/wiki/Wiki Thanks for the comparison. I'm the author of Imposm, so I have some clarifications :) It is written in Python and _uses_ C/C++ libraries. It is used in production for more than a year now, but only became open source recently. It does not use the raw OSM database schema like Osmosis, so in this case it is similar to osm2pgsql, but it offers lots of features that are not possible with osm2pgsql. It supports custom database schemas with separate tables for each data type (highways, buildings, waterareas, etc.), unified values (e.g. tunnel as true/false instead of 1/yes/true/t/0/no/...), generalized tables, etc. In short, it does everything to make the fastest rendering possible and this is one of the reasons that Imposm is used for the annual WMS benchmarking at the FOSS4G in Denver. I think it would be a great idea to move your comparison to the OSm wiki, so that everyone can extend the list of features. What do you think? Regards, Oliver -- Oliver Tonnhofer| Omniscale GmbH Co KG| http://omniscale.de http://mapproxy.org | https://bitbucket.org/olt | @oltonn ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql for 64-bit IDs
Hi, 2011/5/24 Jukka Rahkonen jukka.rahko...@latuviitta.fi: ... Spatialite_osm_map with the finland.osm dataset from Geofabrik is faster than osm2pgsql in slim mode with my laptop. I have not studied spatialite_osm_map thoroughly and I do not know how well it has solved the mystery of OSM polygons but that is something that should really be solved somewhere pretty close to the OSM primary database, if not inside it. True. I'm promoting that change to polygons as a first class data type since years... In fact, if ways would be encoded as a first class data type too (e.g. as linestring encoded by way_id and a set of anon. coordinates) then we would'nt run out of the int4 byte range so fast! = Perhaps it's now time to think about this linestring/polygon issue in OSM? For many use cases it might be nice to have ready made Spatialite country files available, with tables filled with OSM data, with indexes and with a nice set of views for simple querying. Instead of downloading raw country Good idea. I'm supporting in general the proposal of Spatialite as the Shapefile 2.0 which serves as a binary single desktop file exchange format. There are and will be libraries for reading and writing Spatialite as there are libs for Shapefile now. Protocol buffers (PBF) is a nice alternative but not quite at the same level as Spatialite since PBF is purely serializing structured data with no specification on how to encode spatial data types and with no support in the GIS field up to now. Yours, Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] which tool is better for osm data import to postgres
Hi Oliver 2011/6/5 Oliver Tonnhofer o...@omniscale.de: Hi Stefan, On 05.06.2011, at 01:09, Stefan Keller wrote: I'm the project leader of osm2gis. I've tried to do a characterization of the projects mentioned before: http://dev.ifs.hsr.ch/redmine/projects/osminabox/wiki/Wiki Thanks for the comparison. I'm the author of Imposm, so I have some clarifications :) It is written in Python and _uses_ C/C++ libraries. I've added that. (...) it is similar to osm2pgsql, but it offers lots of features that are not possible with osm2pgsql. It supports custom database schemas with separate tables for each data type (highways, buildings, waterareas, etc.), unified values (e.g. tunnel as true/false instead of 1/yes/true/t/0/no/...), generalized tables, etc. So this seems to be similar with our osm2gis. osm2gis supports also diff update but we had problems importing whole planet. So we concentrated on managing regions which in turn makes diff update complicated (because diffs are always against whole planet). = Did you manage to import whole planet? In short, it does everything to make the fastest rendering possible and this is one of the reasons that Imposm is used for the annual WMS benchmarking at the FOSS4G in Denver. Oh, I thought that will be our osm2gis :- I think it would be a great idea to move your comparison to the OSM wiki, so that everyone can extend the list of features. What do you think? Good idea - but where? There's already http://wiki.openstreetmap.org/wiki/Software/Desktop but now it seems to be a need for Tools to import OSM data into PostgreSQL for rendering, servicing and editing use cases. Yours, Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql for 64-bit IDs
Hi, Stefan Keller wrote: True. I'm promoting that change to polygons as a first class data type since years... In fact, if ways would be encoded as a first class data type too (e.g. as linestring encoded by way_id and a set of anon. coordinates) then we would'nt run out of the int4 byte range so fast! = Perhaps it's now time to think about this linestring/polygon issue in OSM? The polygon issue is being thought about (but it is rather unlikely that any solution we chose will have internal sets of coordinates, and only then would your assumption about not running out of integers be true). Linestrings with internal geometries are very unlikely to happen, for a number of reasons, one of them being topology. I think that the OSM model is superior to the classic GIS linestring; the linestring can be a product down the road (e.g. after a processing step like osm2pgsql) but is less suitable as a master data type. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] which tool is better for osm data import to postgres
On 05.06.2011, at 14:01, Stefan Keller wrote: Thanks for the comparison. I'm the author of Imposm, so I have some clarifications :) It is written in Python and _uses_ C/C++ libraries. I've added that. Great. So this seems to be similar with our osm2gis. osm2gis supports also diff update but we had problems importing whole planet. So we concentrated on managing regions which in turn makes diff update complicated (because diffs are always against whole planet). = Did you manage to import whole planet? I use it mostly for country extracts, but it also works with planet imports. The last import of a planet file took 20 hours on a host with 8GB RAM and a SATA RAID 1, it should be much faster on a host with more RAM. In short, it does everything to make the fastest rendering possible and this is one of the reasons that Imposm is used for the annual WMS benchmarking at the FOSS4G in Denver. Oh, I thought that will be our osm2gis :- Sorry if I grabbed that spot :) Dane suggested imposm on one of the last IRC meetings and it was accepted. It is already deployed and there is a first import of Colorado. I think it would be a great idea to move your comparison to the OSM wiki, so that everyone can extend the list of features. What do you think? Good idea - but where? There's already http://wiki.openstreetmap.org/wiki/Software/Desktop but now it seems to be a need for Tools to import OSM data into PostgreSQL for rendering, servicing and editing use cases. Software/GeoSpatialDatabaseImport? Regards, Oliver -- Oliver Tonnhofer| Omniscale GmbH Co KG| http://omniscale.de http://mapproxy.org | https://bitbucket.org/olt | @oltonn ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql for 64-bit IDs
On Sun, Jun 5, 2011 at 8:39 AM, Frederik Ramm frede...@remote.org wrote: Linestrings with internal geometries are very unlikely to happen, for a number of reasons, one of them being topology. That's too bad. More than 90% of node ids are completely useless as they refer to only one way. This is a significant drag on the database, and especially on imports (especially under the current .osm format which requires a huge sort and join in order to import into a form suitable for rendering). Node ids could still be kept for nodes which reference more than one relation/way, as well as for POI nodes, but 90% or so of them could still be eliminated/recycled with very few changes. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql for 64-bit IDs
Hi, Anthony wrote: Imagine there's a way with anonymous points and you make a branch off of that. You'd have to convert that node to a proper node and split the linestring. We'd end up with a crippled system like routable shapefiles where you need an individual linestring for each bit of way between two intersections. Why would you split the linestring? Are you suggesting to have a linestring with anonymous coordinates, but some of them can be assigned IDs? Because if you want a proper topology, the way branching off must somehow reference the node instead of just the coordinate, else you wouldn't know if there's a connection. Do you care to elaborate on what the disadvantages would be? No, I'm not prepared to discuss them with you. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql for 64-bit IDs
Imagine there's a way with anonymous points and you make a branch off of that. You'd have to convert that node to a proper node and split the linestring. We'd end up with a crippled system like routable shapefiles where you need an individual linestring for each bit of way between two intersections. Why would you split the linestring? Are you suggesting to have a linestring with anonymous coordinates, but some of them can be assigned IDs? That's certainly a possibility, though it's not necessarily the best one. Because if you want a proper topology, the way branching off must somehow reference the node instead of just the coordinate, else you wouldn't know if there's a connection. There are many ways to indicate connections. They don't all involve referencing nodes, though some do. Do you care to elaborate on what the disadvantages would be? No, I'm not prepared to discuss them with you. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Rendering of surface
Hi, Sorry if this would be better suited for talk (or another) mailing list. Just following a thread that was started here, for good reason. Sincerely, I am surprised to see that there could be tension around that proposal. It is not about adding objects to the map, but actually about removing part of the casing of objects that are already there, so this should make you happy, Andy ;-) Martin, this is precisely about the rendering of information. Maybe you never checked how the data you had entered was rendered. I confess that I do sometimes. It can help find errors. And I think it can help particularly in the case of fledgling OSM communities, or beginner mappers. If you read the thread of discussion I was pointing to, you can see that there were also worries about not misrepresenting the actual status of roads to map users. I've also seen this worry in other geographical contexts. It is in this sense that a more accurate rendering would help mapping. (And of course, these conditions are not the easiest in which to run customised renderings). The dotted line convention is common to the (French) paper maps that I can look at. Applied to ways that would be there anyway, it does not seem to clutter the map, on the contrary, it makes it more readable to me. Of course, you can say this is subjective. But it happens to be there on topographic as well as road maps. This probably means that this is considered an important attribute of roads. On a related theme, about web maps, there was a recent discussion on talk-fr about G. now using data from the Institut Géographique National. Many examples were given of ways that are rendered the same as standard roads, even though they are either not drivable, private or even exist only as projects. Is this the kind of rendering model OSM would like to follow? Do you think it would encourage a good quality mapping? Or trust from the users? It is only natural, and fair enough, that the rendering compromises were apparently tuned for the UK, since OSM started there, or similar countries. But if the aim is to map the whole world, where the global proportion of unpaved roads or streets is much larger, then maybe slight adjustment to these compromises could be considered. Of course, I don't know what the cost (in time and resources) of this modification would be, nor how the Mapnik team proceeds to test the modifications. It just seemed, maybe naively, like a reasonable modification to suggest for potentially large benefits. Best regards, Jean-Guilhem -- pgp 0x5939EAE2 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Rendering of surface
On Sun, Jun 5, 2011 at 12:02 PM, Jean-Guilhem Cailton j...@arkemie.com wrote: Hi, Sorry if this would be better suited for talk (or another) mailing list. Just following a thread that was started here, for good reason. Sincerely, I am surprised to see that there could be tension around that proposal. I have a counter question. If rendering surface is so important to some users, why have they not rendered it themselves? The database is provided. The tools are provided. Even an interested, enthusiastic and clever community is provided. Why must the problem of rendering surface tags be solved on a specific rendered layer or service? Was not OCM first created by a small, interested community, without official support? Come on. Do it! Render surface values. It will be wonderful and useful! I want to see it. But it isn't high enough on my list of things to do for myself. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Rendering of surface
On 6 June 2011 03:22, Richard Weait rich...@weait.com wrote: Why must the problem of rendering surface tags be solved on a specific rendered layer or service? The answer is obvious, it's easier to push a small change onto the main mapnik layer than running their own system. I'm not sure how much the custom rendering solution will change things, possibly not much, it'll still be easier to ask someone else to change their system to render differently. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Rendering of surface
2011/6/5 John Smith deltafoxtrot...@gmail.com: On 6 June 2011 03:22, Richard Weait rich...@weait.com wrote: Why must the problem of rendering surface tags be solved on a specific rendered layer or service? The answer is obvious, it's easier to push a small change onto the main mapnik layer than running their own system. I'm coming back to the original issue. My two arguments supporting to render surface: 1. In our country we still have about 25% of roads unpaved, so surface (paved/unpaved) is way more important parameter for almost any user of any road map. The current primary/secondary/etc classification is here relevant for officials and no much more. So also in our cartographic tradition we still use primary dimension (colors) for surface, and secondary (width, style) for other road parameters. 2. Current lack of surface style in Mapnik main stylesheet has impact to real mapping: people tend to not add the surface tag, just because they do not see it rendered in The Map. And this is a major issue for many applications. ps. we are working on own rendering server and we will solve it for ourselves. It is not us just being lazy not to set up own custom style. But: (1) it would not solve the issue of non-tagging for most who do not see our customized rendering. And (2) most important style fixes should be in the main stylesheet, and my impression is that this is one of those. For both practical use, and to enforce proper tagging. Maybe non-paved road thing looks like alien for you in Germany or UK, but in less or no-developed countries (covering 95+% of world population/maps) this is important everyday issue. Please don't ignore us. -- Jaak Laineste ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Rendering of surface
On 6 June 2011 04:32, Jaak Laineste jaak.laine...@gmail.com wrote: relevant for officials and no much more. So also in our cartographic tradition we still use primary dimension (colors) for surface, and secondary (width, style) for other road parameters. Australia has a large amount of dirt roads, I'm pretty sure someone tagged a highway=primary as non-paved, however unpaved roads are usually tagged as dashed lines. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql for 64-bit IDs
On Sun, Jun 5, 2011 at 8:41 AM, Anthony o...@inbox.org wrote: On Sun, Jun 5, 2011 at 8:39 AM, Frederik Ramm frede...@remote.org wrote: Linestrings with internal geometries are very unlikely to happen, for a number of reasons, one of them being topology. That's too bad. More than 90% of node ids are completely useless as they refer to only one way. Out of curiosity, I did a test on this, because I was planning on exploiting this 'conventional wisdom'. I was hoping that 99% of nodes occur in one way, but on a 2010-08 planet, 85% of nodes occur in one way, 12% in 2, and 1% in more than two.. Scott ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql for 64-bit IDs
On Sun, Jun 5, 2011 at 7:38 PM, Scott Crosby sc...@sacrosby.com wrote: On Sun, Jun 5, 2011 at 8:41 AM, Anthony o...@inbox.org wrote: On Sun, Jun 5, 2011 at 8:39 AM, Frederik Ramm frede...@remote.org wrote: Linestrings with internal geometries are very unlikely to happen, for a number of reasons, one of them being topology. That's too bad. More than 90% of node ids are completely useless as they refer to only one way. Out of curiosity, I did a test on this, because I was planning on exploiting this 'conventional wisdom'. I was hoping that 99% of nodes occur in one way, but on a 2010-08 planet, 85% of nodes occur in one way, 12% in 2, and 1% in more than two.. Yeah, last time I checked it was just slightly over 90%. But even if it's only 80% of nodes that are completely useless, I think it's worth doing something about. Or at least considering doing something about, and not writing it off as it has to be that way but I'm not going to tell you why. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Help - can't upload JOSM edits
Thanks Jo. I spent yesterday evening starting afresh and re-working through the survey, uploading initially after one edit, after importing the audio, after switching on Bing. It all seemed to work fine, and I kept going, uploading every few minutes without any errors. So I don't know what the problem was - it may have been our internet connection - but whatever - it seems to have gone away (I hope). Thanks for your help. Richard On Fri 03/06/11 23:53 , Jo winfi...@gmail.com sent: Select one element and try Send selection to server. If that works, you know the problem is not with your internet connection. Then you can send all the rest one by one until you find the element that's causing the problem. ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
