[OSM-dev] Getting admin_level=* from relations to render properly
It occurred to me a few weeks ago that ways shouldn't be tagged with the admin_level, but instead use the information from relations, but this doesn't work, in fact I removed the admin_level tag from a way that makes up part of a state border assuming the information would be used from the relation instead, but it just didn't render at all until I reinstate the admin_level on the way. I haven't figured out what needs to be fixed in the mapnik style sheet to fix this, can anyone tell me what needs tweaking to fix this? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
On 28-4-2010 8:28, John Smith wrote: It occurred to me a few weeks ago that ways shouldn't be tagged with the admin_level, but instead use the information from relations, but How did this occur to you? Tagging the ways is even explicitly documented on the wiki. this doesn't work, in fact I removed the admin_level tag from a way that makes up part of a state border assuming the information would be used from the relation instead, but it just didn't render at all until I reinstate the admin_level on the way. If the relation forms a closed loop, and is tagged as a boundary, it should currently render. I haven't figured out what needs to be fixed in the mapnik style sheet to fix this, can anyone tell me what needs tweaking to fix this? Should work currently. The major drawback to also drawing boundary relations is that they can stack (in osm2pgsql+mapnik). Where a tagged boundary way is part of n boundary relations, you will see n+1 overlapping lines in the render. IMO this is ugly, and you get no sense of the actual admin levels involved. Rendering just the boundary way makes for a clean border render, with a predictable appearance. To this end, the wiki documentation describes tagging the way with the highest order (lowest value) of admin_level. As far as I'm aware, ti...@home already take this approach, and does not render boundary relations. I trust people will now bring the don't tag for the renderer mantra into play, but in my mind, a boundary relation(*) describes the polygon, so the administrative *area*, and the ways describe the demarcation between administrative areas. Thus, it's the ways that should appear on the rendered map, not the area. *) Actually tagged as a multipolygon, because, hey, that's what it really is. -- Lennard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
On 28 April 2010 17:00, Lennard l...@xs4all.nl wrote: How did this occur to you? Tagging the ways is even explicitly documented on the wiki. That makes no sense when rendering can derive it from relations to pick the most important (lowest admin_level value) without people needing to know which way is part of which relation. If the relation forms a closed loop, and is tagged as a boundary, it should currently render. Ok, here's the way and here's the relation and here's the rendering, please explain why it's not rendering the relation how ways nearby render: http://www.openstreetmap.org/browse/way/32295414 http://www.openstreetmap.org/browse/relation/80372 http://osm.org/go/uTRwIeeW- Should work currently. The major drawback to also drawing boundary relations is that they can stack (in osm2pgsql+mapnik). Where a tagged boundary way is part of n boundary relations, you will see n+1 overlapping lines in the render. IMO this is ugly, and you get no sense of the actual admin levels involved. That may be what's happening, in which case that's pointing out where we need to look, we should structure the SQL query to return only one way, with the highest admin_level. Rendering just the boundary way makes for a clean border render, with a predictable appearance. To this end, the wiki documentation describes tagging the way with the highest order (lowest value) of admin_level. As far as I'm aware, ti...@home already take this approach, and does not render boundary relations. So you admit the logic can be problematic with the rendering, doesn't that mean we're tagging incorrectly for renderers? I trust people will now bring the don't tag for the renderer mantra How'd ya guess? :) into play, but in my mind, a boundary relation(*) describes the polygon, so the administrative *area*, and the ways describe the demarcation between administrative areas. Thus, it's the ways that should appear on the rendered map, not the area. Ok so you admit there is a problem with rendering, but instead of fixing broken rendering software you suggest we do a lot of manual work to over come the short falls in the software, this imho is the wrong way to go, so my original question still stands, but you have pointed out where I should be looking to fix the problem properly. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
I think the problem with mapnik is this query: select way,admin_level from planet_osm_roads where boundary='administrative' it just select all ways and doesn't try to limit the returned information to distinct ways or order the results at all, and I'm not familiar enough with pgSQL to fix it, if it was MySQL you can tell it to group by the column you want distinct results for, but I think I need subselects or joins to do the same thing in pgSQL... Although you'd probably need to go a step further and order by way and then relation, not just admin_level... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
El 28/04/2010 9:27, John Smith escribió: Ok, here's the way and here's the relation and here's the rendering, please explain why it's not rendering the relation how ways nearby render: http://www.openstreetmap.org/browse/way/32295414 http://www.openstreetmap.org/browse/relation/80372 http://osm.org/go/uTRwIeeW- Looks good to me. Where a tagged boundary way is part of n boundary relations, you will see n+1 overlapping lines in the render. IMO this is ugly, and you get no sense of the actual admin levels involved. Ugly gets short. The overlapping dashed lines just look horrible. http://www.openstreetmap.org/?lat=41.3067lon=-3.1946zoom=12 That may be what's happening, in which case that's pointing out where we need to look, we should structure the SQL query to return only one way, with the highest admin_level. I don't think so. I'd place my wages on a bot. Download the planet, check topology of admin_level ways and relations, calculate the non-needed bits, upload changes. Cheers, -- Iván Sánchez Ortega i...@sanchezortega.es ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
2010/4/28 Iván Sánchez Ortega i...@sanchezortega.es: I'd place my wages on a bot. Download the planet, check topology of admin_level ways and relations, calculate the non-needed bits, upload changes. So in other words instead of areas to figure out what is in the area we should just use a bot to tag each object with is_in=* tags? This is not the right way to do things, we need to fix the rendering software to work properly, not use bots so software devs have a simple way out. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
2010/4/28 Iván Sánchez Ortega i...@sanchezortega.es: Ugly gets short. The overlapping dashed lines just look horrible. http://www.openstreetmap.org/?lat=41.3067lon=-3.1946zoom=12 That usually occurs because of broken relational multipolygons... However it's the same problem I'm trying to solve, limiting the number of times the same shared segments get rendered. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
On 28 April 2010 18:33, Lennard l...@xs4all.nl wrote: No, if they were broken, they wouldn't render in the first place. They render as lines instead of areas, but they still render, I've dealt with this issue a lot of time with broken postcodes in Australia, fix the relation you fix the rendering... Easy: slap a where osm_id0 on that query you posted, and only render the (correctly tagged) ways. :) That doesn't solve the issue, that just excludes information from relations, we need to use relation information if the way information doesn't exist, similar to the way information from relations work for other objects. This is the order things need to be found, through a subselect or join, after finding distinct segments, from those segments we need to sort by way admin_level, and then by relation admin_level, simply excluding information only glosses over the issue by excluding broken multipolygons. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
On 28-4-2010 9:27, John Smith wrote: That makes no sense when rendering can derive it from relations to pick the most important (lowest admin_level value) without people needing to know which way is part of which relation. You are on the same train of thought I was on a year or so ago. Ok, here's the way and here's the relation and here's the rendering, please explain why it's not rendering the relation how ways nearby render: Looks fine. I can't look into the osm.org tile server db to see how it wound (or didn't) up in there. Could be any of a number of reasons. So you admit the logic can be problematic with the rendering, doesn't that mean we're tagging incorrectly for renderers? Don't tag the member boundary ways, and you're right. Ok so you admit there is a problem with rendering, but instead of There is no problem with rendering. The renderer just renders what is there, which includes tagged member ways. -- Lennard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
El 28/04/2010 10:49, Lennard escribió: So you admit the logic can be problematic with the rendering, doesn't that mean we're tagging incorrectly for renderers? Don't tag the member boundary ways, and you're right. So what? Area boundaries will be rendered twice, once per area. You'll be back at the start :-/ -- Iván Sánchez Ortega i...@sanchezortega.es ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
On 28 April 2010 18:49, Lennard l...@xs4all.nl wrote: Looks fine. I can't look into the osm.org tile server db to see how it wound (or didn't) up in there. Could be any of a number of reasons. You don't need to, although it would be great if the OSM tile server setup was better documented on how to put everything together, but that's an issue for another thread. However the style sheet is published to the SVN server: http://svn.openstreetmap.org/applications/rendering/mapnik/osm.xml Don't tag the member boundary ways, and you're right. As someone else pointed out, you will get multiple overlapping ways being rendered which isn't the most ideal outcome. There is no problem with rendering. The renderer just renders what is there, which includes tagged member ways. The rendering software does 2 things, it takes a style sheet and applies that to data it uses, just because the software does what it's told doesn't mean it has been told to do the right thing. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
On 28-4-2010 10:46, John Smith wrote: They render as lines instead of areas, but they still render, I've dealt with this issue a lot of time with broken postcodes in Australia, fix the relation you fix the rendering... type=multipolygon boundary relations do not render. Just realised after sending my previous reply. The fix is easy, the result gobsmackingly ugly (read: overlapping lines). That doesn't solve the issue, that just excludes information from relations, we need to use relation information if the way information doesn't exist, similar to the way information from relations work for other objects. That cannot be solved from mapnik. This is a job to be done during the db import for mapnik, both for non-slim mode (a one off job) and slim mode imports, which have to take into account loading diff files. At some point, it just sounds much simpler to correctly tag the member ways. This is *not* wrong, they *are* boundary demarcation lines. This is the order things need to be found, through a subselect or join, after finding distinct segments, from those segments we need to sort by way admin_level, and then by relation admin_level, simply excluding information only glosses over the issue by excluding broken multipolygons. By the time mapnik sees the data, it has been processed. Relations could have been converted into line geometries, and there is no way for mapnik to discern those from the proper ways, other than the test for osm_id0. I suggest you play around with a limited dataset, and see how that ends up in the pg db, in which tables, for different taggings. -- Lennard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
On 28-4-2010 10:58, Iván Sánchez Ortega wrote: So you admit the logic can be problematic with the rendering, doesn't that mean we're tagging incorrectly for renderers? Don't tag the member boundary ways, and you're right. So what? Area boundaries will be rendered twice, once per area. You'll be back at the start :-/ I know, that wasn't my point. John argues boundary way tags shouldn't matter, when the way is a member of a boundary relations, and it should inherit some admin_level from those. He implies it's unnecessary tagging for the renderer. -- Lennard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
El 28/04/2010 10:13, John Smith escribió: This is not the right way to do things, we need to fix the rendering software to work properly, not use bots so software devs have a simple way out. No, no, no. There is no point being an architecture astronaut[1] and making sure we've got the most perfect rendering system in the whole universe. [1] http://www.joelonsoftware.com/articles/fog18.html This is less about thinking and more about doing. Believe it or not, the first method that is able to solve the boundaries problem will win. No matter if it's a bot, a osm2pgsql patch, or an ugly SQL query on the mapnik stylesheet side of things. And, personally, I think that a bot is the simplest, quickest way of achieving this. Cheers, -- Iván Sánchez Ortega i...@sanchezortega.es ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
On 28 April 2010 19:00, Lennard l...@xs4all.nl wrote: That cannot be solved from mapnik. This is a job to be done during the db import for mapnik, both for non-slim mode (a one off job) and slim mode imports, which have to take into account loading diff files. At some point, it just sounds much simpler to correctly tag the member ways. This is *not* wrong, they *are* boundary demarcation lines. This can be solved in mapnik by using more selective query statements, currently there is limited or no limitation placed on the admin boundary queries... So yes this can be solved in mapnik/pgsql, however I lack the SQL knowledge to fix it, or this thread wouldn't have gone on for this long without a solution. As for pre-processing being more efficient, possibly, but you need to figure out the same sort of logic to figure it out in any case. By the time mapnik sees the data, it has been processed. Relations could have been converted into line geometries, and there is no way for mapnik to discern those from the proper ways, other than the test for osm_id0. I suggest you play around with a limited dataset, and see how that ends up in the pg db, in which tables, for different taggings. You are right about the osm_id, but I lack the ability to come up with a suitable query to do the test, which is why this thread is still going on. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
2010/4/28 Iván Sánchez Ortega i...@sanchezortega.es: This is less about thinking and more about doing. Believe it or not, the first method that is able to solve the boundaries problem will win. No matter if it's a bot, a osm2pgsql patch, or an ugly SQL query on the mapnik stylesheet side of things. That's all hand waving and appeals to emotion and diverting attention away from fixing a problem. And, personally, I think that a bot is the simplest, quickest way of achieving this. I doubt this is the quickest or simplest, and certainly no where near the most efficient. Using a bot to brute force add information that can be derived from other bits of data is a waste of time, just like is_in=* tagging is just as useful for the same reasons. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Getting admin_level=* from relations to render properly
I've come to the conclusion that shared segments aren't stored how I was assuming, so the next best thing would be doing more pre-processing as Lennard suggested. While using a bot is A solution, but I don't think using a bot is the best solution, or even a good one. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSM Data into J2ME Application
Hello everyone, may I introduce myself: My Name is Daniel Hänßgen and I'm a student in Applied Informatics at the University of Applied Siences Hannover in Germany. Currently I'm developing a navigation system for blind and visually impared people (at least I'm trying to!) called LoroDux. LoroDux has to be in Java ME (CLDC-1.1). And I just don't know how to get the OSM data in my application... Has anyone tried this yet and may explain me, what I've got to do so. Thank you! Kind Regards Daniel P.S.: Hope you understand me, my english is not the best... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Data into J2ME Application
Hi Daniel, Can you please be more specific of what you're trying to do? - Do you want to fetch the data during an edge/umts connection? - Should the data be downloaded at home? - Are you planning to read the raw osm-xml? - Or do you want it in some more adequate form (eg. a sqlite database) - Do you already have some kind of internal data format we could convert the osm format into? I'm sure we'll find a way for you. Peter Daniel Hänßgen schrieb: Hello everyone, may I introduce myself: My Name is Daniel Hänßgen and I'm a student in Applied Informatics at the University of Applied Siences Hannover in Germany. Currently I'm developing a navigation system for blind and visually impared people (at least I'm trying to!) called LoroDux. LoroDux has to be in Java ME (CLDC-1.1). And I just don't know how to get the OSM data in my application... Has anyone tried this yet and may explain me, what I've got to do so. Thank you! Kind Regards Daniel P.S.: Hope you understand me, my english is not the best... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Data into J2ME Application
On Wed, 2010-04-28 at 16:27 +0200, Daniel Hänßgen wrote: Hello everyone, may I introduce myself: My Name is Daniel Hänßgen and I'm a student in Applied Informatics at the University of Applied Siences Hannover in Germany. Currently I'm developing a navigation system for blind and visually impared people (at least I'm trying to!) called LoroDux. LoroDux has to be in Java ME (CLDC-1.1). And I just don't know how to get the OSM data in my application... Has anyone tried this yet and may explain me, what I've got to do so. Thank you! Kind Regards Daniel P.S.: Hope you understand me, my english is not the best... you can obtain the entire planet data on planet.osm.org you'll need to compile it somehow so that the data relevant to the area of interest fits into your device ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Data into J2ME Application
Hi Daniel, a good starting point for you could be GPSmid .. This is a midlet dealing with OSM data + it is written in j2me.. you'll need eclipse and j2mepolish for it to compile (but those two are probably good for your application as well).. Since I'm not actively developing on GPSmid you might want to ask the people there (mailinglist / forum) for additional help on some concepts.. Have fun, Christian Daniel Hänßgen schrieb: Hello everyone, may I introduce myself: My Name is Daniel Hänßgen and I'm a student in Applied Informatics at the University of Applied Siences Hannover in Germany. Currently I'm developing a navigation system for blind and visually impared people (at least I'm trying to!) called LoroDux. LoroDux has to be in Java ME (CLDC-1.1). And I just don't know how to get the OSM data in my application... Has anyone tried this yet and may explain me, what I've got to do so. Thank you! Kind Regards Daniel P.S.: Hope you understand me, my english is not the best... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Data into J2ME Application
Hello Daniel, To parse XML data you'll need the JSR 172 API for Java ME, this includes the JAXP parser, which is a SAX-style XML parser. JSR172 also allows you to contact web services from a Java ME client which might be useful for you. if you have had any Java ME experience, you might want to look at http://www.free-map.org.uk/downloads/FreemapMobile/FreemapMobile_r3fe.zip This is a mobile OSM application I did some time ago. It doesn't actually parse OSM data, but it does contact a web service and parse XML, as well as download OSM tiles. Nick Daniel Hänßgen daniel.haenss...@stud.fh-hannover.de Sent by: dev-boun...@openstreetmap.org 28/04/2010 15:27 To dev@openstreetmap.org cc Subject [OSM-dev] OSM Data into J2ME Application Hello everyone, may I introduce myself: My Name is Daniel Hänßgen and I'm a student in Applied Informatics at the University of Applied Siences Hannover in Germany. Currently I'm developing a navigation system for blind and visually impared people (at least I'm trying to!) called LoroDux. LoroDux has to be in Java ME (CLDC-1.1). And I just don't know how to get the OSM data in my application... Has anyone tried this yet and may explain me, what I've got to do so. Thank you! Kind Regards Daniel P.S.: Hope you understand me, my english is not the best... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Data into J2ME Application
Hello Thank you Nick, Thank you Peter. The J2ME Application itself should run completly offline. So the OSM Data has to be processed on the PC / J2SE. I'm thinking of a JavaSE Application (Commandline is enough) that converts the OSM XML into a File, that can be copied into the LoroDux.jar (J2ME) oder copied to the device, and than tell LoroDux where the Data is stored. Ideas? Thanks! Daniel Zitat von Nick Whitelegg nick.whitel...@solent.ac.uk: Hello Daniel, To parse XML data you'll need the JSR 172 API for Java ME, this includes the JAXP parser, which is a SAX-style XML parser. JSR172 also allows you to contact web services from a Java ME client which might be useful for you. if you have had any Java ME experience, you might want to look at http://www.free-map.org.uk/downloads/FreemapMobile/FreemapMobile_r3fe.zip This is a mobile OSM application I did some time ago. It doesn't actually parse OSM data, but it does contact a web service and parse XML, as well as download OSM tiles. Nick Daniel Hänßgen daniel.haenss...@stud.fh-hannover.de Sent by: dev-boun...@openstreetmap.org 28/04/2010 15:27 To dev@openstreetmap.org cc Subject [OSM-dev] OSM Data into J2ME Application Hello everyone, may I introduce myself: My Name is Daniel Hänßgen and I'm a student in Applied Informatics at the University of Applied Siences Hannover in Germany. Currently I'm developing a navigation system for blind and visually impared people (at least I'm trying to!) called LoroDux. LoroDux has to be in Java ME (CLDC-1.1). And I just don't know how to get the OSM data in my application... Has anyone tried this yet and may explain me, what I've got to do so. Thank you! Kind Regards Daniel P.S.: Hope you understand me, my english is not the best... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Data into J2ME Application
On 29 April 2010 01:18, Daniel Hänßgen daniel.haenss...@stud.fh-hannover.de wrote: Hello Thank you Nick, Thank you Peter. The J2ME Application itself should run completly offline. So the OSM Data has to be processed on the PC / J2SE. I'm thinking of a JavaSE Application (Commandline is enough) that converts the OSM XML into a File, that can be copied into the LoroDux.jar (J2ME) oder copied to the device, and than tell LoroDux where the Data is stored. Have you looked at trekbuddy and how it handles data, it also works on j2me devices, but doesn't do any XML data processing, which most non-smartphone j2me handsets would do very poorly anyway. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Data into J2ME Application
El 28/04/2010 17:18, Daniel Hänßgen escribió: I'm thinking of a JavaSE Application (Commandline is enough) that converts the OSM XML into a File, that can be copied into the LoroDux.jar (J2ME) oder copied to the device, and than tell LoroDux where the Data is stored. Ideas? Sounds similar to how GPSmid works (http://wiki.openstreetmap.org/wiki/GpsMid). Maybe you should have a look into it. Cheers, -- Iván Sánchez Ortega i...@sanchezortega.es ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Wrong display of urls in tags
Hello, when you are on a page like: http://www.openstreetmap.org/browse/node/671785382 The http: is cut out of the value. In this example the tag wikipedia has a value of http://* but only //* is displayed. Best regards Jan S. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
On Wed, Apr 28, 2010 at 7:02 PM, Scott Crosby scrosb...@gmail.com wrote: Hello! I would like to announce code implementing a binary OSM format that supports the full semantics of the OSM XML. It is 5x-10x faster at reading and writing and 30-50% smaller; an entire planet, including all metadata, can be read in about 12 minutes and written in about 50 minutes on a 3 year old dual-core machine. I have implemented an osmosis reader and writer and have enhancements to the map splitter to read the format. Code is pure Java and uses Google protocol buffers for the low-level serialization. Thats very interesting. Would like to see this working with c++ as well. Will have to look at the code. thanks for sharing, mike ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Wrong display of urls in tags
On 28/04/10 17:43, Jan Sandbrink wrote: when you are on a page like: http://www.openstreetmap.org/browse/node/671785382 The http: is cut out of the value. In this example the tag wikipedia has a value of http://* but only //* is displayed. It was a bug - it's fixed now. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
On Wed, Apr 28, 2010 at 1:16 PM, OJ W ojwli...@googlemail.com wrote: where's the .proto file? Proto files should be in the osmosis git repository at: src/crosby/binary/fileformat.proto src/crosby/binary/osmformat.proto do you have data files in this format available to download? No, I do not have any files at this time as I am not ready to declare the file format as being stable. You can make your own test files with --write-bin. Scott ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
I have a quick question: does the format support inserting data to an existing file? Or is it just some binary serialization? The format is a binary serialization design and does not support random-access read and write semantics. For that a database is probably more suitable. However, some changes can be done to a file relatively cheaply. Data can be trivially appended. Rewriting a file could be fairly cheap as each fileblock is independently decodable and contains only 8,000 OSM entities. A fileblock can be copied from an input to an output without decompressing or parsing it. Metadata in the block header could be used to find out which fileblocks can be copied unchanged, or used to filter out unwanted blocks. Now we just need the dump tool for the database to create some planet dump file in your format. If osmosis is used as the dump tool, I believe a --write-bin should suffice to make a planet dump. The code just ties into the existing Source/Sink osmosis architecture. Scott ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
On 28/04/10 20:07, Scott Crosby wrote: Now we just need the dump tool for the database to create some planet dump file in your format. If osmosis is used as the dump tool, I believe a --write-bin should suffice to make a planet dump. The code just ties into the existing Source/Sink osmosis architecture. Osmosis isn't the dump tool, no. The planetdump program is. If we were going to offer a binary version for download the it would be better to generate it from the xml one anyway, rather than from the database, so that the two versions are consistent. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 28-04-10 21:50, Tom Hughes schreef: Osmosis isn't the dump tool, no. The planetdump program is. If we were going to offer a binary version for download the it would be better to generate it from the xml one anyway, rather than from the database, so that the two versions are consistent. Since the current XML version is inconsistent, a direct database dump will be more consistent than any conversion. Matt received a lot of examples for inconsistencies already. Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkvYlLAACgkQYH1+F2Rqwn3s7wCfb+I6iG7ieZEXXycojP9BJNgZ UdwAmgK06M/9tESSIxsypsV7XjtWpw2S =d5pO -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
On 28 April 2010 21:04, Stefan de Konink ste...@konink.de wrote: Since the current XML version is inconsistent, a direct database dump will be more consistent than any conversion. Matt received a lot of examples for inconsistencies already. Since: http://trac.openstreetmap.org/changeset/20396 ? / Grant ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Op 28-04-10 22:11, Grant Slater schreef: On 28 April 2010 21:04, Stefan de Konink ste...@konink.de wrote: Since the current XML version is inconsistent, a direct database dump will be more consistent than any conversion. Matt received a lot of examples for inconsistencies already. Since: http://trac.openstreetmap.org/changeset/20396 ? Did that fix *all* the older inconsistencies? I mean did you run a query to verify all referential constraints on the current table? Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEAREKAAYFAkvYmkcACgkQYH1+F2Rqwn2AxQCdHjZt9SI6gpKlqHOKwb7BXBYQ FHIAnicg9/c3qOTOgUjJiyT4HhPvG7nG =EJdz -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
I got it built,
Build instructions :
first get the protobuf, and build manually,
526 svn checkout http://protobuf.googlecode.com/svn/trunk/ protobuf-read-only
527 ls
528 cd protobuf-read-only/
529 ls
530 bash ./autogen.sh
531 ls
532 make
533 ./configure
534 make
535 sudo make install
go into the java dir and use ant to build the jar files.
got to the sources of the OSM tool :
install the protobuf into the lib/compile, normally this will be
handled by ivy, but there are issues.
cp
/home/mdupont/experiments/osm/protobuf-read-only/java/target/protobuf-java-2.3.1-pre.jar
lib/compile/
607 cd src/
611 cd crosby/
613 cd binary/
614 ls
615 protoc --java_out=../.. fileformat.proto
616 protoc --java_out=../.. osmformat.proto
this generates the needed code.
now you can build.
On Wed, Apr 28, 2010 at 7:02 PM, Scott Crosby scrosb...@gmail.com wrote:
Hello!
I would like to announce code implementing a binary OSM format that
supports the full semantics of the OSM XML. It is 5x-10x faster at
reading and writing and 30-50% smaller; an entire planet, including
all metadata, can be read in about 12 minutes and written in about 50
minutes on a 3 year old dual-core machine. I have implemented an
osmosis reader and writer and have enhancements to the map splitter to
read the format. Code is pure Java and uses Google protocol buffers
for the low-level serialization.
Comparing the file sizes:
8.2gb planet-100303.osm.bz2
12 gb planet-100303.osm.gz
5.2gb planet-omitmeta.bin
6.2gb planet.bin
The omitmeta version omits the uid/user/version/timestamp metadata
fields on each entity and are faster to generate and read.
The design is very extensible. The low-level file format is designed
to support random access at the 'fileblock' granularity, where a
fileblock can contain ~8k OSM entities. There is *no* tag hardcoding
used; all keys and values are stored in full as opaque strings. For
future scalability, 64 bit node/way/relation ID's are assumed. The
current serializer preserves the order of OSM entities and tags on OSM
entities. To flexibly handle multiple resolutions, the granularity, or
resolution used for representing locations and timestamps is
adjustable in multiples of 1 millisecond and 1 nanodegree and can be
set independently for each fileblock. The default scaling factor is
1000 milliseconds and 100 nanodegrees, corresponding to about ~1cm at
the equator. These are the current resolution of the OSM database.
Smaller files can be generated. At 10 microdegrees granularity,
corresponding to about 1m of resolution, the filesize decreases by
about 1gb. Space may also be saved by removing uninteresting UUID
tags or perhaps by having stronger geographic locality when building
the file.
I have also tested the binary format on some SRTM contour lines in OSM
0.5 XML format, obtaining about a 50:1 compression ratio. This might
be further improved by choosing a granularity equal to the isohypsis
grid size.
// Testing
I have tested this code on the Cloudmade extract of Rhode
Island. After converting the entire file to and from binary format,
the XML output is bytewise identical to original file except for the
one line indicating the osmosis version number.
When run through the splitter, the output is not bytewise identical to
before because of round-off errors 16 digits after the decimal point;
this could be fixed by having the splitter behave like osmosis and
only output 7 significant digits.
// To use:
Demonstration code is available on github at
http://github.com/scrosby/OSM-Osmosis and
http://github.com/scrosby/OSM-splitter
See the 'master' branches.
Please note that this is at present unpackaged demonstration code and
the fileformat may change to incorporate suggestions. Also note that
the shared code between the splitter and osmosis currently lives in
the osmosis git repository. You'll also need to go into the
crosby.binary directory and run the protocol compiler ('protoc') on
the .proto files (See comments in those files for the command line.).
/// The design ///
I use Google protocol buffers for the low-level store. Given a
specification file of one or more messages, the protocol buffer
compiler writes low-level serialization code. Messages may contain
other messages, forming hierarchical structures. Protocol buffers also
support extensibility; new fields can be added to a message and old
clients can read those messages without recompiling. For more details,
please see http://code.google.com/p/protobuf/. Google officially
supports C++, Java, and Python, but compilers exist for other
languages. An example message specification is:
message Node {
required sint64 id = 1;
required sint64 lat = 7;
required sint64 lon = 8;
repeated uint32 keys = 9 [packed = true]; // Denote strings
repeated uint32 vals = 10 [packed = true];// Denote strings
[OSM-dev] XAPI Server Problems?
Hi Folks, Does anyone know if there is a problem with the http://xapi.openstreetmap.org server? http://osmxapi.hypercube.telascience.org works ok, but the other just seems to hang once a connection is made. Unfortunately informationfreeway.org keeps redirecting to the openstreetmap one. Is it possible to change the informationfreeway service to redirect to a different server if this one has a problem please? Thanks Graham. -- Dr. Graham Jones Hartlepool, UK email: grahamjones...@gmail.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
I have created a branch with the cpp generated code, g...@github.com:h4ck3rm1k3/OSM-Osmosis.git the new c++ lib is called libosmprotobuf, what a great name. OSM-Osmosis/src/crosby/binary/ run make to generate the code, but i checked in the results. in the subdir : OSM-Osmosis/src/crosby/binary/cpp bash ./autogen.sh ./configure make it uses /usr/local/lib/libprotobuf.la which is a bit of a hack. my next step will be to hook this up to my existing c++ code https://code.launchpad.net/~jamesmikedupont/+junk/EPANatReg I hope that I will be able to make nice small C++ tools that can then process and or produce these buffer files. very interesting stuff that google has produced, thanks scott for making this public, mike ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New OSM binary fileformat implementation.
On 28 April 2010 21:27, Stefan de Konink ste...@konink.de wrote: Since the current XML version is inconsistent, a direct database dump will be more consistent than any conversion. Matt received a lot of examples for inconsistencies already. Since: http://trac.openstreetmap.org/changeset/20396 ? Did that fix *all* the older inconsistencies? I mean did you run a query to verify all referential constraints on the current table? Well if there are any inconsistencies; don't keep them secret. Lets get them fixed. :-) / Grant ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] XAPI Server Problems?
Graham The server has been heavily loaded for the last day or so. Requests are getting queue up. Increasing your client timeout might help, if you are able to do that. I'll turn on some load balancing so that some requests are shed to hypercube. Etienne On Wed, Apr 28, 2010 at 10:13 PM, Graham Jones grahamjones...@googlemail.com wrote: Hi Folks, Does anyone know if there is a problem with the http://xapi.openstreetmap.org server? http://osmxapi.hypercube.telascience.org works ok, but the other just seems to hang once a connection is made. Unfortunately informationfreeway.org keeps redirecting to the openstreetmap one. Is it possible to change the informationfreeway service to redirect to a different server if this one has a problem please? Thanks Graham. -- Dr. Graham Jones Hartlepool, UK email: grahamjones...@gmail.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
