Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
On Tue, Jan 6, 2009 at 10:34 PM, marcus.wolsc...@googlemail.com wrote: On Tue, 6 Jan 2009 17:26:18 +, Thomas Wood grand.edgemas...@gmail.com wrote: 2009/1/6 Oliver Ford o...@oliford.co.uk: marcus.wolsc...@googlemail.com wrote: I added some clarifications to the wiki-page on the Osmbin-format. (What it is intended for, what the status is and that it is not related to this osm mobile binary protocoll.) http://wiki.openstreetmap.org/wiki/OSMbin(file_format)http://wiki.openstreetmap.org/wiki/OSMbin%28file_format%29 Binary data formats has been thrashed over several times, often with different outcomes (as the number of proposed formats/protocols etc show). However, I'm pretty sure that your code would be welcomed by many in svn :) I've talked with Brett yesterday about integrating support for OSMbin into osmosis. I'll write that code in the next weeks. One day at a time. ;) Marcus Heh, I was thinking along those lines, too, but I thought I'd wait until your format stabilizes. It would be quite handy to have a random-access data source for Osmosis without requiring a database installation. Karl ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
Karl Newman wrote: Heh, I was thinking along those lines, too, but I thought I'd wait until your format stabilizes. It would be quite handy to have a random-access data source for Osmosis without requiring a database installation. I'm not sure if you realise this but there already is one of those. I can't reach the wiki at the moment but the tasks in question are: --write-customdb --read-customdb I forget the details of the on-disk format but it's built around the osmosis store classes so uses osmosis specific object serialisation. The --read-customdb task doesn't actually stream data to downstream tasks, it provides the data in the form of a dataset object. This can be consumed by tasks such as --dataset-dump which will stream from the dataset, but it can also be consumed by any task wishing to access data randomly. Performance isn't great for large datasets which is why I created the pgsql-simple schema. I've never been able to get an on-disk format to scale nearly as well as a real database. I also wrote a set of tasks based on one the Berkeley DB Java Edition but deleted it because performance was even worse. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
Brett Henderson wrote: Karl Newman wrote: Heh, I was thinking along those lines, too, but I thought I'd wait until your format stabilizes. It would be quite handy to have a random-access data source for Osmosis without requiring a database installation. I'm not sure if you realise this but there already is one of those. I can't reach the wiki at the moment but the tasks in question are: --write-customdb --read-customdb Hmm, perhaps that's because I never documented it on the wiki. I was never very happy with it and didn't circulate it. But it's there anyway. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
On Wed, Jan 7, 2009 at 2:28 PM, Brett Henderson br...@bretth.com wrote: Karl Newman wrote: Heh, I was thinking along those lines, too, but I thought I'd wait until your format stabilizes. It would be quite handy to have a random-access data source for Osmosis without requiring a database installation. I'm not sure if you realise this but there already is one of those. I can't reach the wiki at the moment but the tasks in question are: --write-customdb --read-customdb I forget the details of the on-disk format but it's built around the osmosis store classes so uses osmosis specific object serialisation. The --read-customdb task doesn't actually stream data to downstream tasks, it provides the data in the form of a dataset object. This can be consumed by tasks such as --dataset-dump which will stream from the dataset, but it can also be consumed by any task wishing to access data randomly. Performance isn't great for large datasets which is why I created the pgsql-simple schema. I've never been able to get an on-disk format to scale nearly as well as a real database. I also wrote a set of tasks based on one the Berkeley DB Java Edition but deleted it because performance was even worse. Brett Actually yes, I was aware of that one, but I recall that you said the performance was not good and I thought you had removed that one for performance reasons (I didn't pay close attention; I guess it was the BDB that you removed instead). Karl ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
Karl Newman wrote: On Wed, Jan 7, 2009 at 2:28 PM, Brett Henderson br...@bretth.com mailto:br...@bretth.com wrote: Karl Newman wrote: Heh, I was thinking along those lines, too, but I thought I'd wait until your format stabilizes. It would be quite handy to have a random-access data source for Osmosis without requiring a database installation. I'm not sure if you realise this but there already is one of those. I can't reach the wiki at the moment but the tasks in question are: --write-customdb --read-customdb I forget the details of the on-disk format but it's built around the osmosis store classes so uses osmosis specific object serialisation. The --read-customdb task doesn't actually stream data to downstream tasks, it provides the data in the form of a dataset object. This can be consumed by tasks such as --dataset-dump which will stream from the dataset, but it can also be consumed by any task wishing to access data randomly. Performance isn't great for large datasets which is why I created the pgsql-simple schema. I've never been able to get an on-disk format to scale nearly as well as a real database. I also wrote a set of tasks based on one the Berkeley DB Java Edition but deleted it because performance was even worse. Brett Actually yes, I was aware of that one, but I recall that you said the performance was not good and I thought you had removed that one for performance reasons (I didn't pay close attention; I guess it was the BDB that you removed instead). I forget now :-) I might have been going to remove it and changed my mind. If new features come along that are difficult to support I'll drop it. I don't think anybody uses it. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
marcus.wolsc...@googlemail.com wrote: I added some clarifications to the wiki-page on the Osmbin-format. (What it is intended for, what the status is and that it is not related to this osm mobile binary protocoll.) http://wiki.openstreetmap.org/wiki/OSMbin(file_format) Ok, that helps. You probably ought to say whether it relates to the 'OSM Binary Format' (http://wiki.openstreetmap.org/wiki/OSM_Binary_Format) I've also just noticed the wiki distinguishes between 'OSM_Mobile_Binary_Protocol' and 'OSM_Mobile_Binary_Protocol'. So now we have: 1) http://wiki.openstreetmap.org/wiki/OSM_Binary_Format 2) http://wiki.openstreetmap.org/wiki/OSM_Mobile_Binary_Format 3) http://wiki.openstreetmap.org/wiki/OSM_Mobile_Binary_Protocol 4) http://wiki.openstreetmap.org/wiki/OSMbin(file_format) Is it just me or would some general clarification/coordination be useful here? Oli ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
On Tue, 06 Jan 2009 11:16:32 +, Oliver Ford o...@oliford.co.uk wrote: marcus.wolsc...@googlemail.com wrote: I added some clarifications to the wiki-page on the Osmbin-format. (What it is intended for, what the status is and that it is not related to this osm mobile binary protocoll.) http://wiki.openstreetmap.org/wiki/OSMbin(file_format) Ok, that helps. You probably ought to say whether it relates to the 'OSM Binary Format' (http://wiki.openstreetmap.org/wiki/OSM_Binary_Format) Done. Thanks. I've also just noticed the wiki distinguishes between 'OSM_Mobile_Binary_Protocol' and 'OSM_Mobile_Binary_Protocol'. What IS the difference? So now we have: 1) http://wiki.openstreetmap.org/wiki/OSM_Binary_Format 2) http://wiki.openstreetmap.org/wiki/OSM_Mobile_Binary_Format 3) http://wiki.openstreetmap.org/wiki/OSM_Mobile_Binary_Protocol 4) http://wiki.openstreetmap.org/wiki/OSMbin(file_format) Is it just me or would some general clarification/coordination be useful here? I guess so. We have http://wiki.openstreetmap.org/wiki/Category:DataFormats and I guess we can group the different formats somewhat. Marcus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
On Tue, 06 Jan 2009 11:23:24 +, Oliver Ford o...@oliford.co.uk wrote: Oliver Ford wrote: I've also just noticed the wiki distinguishes between 'OSM_Mobile_Binary_Protocol' and 'OSM_Mobile_Binary_Protocol'. err, that should be between 'OSM_Mobile_Binary_Format' and 'OSM_Mobile_Binary_Protocol' . Well, the page states: Open Street Map Binary Protocol: This page is about additions to permit a particular way of using the OSM Binary Format for mobile devices in ... so there is a difference, it is clearly stated and the length of the page suggests that splitting it into 2 pages is perfectly fine. Marcus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
Oliver Ford wrote: I've also just noticed the wiki distinguishes between 'OSM_Mobile_Binary_Protocol' and 'OSM_Mobile_Binary_Protocol'. err, that should be between 'OSM_Mobile_Binary_Format' and 'OSM_Mobile_Binary_Protocol' . Oliver ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
2009/1/6 Oliver Ford o...@oliford.co.uk: marcus.wolsc...@googlemail.com wrote: I added some clarifications to the wiki-page on the Osmbin-format. (What it is intended for, what the status is and that it is not related to this osm mobile binary protocoll.) http://wiki.openstreetmap.org/wiki/OSMbin(file_format) Ok, that helps. You probably ought to say whether it relates to the 'OSM Binary Format' (http://wiki.openstreetmap.org/wiki/OSM_Binary_Format) I've also just noticed the wiki distinguishes between 'OSM_Mobile_Binary_Protocol' and 'OSM_Mobile_Binary_Protocol'. So now we have: 1) http://wiki.openstreetmap.org/wiki/OSM_Binary_Format 2) http://wiki.openstreetmap.org/wiki/OSM_Mobile_Binary_Format 3) http://wiki.openstreetmap.org/wiki/OSM_Mobile_Binary_Protocol 4) http://wiki.openstreetmap.org/wiki/OSMbin(file_format) Is it just me or would some general clarification/coordination be useful here? Oli Binary data formats has been thrashed over several times, often with different outcomes (as the number of proposed formats/protocols etc show). However, I'm pretty sure that your code would be welcomed by many in svn :) Of course, there's also probably a myriad of other open data formats for this sort of thing also -- Regards, Thomas Wood (Edgemaster) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
On Tue, 6 Jan 2009 17:26:18 +, Thomas Wood grand.edgemas...@gmail.com wrote: 2009/1/6 Oliver Ford o...@oliford.co.uk: marcus.wolsc...@googlemail.com wrote: I added some clarifications to the wiki-page on the Osmbin-format. (What it is intended for, what the status is and that it is not related to this osm mobile binary protocoll.) http://wiki.openstreetmap.org/wiki/OSMbin(file_format) Binary data formats has been thrashed over several times, often with different outcomes (as the number of proposed formats/protocols etc show). However, I'm pretty sure that your code would be welcomed by many in svn :) I've talked with Brett yesterday about integrating support for OSMbin into osmosis. I'll write that code in the next weeks. One day at a time. ;) Marcus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
Hi everyone, I started looking at/using OSM a few weeks ago with the aim of getting the UK map (~7M nodes, 900K ways) on my new iPAQ which I've almost finished porting linux to. Firstly, I'd like to say the whole OSM project is pure genius and am very thankful for the work every last developer/contributor has done. I'd certainly like to give something back by helping out. I spent most of the time looking at a few PDA apps that are already out there and trying to convert the UK planet extract to the required format for each app. I wrote up what I found out as an e-mail but it was a bit long so I've put it on my user page ( http://wiki.openstreetmap.org/wiki/User:Oliford ) as a sort of (subjective) review for anyone who's interested in PDA OSM use or interested in what working with the OSM data is like for a newbie. The short(er) story... Getting anything to handle the whole UK data without running out of memory, seg-faulting or falling over on invalid UTF-8 is hard work. In the end I used roadmap which converts from the OSM Mobile Binary (OSMMB) format and I've written my own C program which converts the whole UK XML extract to OSMMB in about a minute. I'm considering writing my own PDA app (open source of course) which would work directly from the OSMMB tiled vector data, give a slippy/scrollable and zoomable interface like Roadmap's but also provide the place/street name lookup and routing on all loaded tile data. The newer proposal on the OSM wiki for the OSMMB seems to be well suited to this. Currently the whole of the UK ends up in a 100MB OSSMB file which is really quick to parse. I can slice it into tiles, linearly search it for street names etc in a few seconds on a machine it takes several minutes to do the same on the XML. I have some thoughts about things like merging multiple tiles' data and routing across different tiles but there are still some things I don't quite understand about the format (the wiki pages are a bit confusing on what is actually decided and what are just ideas). Is anyone actively working on the OSMMB format now that I could direct my questions at? Thanks, Oliver Ford PS: Just as I'm signing this off I've stumbled over the 'OSMbin(file_format)' wiki page. Is that format related to the OSM Mobile binary somehow? Is one more developed/used than the other, has one been abandoned? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
2009/1/6, Oliver Ford o...@oliford.co.uk: PS: Just as I'm signing this off I've stumbled over the 'OSMbin(file_format)' wiki page. Is that format related to the OSM Mobile binary somehow? Is one more developed/used than the other, has one been abandoned? It is not related to OSM-MB at all. It is another binary-format I am currently specifying version 1.0 of that can be used with the next API-version (0.6). It is optimized for different scenarios then osmmb. I am currently debugging it's reference-implementation. As stated on the wiki-page of it, it is not ready yet but making good progress. (95% done). Marcus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'My first OSM experiences' - PDA Mapping software and the OSM mobile binary protocol
On Tue, 6 Jan 2009 06:44:58 +0100, Marcus Wolschon mar...@wolschon.biz wrote: 2009/1/6, Oliver Ford o...@oliford.co.uk: PS: Just as I'm signing this off I've stumbled over the 'OSMbin(file_format)' wiki page. Is that format related to the OSM Mobile binary somehow? Is one more developed/used than the other, has one been abandoned? It is not related to OSM-MB at all. It is another binary-format I am currently specifying version 1.0 of that can be used with the next API-version (0.6). It is optimized for different scenarios then osmmb. I am currently debugging it's reference-implementation. As stated on the wiki-page of it, it is not ready yet but making good progress. (95% done). I added some clarifications to the wiki-page on the Osmbin-format. (What it is intended for, what the status is and that it is not related to this osm mobile binary protocoll.) http://wiki.openstreetmap.org/wiki/OSMbin(file_format) Marcus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
