Re: [OSM-dev] ODbL full history planet
On Thu, Oct 25, 2012 at 02:24:48PM +0200, Stephan Knauss wrote: Pascal Neis writes: Jochen Topf schrieb: What about redacted versions of nodes without coordinates? can you please give an example Node ID for that? I mean, did the bot really only removed lat/lon of a node and not the entire version no. of a node? for example here: http://www.openstreetmap.org/api/0.6/node/573785524/history entry does not contain lat/lon, also versions are missing. Was also discussed in this thread: http://lists.openstreetmap.org/pipermail/rebuild/2012-July/000299.html Can the software/osmium cope with that? what will end up in the pbf file? PBF files don't have a way of storing non-existing coordinates. When PBF was invented this couldn't happen and nobody ever thought of adding it. Osmium uses MAXINT (boost::integer_traitsint32_t::const_max) to mark invalid positions internally. This will show up as 214.7483647 externally if whatever code you have doesn't check the result of the defined() method. It will happily write those values to the PBF file, because it doesn't know anything better to do. So yes, Osmium will handle this case, it doesn't crash or so. But it might not handle it in the way you expect. And it can't handle it properly until somebody defines what properly is. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hello Jochen, thank you for this explanation. Explains what I'm seeing with an osh.pbf On 30.10.2012 10:46, Jochen Topf wrote: PBF files don't have a way of storing non-existing coordinates. When PBF was invented this couldn't happen and nobody ever thought of adding it. could specifying MAXINT as the value for non-exisiting lat/lon values in a full history pbf be a solution? I still have no stock tooling for reading either the xml file and grep for a node nor pbf tools to inspect how the version history of a redacted node looks like. So unfortunately can't simply try to see what's happening. Osmium uses MAXINT (boost::integer_traitsint32_t::const_max) to mark invalid positions internally. This will show up as 214.7483647 externally if whatever code you have doesn't check the result of the defined() method. I used your osmium_convert example to convert the osh.xml into osh.pbf. As you said, it does not include any checks for defined(), so it all ends up in PBF. I did not see anything special about lat. Both fields would be MAXINT, right? Maybe it's limited in osmconvert. Can't follow that code :( O:\osmconvert.exe --out-statistics history_2012-10-13_13_35.osh.pbf timestamp min: 2005-04-18T14:12:45Z timestamp max: 2012-10-13T11:35:31Z lon min: -,.),(-*,( lon max: 214.7483647 lat min: -90.000 lat max: 90.000 nodes: 2325354854 ways: 273053641 relations: 6210920 node id min: 1 node id max: 1962486877 way id min: 35 way id max: 185598826 relation id min: 2 relation id max: 2474614 keyval pairs max: 7838 keyval pairs max object: way 17441262 noderefs max: 49189 noderefs max object: way 29310085 relrefs max: 69643 relrefs max object: relation 82682 Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Am 25.10.2012 17:09, schrieb kimaidou: Hi Peter, I would like to know if you could build a small extract for me (or point me to a tutorial to do so myself) . I need to run a test for the europe/france/provence-alpes-cote-d-azur.poly contained in the geofabrik zip here : http://download.geofabrik.de/clipbounds/clipbounds.tgz Or the bbox if easier : europe/germany/rheinland-pfalz.osh.pbf BBOX 4.303,43.847,5.375,44.493 Here you go: http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/france/ Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
52M only ? Current Geofabrik PBF extract weights 187M http://download.geofabrik.de/openstreetmap/europe/france/ 2012/10/29 Peter Körner osm-li...@mazdermind.de Here you go: http://osm.personalwerk.de/**full-history-extracts/history_** 2012-10-13_13:35/europe/**france/http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/france/ Peter __**_ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.**org/listinfo/devhttp://lists.openstreetmap.org/listinfo/dev -- ab_fab http://wiki.openstreetmap.org/wiki/User:Ab_fab Il n'y a pas de pas perdus, Nadja ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi Peter 2012/10/29 Peter Körner osm-li...@mazdermind.de Here you go: http://osm.personalwerk.de/**full-history-extracts/history_** 2012-10-13_13:35/europe/**france/http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/france/ Thanks a lot ! Has you generated it via the given BBOX or via the Geofabrik polygon for the provence-alpes-cote-d-azur ? I will give it a try asap. Regards, Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
On Wed, Oct 24, 2012 at 05:51:49PM +0200, Peter Körner wrote: Am 23.10.2012 23:26, schrieb Pedro Larroy: Do you have a pointer to the pbf definition files? It's in the usual osm.pbf-File-Format: https://github.com/scrosby/OSM-binary The only difference is, that the visible-field is filled and there are more then one entity per id (one for each version). See: https://github.com/scrosby/OSM-binary/blob/master/src/osmformat.proto#L130 What about redacted versions of nodes without coordinates? Does that work properly? Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
On 23 October 2012 09:15, kimaidou kimai...@gmail.com wrote: Hi Paul, Thanks for pointing this out. I would say this file is the one you pointed : https://github.com/gravitystorm/openstreetmap-license-change/blob/master/check_history.rb It's actually a different file for loading history extracts into an apidb database: https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb There's a lot of stuff in there about license acceptance, which of course was more relevant to the original purpose of the script. Don't worry though, all the licensing stuff is only for calculating the agreed_time column in the users table - it doesn't affect the main output of nodes/ways/relations. As far as I'm aware, it's still the only code that can load .osh files into an apidb database. Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi, Jochen Topf schrieb: What about redacted versions of nodes without coordinates? can you please give an example Node ID for that? I mean, did the bot really only removed lat/lon of a node and not the entire version no. of a node? All the best, Pascal ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
On Thu, Oct 25, 2012 at 10:30:50AM +0200, Pascal Neis wrote: Jochen Topf schrieb: What about redacted versions of nodes without coordinates? can you please give an example Node ID for that? I mean, did the bot really only removed lat/lon of a node and not the entire version no. of a node? I don't have an example. I just heard months back that this would happen when the redaction bot went through. I have never looked at the data to see how things were done. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Pascal Neis writes: Jochen Topf schrieb: What about redacted versions of nodes without coordinates? can you please give an example Node ID for that? I mean, did the bot really only removed lat/lon of a node and not the entire version no. of a node? for example here: http://www.openstreetmap.org/api/0.6/node/573785524/history entry does not contain lat/lon, also versions are missing. Was also discussed in this thread: http://lists.openstreetmap.org/pipermail/rebuild/2012-July/000299.html Can the software/osmium cope with that? what will end up in the pbf file? Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi, Looking at this file https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb It seems if we do not use the --file, --users-agreed nor --changesets-agreed options, this will import the whole history file into an apidb. This looks promising ! I am not a ruby guy, so I will surely need some time to understand how to install and run this file, but I will give it a try. What about performance ? Does it handle a big history file ? Michael 2012/10/25 Andy Allan gravityst...@gmail.com On 23 October 2012 09:15, kimaidou kimai...@gmail.com wrote: Hi Paul, Thanks for pointing this out. I would say this file is the one you pointed : https://github.com/gravitystorm/openstreetmap-license-change/blob/master/check_history.rb It's actually a different file for loading history extracts into an apidb database: https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb There's a lot of stuff in there about license acceptance, which of course was more relevant to the original purpose of the script. Don't worry though, all the licensing stuff is only for calculating the agreed_time column in the users table - it doesn't affect the main output of nodes/ways/relations. As far as I'm aware, it's still the only code that can load .osh files into an apidb database. Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
On 25 October 2012 14:43, kimaidou kimai...@gmail.com wrote: I am not a ruby guy, so I will surely need some time to understand how to install and run this file, but I will give it a try. There is a readme, but in short use ruby 1.9 and run bundle install to install the dependencies. What about performance ? Does it handle a big history file ? It was used to load a whole-country extract (Ireland) before, but it's no where near as fast as other tools. I'd suggest creating a very small history extract first, and see how you get on. Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Am 25.10.2012 15:43, schrieb kimaidou: Hi, Looking at this file https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb It seems if we do not use the --file, --users-agreed nor --changesets-agreed options, this will import the whole history file into an apidb. This looks promising ! I am not a ruby guy, so I will surely need some time to understand how to install and run this file, but I will give it a try. What about performance ? Does it handle a big history file ? AFAIK it took multiple days to import Ireland (which indicates a year or so for the whole planet). Simon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Thanks for your answers. I will try with a smal extract first. 2012/10/25 Simon Poole si...@poole.ch Am 25.10.2012 15:43, schrieb kimaidou: Hi, Looking at this file https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb It seems if we do not use the --file, --users-agreed nor --changesets-agreed options, this will import the whole history file into an apidb. This looks promising ! I am not a ruby guy, so I will surely need some time to understand how to install and run this file, but I will give it a try. What about performance ? Does it handle a big history file ? AFAIK it took multiple days to import Ireland (which indicates a year or so for the whole planet). Simon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Me again : I managed to install ruby on ubuntu 12.04 with these commands : sudo apt-get install ruby1.9.1 libxslt-dev ruby1.9.1-dev sudo gem1.9.1 install text libxml-ruby mechanize algorithms # Then I downloaded the sources : cd /tmp/ wget https://github.com/gravitystorm/openstreetmap-license-change/zipball/master unzip gravitystorm-openstreetmap-license-change-2d01fab.zip cd /tmp/gravitystorm-openstreetmap-license-change-2d01fab cp example.auth.yaml auth.yaml Now I need a small history extract to perform tests. I have seen these places : * full planet : BIG file (too big for my small machine) : http://planet.openstreetmap.org/planet/full-history/ * Simon's extracts : available only unti redaction period, but there are some small files to try : http://odbl.poole.ch/extracts/ Am I missing some other URLs ? I will try asap with Andora : http://odbl.poole.ch/extracts/andorra.osh.bz2 ++ 2012/10/25 kimaidou kimai...@gmail.com Thanks for your answers. I will try with a smal extract first. 2012/10/25 Simon Poole si...@poole.ch Am 25.10.2012 15:43, schrieb kimaidou: Hi, Looking at this file https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb It seems if we do not use the --file, --users-agreed nor --changesets-agreed options, this will import the whole history file into an apidb. This looks promising ! I am not a ruby guy, so I will surely need some time to understand how to install and run this file, but I will give it a try. What about performance ? Does it handle a big history file ? AFAIK it took multiple days to import Ireland (which indicates a year or so for the whole planet). Simon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi, kimaidou schrieb: Now I need a small history extract to perform tests. I have seen these places : * full planet : BIG file (too big for my small machine) : http://planet.openstreetmap.org/planet/full-history/ * Simon's extracts : available only unti redaction period, but there are some small files to try : http://odbl.poole.ch/extracts/ Am I missing some other URLs ? You will find Peter's files here: http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/ and some small files here: http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/germany/ http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/germany/rheinland-pfalz/ All the best, Pascal ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Thanks a lot for the updated url. One small addition to the installation for ubuntu : I needed to add the pg gem package : sudo gem1.9.1 install text libxml-ruby mechanize algorithms pg Then download the andora.osh, untar it and run the importer via the command: ruby1.9.1 extract_loader.rb -f andorra.osh It took ~ 20 minutes for a 82Mo uncompressed file Now I will read some lines in the database to see the result, and report back here 2012/10/25 Pascal Neis pascal.n...@gmail.com Hi, You will find Peter's files here: http://osm.personalwerk.de/**full-history-extracts/history_** 2012-10-13_13:35/europe/http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/ and some small files here: http://osm.personalwerk.de/**full-history-extracts/history_** 2012-10-13_13:35/europe/**germany/http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/germany/ http://osm.personalwerk.de/**full-history-extracts/history_** 2012-10-13_13:35/europe/**germany/rheinland-pfalz/http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/europe/germany/rheinland-pfalz/ All the best, Pascal ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi Peter, I would like to know if you could build a small extract for me (or point me to a tutorial to do so myself) . I need to run a test for the europe/france/provence-alpes-cote-d-azur.poly contained in the geofabrik zip here : http://download.geofabrik.de/clipbounds/clipbounds.tgz Or the bbox if easier : europe/germany/rheinland-pfalz.osh.pbf BBOX 4.303,43.847,5.375,44.493 Thanks in advance Michael 2012/10/22 Peter Körner osm-li...@mazdermind.de As always, if you need another extract (and don't want/can't to generate it yourself), drop me a line. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi again peter. sorry for the noise, I just read your very complete tutorial here : https://github.com/MaZderMind/osm-history-splitter I think I would give it a try to split the France extract into a smaller extract. Michael 2012/10/25 kimaidou kimai...@gmail.com Hi Peter, I would like to know if you could build a small extract for me (or point me to a tutorial to do so myself) . I need to run a test for the europe/france/provence-alpes-cote-d-azur.poly contained in the geofabrik zip here : http://download.geofabrik.de/clipbounds/clipbounds.tgz Or the bbox if easier : europe/germany/rheinland-pfalz.osh.pbf BBOX4.303,43.847,5.375,44.493 Thanks in advance Michael 2012/10/22 Peter Körner osm-li...@mazdermind.de As always, if you need another extract (and don't want/can't to generate it yourself), drop me a line. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Am 23.10.2012 10:13, schrieb kimaidou: Could you give the github repository url for the osm2pgsql-style table tool ? https://github.com/MaZderMind/osm-history-renderer To start I'd recommend following the Tutorial: https://github.com/MaZderMind/osm-history-renderer/blob/master/TUTORIAL.md I'd say just take the osmium lib and the postgres lib and write yourself an importer (and put it on github). This is what I would do I think. I just need time to get used to osmium... I usually code in python, not C++ ... And obviously to find enough time. I will sure put it on github if I managed to developp this tool Sound's nice. Doing the processing with C++ gains you a lt of speed over python. (a lot as in: 1 hour for a planet instead of 1 day) Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi Am 23.10.2012 23:26, schrieb Pedro Larroy: Do you have a pointer to the pbf definition files? It's in the usual osm.pbf-File-Format: https://github.com/scrosby/OSM-binary The only difference is, that the visible-field is filled and there are more then one entity per id (one for each version). See: https://github.com/scrosby/OSM-binary/blob/master/src/osmformat.proto#L130 Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi, 2012/10/22 Peter Körner osm-li...@mazdermind.de Am 22.10.2012 11:48, schrieb kimaidou: I would like to knwo if someone had already coded a loader for the history dumps into an api-db ? As far as I know there's no such thing. I have a loader into an osm2ogsql-style table on github, although an apidb-loader would be much easier to implement -- but as I never needed one.. Could you give the github repository url for the osm2pgsql-style table tool ? I'd say just take the osmium lib and the postgres lib and write yourself an importer (and put it on github). This is what I would do I think. I just need time to get used to osmium... I usually code in python, not C++ ... And obviously to find enough time. I will sure put it on github if I managed to developp this tool Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi Paul, Thanks for pointing this out. I would say this file is the one you pointed : https://github.com/gravitystorm/openstreetmap-license-change/blob/master/check_history.rb Hey, another language, it is ruby ;) I will have a deep look to understand the basics here... Thanks again Michael 2012/10/22 Paul Norman penor...@mac.com I’m dashing off to class, but I believe there’s one somewhere in the redaction bot codebase. ** ** No idea exactly where or how to run it, but hopefully that can get you started. ** ** *From:* kimaidou [mailto:kimai...@gmail.com] *Sent:* Monday, October 22, 2012 2:49 AM *To:* Peter Körner *Cc:* Martijn van Exel; Pascal Neis; dev@openstreetmap.org *Subject:* Re: [OSM-dev] ODbL full history planet ** ** Hi all , As I told a long time ago, I am very interested in these extracts, as they would help me a lot to load history data for LizWatch : http://demo.3liz.com/lizwatch/index.php/view/map?bbox=4.796043269302758%2C44.13073779318937%2C4.824303024561808%2C44.140039449425494number=500zoom=16from=2011-10-22+00%3A00to=2012-10-23+00%3A00xapi=amenity To perform this map of changes, I use the api-db osmosis database, populated for normal dumps, and some api queries to try to retrieve some historical changesets data. Dealing with deletes is not that simple, because osmosis creates fake deletion to keep a bounding box. Before, I could use the lon/lat of deleted objects to see if there were out of the bbox (so deleted by osmosis, nor a regular user), but now the api does not provide long and latitude for deleted objects. I could download the latest version to get the lon/lat, but it becomes heavier and heavier. I would like to knwo if someone had already coded a loader for the history dumps into an api-db ? I heard about a plugin for osmosis, or other scripts, but it seems these tools are alpha. Should I redevelop a whole python script to read the history and populate a database, or has someone already something ? Thanks for advance, and apologies if I am asking for too much :) (and sorry for my average english...) Michael 2012/10/22 Peter Körner osm-li...@mazdermind.de Hi Am 21.10.2012 20:42, schrieb Martijn van Exel: ** ** Also, I was going to make US state and county cuts of this file and possibly make them available on our OSM US server - Peter, my osm-history-splitter is a few months old, any reason to pull and recompile? Just get a fresh osmium and splitter just to be on the safe side. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ** ** ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Thanks. Do you have a pointer to the pbf definition files? Pedro. On Mon, Oct 22, 2012 at 10:55 AM, Peter Körner osm-li...@mazdermind.de wrote: Am 22.10.2012 00:43, schrieb Stephan Knauss: On 19.10.2012 16:53, Matt Amos wrote: now available from: http://planet.openstreetmap.org/planet/full-history/ only limited testing has been done on this file (checking for XML well-formed-ness and various spot-checks on elements), so please let me know if you find any errors with it. I have not yet downloaded it, hoping Peter will make the pbf conversion available which is a lot faster to process and saving my time to do the conversion by myself. Here we go: http://osm.personalwerk.de/full-experimental/ And the generated extracts are here: http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/ As always, if you need another extract (and don't want/can't to generate it yourself), drop me a line. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Am 22.10.2012 00:43, schrieb Stephan Knauss: On 19.10.2012 16:53, Matt Amos wrote: now available from: http://planet.openstreetmap.org/planet/full-history/ only limited testing has been done on this file (checking for XML well-formed-ness and various spot-checks on elements), so please let me know if you find any errors with it. I have not yet downloaded it, hoping Peter will make the pbf conversion available which is a lot faster to process and saving my time to do the conversion by myself. Here we go: http://osm.personalwerk.de/full-experimental/ And the generated extracts are here: http://osm.personalwerk.de/full-history-extracts/history_2012-10-13_13:35/ As always, if you need another extract (and don't want/can't to generate it yourself), drop me a line. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi Am 21.10.2012 20:42, schrieb Martijn van Exel: Also, I was going to make US state and county cuts of this file and possibly make them available on our OSM US server - Peter, my osm-history-splitter is a few months old, any reason to pull and recompile? Just get a fresh osmium and splitter just to be on the safe side. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi all , As I told a long time ago, I am very interested in these extracts, as they would help me a lot to load history data for LizWatch : http://demo.3liz.com/lizwatch/index.php/view/map?bbox=4.796043269302758%2C44.13073779318937%2C4.824303024561808%2C44.140039449425494number=500zoom=16from=2011-10-22+00%3A00to=2012-10-23+00%3A00xapi=amenity To perform this map of changes, I use the api-db osmosis database, populated for normal dumps, and some api queries to try to retrieve some historical changesets data. Dealing with deletes is not that simple, because osmosis creates fake deletion to keep a bounding box. Before, I could use the lon/lat of deleted objects to see if there were out of the bbox (so deleted by osmosis, nor a regular user), but now the api does not provide long and latitude for deleted objects. I could download the latest version to get the lon/lat, but it becomes heavier and heavier. I would like to knwo if someone had already coded a loader for the history dumps into an api-db ? I heard about a plugin for osmosis, or other scripts, but it seems these tools are alpha. Should I redevelop a whole python script to read the history and populate a database, or has someone already something ? Thanks for advance, and apologies if I am asking for too much :) (and sorry for my average english...) Michael 2012/10/22 Peter Körner osm-li...@mazdermind.de Hi Am 21.10.2012 20:42, schrieb Martijn van Exel: Also, I was going to make US state and county cuts of this file and possibly make them available on our OSM US server - Peter, my osm-history-splitter is a few months old, any reason to pull and recompile? Just get a fresh osmium and splitter just to be on the safe side. Peter __**_ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.**org/listinfo/devhttp://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Im dashing off to class, but I believe theres one somewhere in the redaction bot codebase. No idea exactly where or how to run it, but hopefully that can get you started. From: kimaidou [mailto:kimai...@gmail.com] Sent: Monday, October 22, 2012 2:49 AM To: Peter Körner Cc: Martijn van Exel; Pascal Neis; dev@openstreetmap.org Subject: Re: [OSM-dev] ODbL full history planet Hi all , As I told a long time ago, I am very interested in these extracts, as they would help me a lot to load history data for LizWatch : http://demo.3liz.com/lizwatch/index.php/view/map?bbox=4.796043269302758%2C44 .13073779318937%2C4.824303024561808%2C44.140039449425494 http://demo.3liz.com/lizwatch/index.php/view/map?bbox=4.796043269302758%2C4 4.13073779318937%2C4.824303024561808%2C44.140039449425494number=500zoom=16 from=2011-10-22+00%3A00to=2012-10-23+00%3A00xapi=amenity number=500zoom=16from=2011-10-22+00%3A00to=2012-10-23+00%3A00xapi=ameni ty To perform this map of changes, I use the api-db osmosis database, populated for normal dumps, and some api queries to try to retrieve some historical changesets data. Dealing with deletes is not that simple, because osmosis creates fake deletion to keep a bounding box. Before, I could use the lon/lat of deleted objects to see if there were out of the bbox (so deleted by osmosis, nor a regular user), but now the api does not provide long and latitude for deleted objects. I could download the latest version to get the lon/lat, but it becomes heavier and heavier. I would like to knwo if someone had already coded a loader for the history dumps into an api-db ? I heard about a plugin for osmosis, or other scripts, but it seems these tools are alpha. Should I redevelop a whole python script to read the history and populate a database, or has someone already something ? Thanks for advance, and apologies if I am asking for too much :) (and sorry for my average english...) Michael 2012/10/22 Peter Körner osm-li...@mazdermind.de Hi Am 21.10.2012 20:42, schrieb Martijn van Exel: Also, I was going to make US state and county cuts of this file and possibly make them available on our OSM US server - Peter, my osm-history-splitter is a few months old, any reason to pull and recompile? Just get a fresh osmium and splitter just to be on the safe side. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Hi Matt, Matt Amos schrieb: http://planet.openstreetmap.org/planet/full-history/ only limited testing has been done on this file (checking for XML well-formed-ness and various spot-checks on elements), so please let me know if you find any errors with it. thx, worked for me. Read Nodes : 1861090208 Read Ways : 170709587 Read Relations : 2251687 Cheers, Pascal ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Cool! I want to use it to figure out how many active contributors we currently have in the US, I'm sure there is already a tried and tested approach for that? Also, I was going to make US state and county cuts of this file and possibly make them available on our OSM US server - Peter, my osm-history-splitter is a few months old, any reason to pull and recompile? Best Martijn On Sun, Oct 21, 2012 at 12:21 PM, Pascal Neis pascal.n...@gmail.com wrote: Hi Matt, Matt Amos schrieb: http://planet.openstreetmap.**org/planet/full-history/http://planet.openstreetmap.org/planet/full-history/ only limited testing has been done on this file (checking for XML well-formed-ness and various spot-checks on elements), so please let me know if you find any errors with it. thx, worked for me. Read Nodes : 1861090208 Read Ways : 170709587 Read Relations : 2251687 Cheers, Pascal __**_ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.**org/listinfo/devhttp://lists.openstreetmap.org/listinfo/dev -- martijn van exel http://oegeo.wordpress.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
On 19.10.2012 16:53, Matt Amos wrote: now available from: http://planet.openstreetmap.org/planet/full-history/ only limited testing has been done on this file (checking for XML well-formed-ness and various spot-checks on elements), so please let me know if you find any errors with it. I have not yet downloaded it, hoping Peter will make the pbf conversion available which is a lot faster to process and saving my time to do the conversion by myself. Can you give a rough outline on how the content changed compared to the pre-redaction full history planet? In what way will be redacted entries presented? I expect the representation to be similar to the /history API call. Is that true? any other gotchas? Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] ODbL full history planet
now available from: http://planet.openstreetmap.org/planet/full-history/ only limited testing has been done on this file (checking for XML well-formed-ness and various spot-checks on elements), so please let me know if you find any errors with it. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ODbL full history planet
Am 19.10.2012 16:53, schrieb Matt Amos: now available from: http://planet.openstreetmap.org/planet/full-history/ only limited testing has been done on this file (checking for XML well-formed-ness and various spot-checks on elements), so please let me know if you find any errors with it. Thanks for this, I just kicked on the pbf-converter-and-split-process and on Monday we'll see if it worked out. Will report back, then. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
