Re: [OSM-dev] Exhaustion of 32bit signed integer range expected this year
On Wed, 11 Jul 2012 22:32:30 -0700 (PDT) Kai Krueger kakrue...@gmail.com wrote: SimonPoole wrote It seems as if it would really make sense to make the 64bit ID version of osm2pgsql the default now and communicate that it might be a good idea to switch on the upcoming reload. So I'll bring this topic up again: Should the default of osm2pgsql be changed to the 64 bit ID version now? absolutely!!! ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Exhaustion of 32bit signed integer range expected this year
SimonPoole wrote RSN a large number of sites using OSM data will be reloading their databases, due to a certain well known change :-). Well, this time has come closer now... SimonPoole wrote It seems as if it would really make sense to make the 64bit ID version of osm2pgsql the default now and communicate that it might be a good idea to switch on the upcoming reload. So I'll bring this topic up again: Should the default of osm2pgsql be changed to the 64 bit ID version now? -- View this message in context: http://gis.19327.n5.nabble.com/Exhaustion-of-32bit-signed-integer-range-expected-this-year-tp5638421p5716169.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Exhaustion of 32bit signed integer range expected this year
Hi, On 04/13/2012 04:42 PM, Simon Poole wrote: It seems as if it would really make sense to make the 64bit ID version of osm2pgsql the default now and communicate that it might be a good idea to switch on the upcoming reload. Do we have any idea what the actual impact on DB size etc is? I have just run a series of osm2pgsql benchmarks (on a system equipped with SSDs and 32 GB RAM, PostgreSQL 9.1). A standard osm2pgsql import with --slim looked like this: 32bit IDs 64bit IDs time for planet load 38956s 45955s bytes required on disk 276594360 284217608 So you're roughly looking at at a 15% penalty in speed and only about 2% in disk space. (I haven't benchmarked updates or rendering yet.) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Exhaustion of 32bit signed integer range expected this year
Hi, this is just a reminder that our current highest node ID is about 1.7 billion, and that it grows by about 0.05 billion every month. This means that it is likely that by the end of 2012, we will have reached (or be very close to reaching) the end of the 32bit signed integer range (2.15 billion, or 2^31-1). See also: http://tools.geofabrik.de/munin/osm_nodes-year.png Software processing OSM data will need to either use unsigned integers (which can be problematic in cases where negative values are also required), or switch to 64bit integers altogether. This is also relevant when dealing with database tables; depending on what schema you are using, you might have 32bit signed IDs there as well. osm2pgsql can be compiled with 64bit ID support but that is not enabled in the standard binary distributions. If you are in any way involved with writing software for OSM, it may be worth thinking about that now. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Exhaustion of 32bit signed integer range expected this year
Frederik Ramm frede...@remote.org wrote: Software processing OSM data will need to either use unsigned integers (which can be problematic in cases where negative values are also required), or switch to 64bit integers altogether. Or used for other purposes like in osm2pgsql. Will the upcoming licence change add a delay to this problem? Sven -- We just typed make (Stephen Lambrigh, Director of Server Product Marketing at Informix about porting their Database to Linux) /me is giggls@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Exhaustion of 32bit signed integer range expected this year
Hi, On 04/13/2012 04:30 PM, Sven Geggus wrote: Or used for other purposes like in osm2pgsql. Will the upcoming licence change add a delay to this problem? No; if anything, node IDs will be used up faster because if the something is deleted by the bot and later re-created with a new ID then that would count extra. However, even if *all* problematic nodes were deleted and re-created, that would be less edit activity than we normally see in one month, so it wouldn't have a big impact on the outcome. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Exhaustion of 32bit signed integer range expected this year
RSN a large number of sites using OSM data will be reloading their databases, due to a certain well known change :-). It seems as if it would really make sense to make the 64bit ID version of osm2pgsql the default now and communicate that it might be a good idea to switch on the upcoming reload. Do we have any idea what the actual impact on DB size etc is? Simon PS: CC'd to the CWG for good form. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Exhaustion of 32bit signed integer range expected this year
On 13/04/2012 15:15, Frederik Ramm wrote: This means that it is likely that by the end of 2012, we will have reached (or be very close to reaching) the end of the 32bit signed integer range (2.15 billion, or 2^31-1). Thanks for the reminder. I was a bit worried about this having coded a lot of stuff in PHP. I now realise that so long as it is running on 64-bit hardware, PHP uses 64-bit ints, But is is an issue for people running PHP utilities on 32-bit hardware, who would have to either move or treat the IDs as strings. David ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Exhaustion of 32bit signed integer range expected this year
The latest Windows build of osm2pgsql is just over two years old now. Perhaps we will reach a new build by the end of this year. -Jukka Rahkonen- Frederik Ramm wrote: Hi, this is just a reminder that our current highest node ID is about 1.7 billion, and that it grows by about 0.05 billion every month. This means that it is likely that by the end of 2012, we will have reached (or be very close to reaching) the end of the 32bit signed integer range (2.15 billion, or 2^31-1). See also: http://tools.geofabrik.de/munin/osm_nodes-year.png Software processing OSM data will need to either use unsigned integers (which can be problematic in cases where negative values are also required), or switch to 64bit integers altogether. This is also relevant when dealing with database tables; depending on what schema you are using, you might have 32bit signed IDs there as well. osm2pgsql can be compiled with 64bit ID support but that is not enabled in the standard binary distributions. If you are in any way involved with writing software for OSM, it may be worth thinking about that now. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Exhaustion of 32bit signed integer range expected this year
On Fri, 13 Apr 2012, David Earl wrote: On 13/04/2012 15:15, Frederik Ramm wrote: This means that it is likely that by the end of 2012, we will have reached (or be very close to reaching) the end of the 32bit signed integer range (2.15 billion, or 2^31-1). Thanks for the reminder. I was a bit worried about this having coded a lot of stuff in PHP. I now realise that so long as it is running on 64-bit hardware, PHP uses 64-bit ints, But is is an issue for people running PHP utilities on 32-bit hardware, who would have to either move or treat the IDs as strings. Unless you're using PHP on Windows, where even in a 64-bit build on 64-bit hardware, the integer size is only a signed 32-bit integer. cheers, Derick -- http://derickrethans.nl | http://xdebug.org Like Xdebug? Consider a donation: http://xdebug.org/donate.php twitter: @derickr and @xdebug ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
