Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Andrew M. Bishop a...@gedanken.demon.co.uk wrote: If using the standard toolchain of osm2psql, postgresql and mapnik what is the minimum software versions that are needed to continue creating maps after ids reach 2^31-1? osm2pgsql supports 64bit id space already for a long time. But since this is a build-time option the default has been changed to 64bit id space not that long ago )see svn log for exact date). As far as mapnik is concerned 2.1 did already work well using osm2psql in 64 bit id mode. Sven -- Whenever there is a conflict between human rights and property rights, human rights must prevail. (Abraham Lincoln) /me is giggls@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
On 08.02.2013 06:59, Ilya Zverev wrote: Though I fail to understand what would be broken if you continue to use 32-bit osm2pgsql. Indices, maybe. don't be confused by the term 32-bit. It is required that the software can handle 64-bit data types for processing the node id. In postgresql it will be stored in a column declared as BIGINT. Data types often are signed. So one of the 32 bits is used to store the sign. The remaining 31 bits to store the ID are exhausted now. Software can be compiled as a 32-bit or 64-bit executable. If you can, you should use a 64-bit executable as this can access over 4GB of memory, giving you the chance to speed up the import a lot. What happens when you continue to use osm2pgsql with 32 bit ids (int4) instead of int8? Have a look at osmtypes.h. You'll see that osmid_t is declared as int32_t which is signed: typedef signed long int int32_t So the value will overflow. In the best case osm2pgsql will abort with an error. In the worst case your database content will be corrupted. Your database structure still is only int4, again signed. So it all needs to be converted into int8. Probably easiest is to re-import into a fresh DB with 64-bit IDs. Most people have done this step during the re-import after license change. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
A while ago I announced that I have newly compiled windows versions of osm2pgsql (Intel C++ Compiler XE 13.0, MS Visual Studio 2010). After my last build someone added the no_fork option (thanks!) which made my life easier. Compared to the master version, I basically only needed to remove the flat-nodes option due to file-sync issues. I guess it could be made to work, but I didn't want to invest the time. Remember that windows has no fork, therefore this binary version will run sequentially only. All other build options should be at their default values. I still haven't had the time to properly document what I did. Since 32bit seems to be a bit of an issue though, I've decided to share the binaries anyway. You can get them at http://customdebug.com/osm/osm2pgsql.zip I didn't have the time to properly test them, but I've used previous builds for full planet imports and rendering maps with mapnik (as well as some postgis queries for other things). The zip file includes 32 and 64 bit versions of 32 and 64 bit node id each (yes, four binaries). I've included all dependencies as well as I could. Please mind the licenses! Please test before using in any kind of production manner (seriously, who wouldn't?). In my environment I've only really used non-slim and in-memory modes with osm-planet, since I don't have access to a large enough ssd space (ironically more ram than ssd). If people are interested, I can share the entire chain: script to create and prepare the postgis db, import planet and a small qt app to render tiles using mapnik (2.0 currently) Feedback and bug reports welcome. I will share the source and build files soon. -Dominik my release notes: -- Compiled by Dominik Perpeet Feb 06 2013 git rev af61cae663 Dec 17, 2012 no fork (not supported in windows) no flatnodes support (node-persistent-cache) 64 bit node id: osm2pgsql.exe 32 bit node id: osm2pgsql_32bitId.exe Please keep in mind licenses of all parts: osm2pgsql (http://wiki.openstreetmap.org/wiki/Osm2pgsql) versions of libraries: bzip2 1.0.6 geos 3.3.5 libxml 2.9.0 pgsql 9.2.1 proj 4.8.0 proto_buf_c 0.1.5 (2.4.1) zlib 1.2.7 -- ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Stephan Knauss wrote: could you give details about a use case where mapnik needs the osm_id? The official styles do not contain a reference to osm_id, it's an internal thing in the database. Or is it going into using osm directly as a datasource instead of postrgres? While it is possible to feed osm files to mapnik directly, as far as I understand, osm_id can be fetched even from postgresql database as a feature id. Judging by a relevant issue https://github.com/mapnik/mapnik/issues/1662 before December it was 32-bit. Latest commits dealing with this problem were uploaded in January, including an option to turn off BIGINT support (it will be on by default) and some tests. IZ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Stephan Knauss o...@stephans-server.de writes: On 06.02.2013 21:25, Andrew M. Bishop wrote: Does anybody know if there is a released version of Mapnik that supports ids up to (2^32)-1 rather than requiring an unreleased 64-bit version? could you give details about a use case where mapnik needs the osm_id? The official styles do not contain a reference to osm_id, it's an internal thing in the database. Might be worth being clarified on the wiki page. Currently it reads as mapnik is broken in general. I asked the question based on the new wiki page which says that the unreleased Mapnik version 2.2 is required for 64-bit ids. My question could have been more accurately stated as: If using the standard toolchain of osm2psql, postgresql and mapnik what is the minimum software versions that are needed to continue creating maps after ids reach 2^31-1? The wiki page says that osm2psql version 0.81.1 is required but the version that I have reports itself as osm2pgsql SVN version 0.80.0 (32bit id space). This suggests to me that it will work up until id 2^32-1 appears but this may be wishful thinking on my part. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Andrew M. Bishop wrote: If using the standard toolchain of osm2psql, postgresql and mapnik what is the minimum software versions that are needed to continue creating maps after ids reach 2^31-1? I doubt that mapnik's handling of 32bit ids would in any way affect OSM rendering. So you can still use older versions of it without worrying that something will break. The wiki page says that osm2psql version 0.81.1 is required but the version that I have reports itself as osm2pgsql SVN version 0.80.0 (32bit id space). This suggests to me that it will work up until id 2^32-1 appears but this may be wishful thinking on my part. Well, you've got less than 24 hours to update your osm2pgsql. http://textual.ru/64/ shows that there are 1.3 million nodes left, and for the last two days nearly 1.5 million nodes were added daily. Though I fail to understand what would be broken if you continue to use 32-bit osm2pgsql. Indices, maybe. IZ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
FYI, I have dubbed this the N2B (Node 2 Billion(ish)) problem :) Just think what would happen if we could get 1/1000th the media coverage of Y2K... Toby ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
From: Ilya Zverev [mailto:zve...@textual.ru] Subject: Re: [OSM-dev] Reminder: Node 32-bit exhaustion Well, you've got less than 24 hours to update your osm2pgsql. http://textual.ru/64/ shows that there are 1.3 million nodes left, and for the last two days nearly 1.5 million nodes were added daily. Though I fail to understand what would be broken if you continue to use 32-bit osm2pgsql. Indices, maybe. Any tables which have node IDs in them will need to be using bigint, not int. You'll also have issues with anything that keeps track of node IDs for geometry building, etc. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Andrew M. Bishop wrote: Since 2^31 nodes will make older versions of some software unusable and 2^32 nodes will make other versions unusable it would be helpful to have a wiki page to record this information. I think that most useful would be a table of the minimum version of each piece of software to support true 32-bit ( 2^31) and 64-bit node numbers. I've started it at http://wiki.openstreetmap.org/wiki/64-bit_Identifiers Please update with any software I've missed. IZ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Stephan Knauss o...@stephans-server.de wrote:
How do I find out how many bits my server OS has?
A portable ANSI C code for bitsize detection would be this one:
#include stdio.h
#include stdint.h
int main() {
printf(%d\n,8*sizeof(uintptr_t));
return(0);
}
I'm using the size of an unsigned integer pointer because this is the one
used for memory access.
Sven
--
Whenever there is a conflict between human rights and property
rights, human rights must prevail. (Abraham Lincoln)
/me is giggls@ircnet, http://sven.gegg.us/ on the Web
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Ilya Zverev zve...@textual.ru writes: Andrew M. Bishop wrote: Since 2^31 nodes will make older versions of some software unusable and 2^32 nodes will make other versions unusable it would be helpful to have a wiki page to record this information. I think that most useful would be a table of the minimum version of each piece of software to support true 32-bit ( 2^31) and 64-bit node numbers. I've started it at http://wiki.openstreetmap.org/wiki/64-bit_Identifiers Please update with any software I've missed. That's very helpful, I have added my software (Routino) to the list. Does anybody know if there is a released version of Mapnik that supports ids up to (2^32)-1 rather than requiring an unreleased 64-bit version? -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
On 06.02.2013 21:25, Andrew M. Bishop wrote: Does anybody know if there is a released version of Mapnik that supports ids up to (2^32)-1 rather than requiring an unreleased 64-bit version? could you give details about a use case where mapnik needs the osm_id? The official styles do not contain a reference to osm_id, it's an internal thing in the database. Or is it going into using osm directly as a datasource instead of postrgres? Might be worth being clarified on the wiki page. Currently it reads as mapnik is broken in general. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
On Mon, Feb 04, 2013 at 06:34:46PM -0800, Paul Norman wrote: As of right now, we are 5.31 million nodes away from hitting our 2^31th node. This is likely to cause some software to break. Hopefully everything major has been tested, but if you haven't checked your software now would be a good time to do so. I think coastcheck will break in substantial untested ways, but I haven't checked. If it does, use OSMCoastline instead. http://wiki.openstreetmap.org/wiki/OSMCoastline It has been supplying Geofabrik map and tile.openstreetmap.de now for nearly a year without any problems. You can download the coastlines from http://openstreetmapdata.com/ , they are updated daily, unless there is some big problem with broken data in which case the old version is kept. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Paul Norman penor...@mac.com writes: As of right now, we are 5.31 million nodes away from hitting our 2^31th node. This is likely to cause some software to break. Hopefully everything major has been tested, but if you haven't checked your software now would be a good time to do so. I think coastcheck will break in substantial untested ways, but I haven't checked. Since 2^31 nodes will make older versions of some software unusable and 2^32 nodes will make other versions unusable it would be helpful to have a wiki page to record this information. I think that most useful would be a table of the minimum version of each piece of software to support true 32-bit ( 2^31) and 64-bit node numbers. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Hello Andrew, On 05.02.2013 21:13, Andrew M. Bishop wrote: Since 2^31 nodes will make older versions of some software unusable and 2^32 nodes will make other versions unusable it would be helpful to have a wiki page to record this information. I think that most useful would be a table of the minimum version of each piece of software to support true 32-bit ( 2^31) and 64-bit node numbers. I was going to suggest exactly the same. I recently double checked and contrary to my believe, my server was not a 64bit installation but 32bit userland inside a 64bit kernel. Maybe I selected the wrong basic image when buying the vserver years ago. And PHP has only 64bit ints as a 64bit binary :( I verified my scripts and think they are safe. You might want to check yours as well. I hope most software should be able to handle a long long (64bit int) on a 32 bit platform, checking does help bringing clarity. Sharing some basic stuff for an upcoming wiki page: How do I find out how many bits my server OS has? Windows: wmic os get osarchitecture Linux kernel: uname -m software: dpkg --print-architecture Webspace with PHP: ?php echo PHP int size: .PHP_INT_SIZE. which is max .PHP_INT_MAX; Consequences of 32 bit: PHP doing postgresql queries return ID from bigint column not as integer, but as string. If doing numerical operations a lib like GMP is required. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
On Wed, Feb 6, 2013 at 7:13 AM, Andrew M. Bishop a...@gedanken.demon.co.uk wrote: Since 2^31 nodes will make older versions of some software unusable and 2^32 nodes will make other versions unusable it would be helpful to have a wiki page to record this information. I think that most useful would be a table of the minimum version of each piece of software to support true 32-bit ( 2^31) and 64-bit node numbers. I might be pointing out the obvious here, but isn't the bigger risk that some software will fail silently rather than catastrophically? Like, an editor tries to save node 2^32+100, and saves it as 100, or something? Steve ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
On 06.02.2013 00:17, Steve Bennett wrote: I might be pointing out the obvious here, but isn't the bigger risk that some software will fail silently rather than catastrophically? Like, an editor tries to save node 2^32+100, and saves it as 100, or something? It is, but API requires you to provide a version along with PUT. So hopefully it would trigger a conflict in some cases. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Reminder: Node 32-bit exhaustion
As of right now, we are 5.31 million nodes away from hitting our 2^31th node. This is likely to cause some software to break. Hopefully everything major has been tested, but if you haven't checked your software now would be a good time to do so. I think coastcheck will break in substantial untested ways, but I haven't checked. At the node creation rate of 49k/day that has held over the last 10M nodes, we will hit this on Saturday February 9th at about 3 PM, UTC. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Reminder: Node 32-bit exhaustion
Paul, Thanks for the reminder, though you may want to check your math here. Ether the nodes per day is off or the expected exhaustion date is off! Skye Book On Feb 4, 2013, at 9:36 PM, Paul Norman penor...@mac.com wrote: As of right now, we are 5.31 million nodes away from hitting our 2^31th node. This is likely to cause some software to break. Hopefully everything major has been tested, but if you haven't checked your software now would be a good time to do so. I think coastcheck will break in substantial untested ways, but I haven't checked. At the node creation rate of 49k/day that has held over the last 10M nodes, we will hit this on Saturday February 9th at about 3 PM, UTC. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
