Re: [OSM-talk-nl] [OSM-dev] conversion AND data
Hi Marc, I want to convert _all_ NL data into .osm so I disabled bounds checking. 2AND runs forever and takes lots of memory :( Did you manage to convert the whole dataset? I'm using latest v5. Regards Artem On 29 Jul 2007, at 21:39, Marc Kessels wrote: latest version (v5) is online at http://www.kessels.name/and2osm/ this one seems to result in a proper output of the data, including all waterways and their islands (which were causing problems see mail below from Jon). only remaining urgent item is that some AND nodes (~800, so about 0.05%) have more than one ID. this might give problems for one-way streets. have to check whether this is a real problem or not. greetz, Marc On Sat, 2007-07-28 at 19:03 +0200, Marc Kessels wrote: On Sat, 2007-07-28 at 17:47 +0100, Jon Burgess wrote: On Sat, 2007-07-28 at 17:55 +0200, Marc Kessels wrote: Hi Jon, thanks for your feedback. At this moment no aggregation of roads is done in my code. Apparantly that is really required for good rendering. I will start to have a look at that. I also found several bugs in the code, so please download version 3 from http://www.kessels.name/and2osm/ It now also includes a bounding box, for faster conversion of a problamatic area. greetz, Marc Hi, I've just tried version 3. Initially I thought there was a problem becuase ig genreated lots of errors and finished in a few seconds but it looks like you've just got a tiny bounding box. It took me a while to find the data on the map :-) There is an example of where an area isn't closed properly in this data. See the park in the S-W corner: leisure=park, AND=1013. This has a long N-S segment joining it to another area instead of being closed. I'd be tempted to use the libavl binary tree. It should be more maintainable in the long term, for an example see http://trac.openstreetmap.org/browser/applications/rendering/ mapnik/all_tiles/C About the only thing you need to do is link against bst.[ch] and write the comparison function. All the tree management and searching is taken care of for you. libavl also provides other more advanced variants of the binary tree which can be dropped in with just a few lines of code change. Jon Hi, well, those errors were just reminders to the things that need fixing: some nodes (having a ND_ID) in the AND data appear to be a way (or area). This would result in a problem: to which point does the ID belong to... second problem: some (lat,lon) coordinates appear as different nodes having a different ID, so an easy lookup is not possible for checking the direction of a one-way street (which is not always the direction of the and-shape...). third problem: one-way streets have to be corresponding to the from-ID and to-ID and not to the shape direction (requires reversing segments, etc). compared to that, merging of ways is only a tiny problem... I have to leave now for a party, so feel free to submit a patch for the b-tree improvement. That was source-code I was looking for! greetz, Marc ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl ___ dev mailing list [EMAIL PROTECTED] http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev Artem Pavlenko http://mapnik.org ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
At Mon, 30 Jul 2007 14:55:28 +0100, Artem Pavlenko wrote: I want to convert _all_ NL data into .osm so I disabled bounds checking. 2AND runs forever and takes lots of memory :( Did you manage to convert the whole dataset? I'm using latest v5. It's important that you have enough memory. 2G is probably the minimum. If you have less, then 2AND doesn't fit in memory and you will have to wait forever while your operating system is busy swapping things in and out. Jeroen Dekkers ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
On Mon, 2007-07-30 at 17:02 +0200, Jeroen Dekkers wrote: At Mon, 30 Jul 2007 14:55:28 +0100, Artem Pavlenko wrote: I want to convert _all_ NL data into .osm so I disabled bounds checking. 2AND runs forever and takes lots of memory :( Did you manage to convert the whole dataset? I'm using latest v5. It's important that you have enough memory. 2G is probably the minimum. If you have less, then 2AND doesn't fit in memory and you will have to wait forever while your operating system is busy swapping things in and out. Jeroen Dekkers indeed, I just tested it, and it uses 1.431.296.000 Bytes of memory (on 64 bit system, could be less on a 32 bit system). the reason behind this huge memory consumption is that it needs to know all nodes and segments, in order to prevent nodes with the same coordinates, or multiple segments between the same two nodes. greetz, Marc ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
On 30 Jul 2007, at 16:02, Jeroen Dekkers wrote: At Mon, 30 Jul 2007 14:55:28 +0100, Artem Pavlenko wrote: I want to convert _all_ NL data into .osm so I disabled bounds checking. 2AND runs forever and takes lots of memory :( Did you manage to convert the whole dataset? I'm using latest v5. It's important that you have enough memory. 2G is probably the minimum. If you have less, then 2AND doesn't fit in memory and you will have to wait forever while your operating system is busy swapping things in and out. Jeroen Dekkers Hi Jeroen, I understand that current and2osm implementation is memory hungry. I just have a feeling it can be improved and this will benefit everyone in the longer run. NL data is fairly small in comparison to other countries. Cheers Artem Pavlenko http://mapnik.org ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
On Mon, 30 Jul 2007, Marc Kessels wrote: the reason behind this huge memory consumption is that it needs to know all nodes and segments, in order to prevent nodes with the same coordinates, or multiple segments between the same two nodes. On the main list I already mentioned why this not the way to go. You don't need to know *all* nodes in memory. You need to set a bounding box (based on next and previous segments), this reduces the the requirement memory and is able to 'stream' the through the AND file, with (indexed-)lookups in the output OSM file. Stefan ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
On Mon, 2007-07-30 at 18:53 +0200, Stefan de Konink wrote: On Mon, 30 Jul 2007, Marc Kessels wrote: the reason behind this huge memory consumption is that it needs to know all nodes and segments, in order to prevent nodes with the same coordinates, or multiple segments between the same two nodes. On the main list I already mentioned why this not the way to go. You don't need to know *all* nodes in memory. You need to set a bounding box (based on next and previous segments), this reduces the the requirement memory and is able to 'stream' the through the AND file, with (indexed-)lookups in the output OSM file. Stefan first: I know dev, talk and talk-nl, but could not find a post where you mention this. could be I don't have all the messages, or is there another list? Anyhow: my program actually does this: you can specify a bounding box (in the source, could be a command line parameter, but make 2AND is just as simple for me) and it will look at ways which have at least one point in this bounding box. The program will use much less memory in that case, of course. In case you make the bounding box as large as the Netherlands (or make the bouding box always return -1) you do need to remember all nodes, and it will consume a lot of memory. I suppose you are suggesting to divide the complete country into several bounding boxes, and export per box. Am I right? This would require that you go through the AND data a multiple number of times, which would take quite long. Also you will have an issue with ways that extend through a multiple number of boxes. Of course you could write some logic to catch this (e.g. by using only the most top left coordinate of a way) and output each way only one time, but you still have an issue that you are most likely creating duplicates of nodes outside of the current box. (the way will be attached to another way in another box, for wich also a new nodeID will be created). This could be solved by looking through the already created OSM file, but than you again have a file-access, and I think it will then be as slow as swapping data from and to memory... Obviously, I did not need to go through this hassle, since my machine did have enough memory :) ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
On 30 Jul 2007, at 18:23, Marc Kessels wrote: Obviously, I did not need to go through this hassle, since my machine did have enough memory :) Could you share *.osm file , then :) Artem Pavlenko http://mapnik.org ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
Stefan de Konink wrote: Sure, I don't want to sound hard because you made this in the first place, which is a very good job. But I could consider this lazy programming and bad design. Sure *it* works and it should probably only run one time, but since people are now experimenting with it, it could have been designed better reducing the memory requirements. Since this was no objective or requirement for you we should not bother... Anyway my comments can always be dropped to the dustbin :) I think the comments are very valuable, certainly concidering the fact that AND also gave India and China. A proper conversion routine will be very helpful. Maarten ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
At Mon, 30 Jul 2007 19:14:12 +0100, Artem Pavlenko wrote: On 30 Jul 2007, at 18:23, Marc Kessels wrote: Obviously, I did not need to go through this hassle, since my machine did have enough memory :) Could you share *.osm file , then :) Here is the .osm generated by and2osm version 5: http://mirror.openstreetmap.nl/and/ANDv5.osm.bz2 Jeroen Dekkers ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
On 30/07/07, Jeroen Dekkers [EMAIL PROTECTED] wrote: At Mon, 30 Jul 2007 19:14:12 +0100, Artem Pavlenko wrote: On 30 Jul 2007, at 18:23, Marc Kessels wrote: Obviously, I did not need to go through this hassle, since my machine did have enough memory :) Could you share *.osm file , then :) Here is the .osm generated by and2osm version 5: http://mirror.openstreetmap.nl/and/ANDv5.osm.bz2 Thanks, very much! A. Jeroen Dekkers ___ dev mailing list [EMAIL PROTECTED] http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
Anyway my comments can always be dropped to the dustbin :) comments are certainly welcome, patches even more! especially because I cannot follow your pseudo-code, but that can be because I am not a professional programmer. ;-) i'm clueless as well :) my script also consumed a lot of memory... I can think of a number of improvements to create a better import than the current one, like one that is able to merge the data directly into the osm-database via the api, looking at already existing data, and deciding on all the available info whether a way is already in the osm-database, even taking into account inaccuracies in osm and AND data... However, the aim I had in mind, was that using this import module, one could convert a limited area to osm, read that with JOSM into one layer and download the existing data in another layer. (which is possible in the latest version of JOSM). For empty area's merging can be done quickly (and even automatically, I think). and for already mapped data it will be more (hand)work. My current priority however, is first fixing some loose ends: -duplicate nodes in AND data may do a correction run over the AND data first ? -area's having a node-id in AND data, can you give an example? -merging ways having the same tags (exluding branching ways) i don't think there's any need for this. it will also be tricky with the turn restrictions. if you look at Assen, they also have lots of ways and mapnik renders that just nice. -put housenumbers in the data (conflicts with the previous) see above :) just don't merge ways -... like remove the source-url=www.and.com? help is appriciated. Marc Stefan ___ dev mailing list [EMAIL PROTECTED] http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
latest version (v5) is online at http://www.kessels.name/and2osm/ this one seems to result in a proper output of the data, including all waterways and their islands (which were causing problems see mail below from Jon). only remaining urgent item is that some AND nodes (~800, so about 0.05%) have more than one ID. this might give problems for one-way streets. have to check whether this is a real problem or not. greetz, Marc On Sat, 2007-07-28 at 19:03 +0200, Marc Kessels wrote: On Sat, 2007-07-28 at 17:47 +0100, Jon Burgess wrote: On Sat, 2007-07-28 at 17:55 +0200, Marc Kessels wrote: Hi Jon, thanks for your feedback. At this moment no aggregation of roads is done in my code. Apparantly that is really required for good rendering. I will start to have a look at that. I also found several bugs in the code, so please download version 3 from http://www.kessels.name/and2osm/ It now also includes a bounding box, for faster conversion of a problamatic area. greetz, Marc Hi, I've just tried version 3. Initially I thought there was a problem becuase ig genreated lots of errors and finished in a few seconds but it looks like you've just got a tiny bounding box. It took me a while to find the data on the map :-) There is an example of where an area isn't closed properly in this data. See the park in the S-W corner: leisure=park, AND=1013. This has a long N-S segment joining it to another area instead of being closed. I'd be tempted to use the libavl binary tree. It should be more maintainable in the long term, for an example see http://trac.openstreetmap.org/browser/applications/rendering/mapnik/all_tiles/C About the only thing you need to do is link against bst.[ch] and write the comparison function. All the tree management and searching is taken care of for you. libavl also provides other more advanced variants of the binary tree which can be dropped in with just a few lines of code change. Jon Hi, well, those errors were just reminders to the things that need fixing: some nodes (having a ND_ID) in the AND data appear to be a way (or area). This would result in a problem: to which point does the ID belong to... second problem: some (lat,lon) coordinates appear as different nodes having a different ID, so an easy lookup is not possible for checking the direction of a one-way street (which is not always the direction of the and-shape...). third problem: one-way streets have to be corresponding to the from-ID and to-ID and not to the shape direction (requires reversing segments, etc). compared to that, merging of ways is only a tiny problem... I have to leave now for a party, so feel free to submit a patch for the b-tree improvement. That was source-code I was looking for! greetz, Marc ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
On Sun, 2007-07-29 at 22:39 +0200, Marc Kessels wrote: latest version (v5) is online at http://www.kessels.name/and2osm/ this one seems to result in a proper output of the data, including all waterways and their islands (which were causing problems see mail below from Jon). only remaining urgent item is that some AND nodes (~800, so about 0.05%) have more than one ID. this might give problems for one-way streets. have to check whether this is a real problem or not. greetz, Marc It would be good to output only a single node in OSM. Unfortunately I don't think an object can have the same key twice, so you will not be able to reference multiple AND ids (unless you do something like external-ID=AND:1234;5678). Also I was wondering if these ID tags would be better as 'external-ID:AND=1234'. Any tool looking for AND specific keys in future might find these easier to find and parse. The TIGER dataset has the similar issues for node and road aggregation. Have you checked out what they've been working on to see if any of the code can be re-used? They've got to the point of creating .osm files too http://lists.openstreetmap.org/pipermail/dev/2007-July/005869.html but I've not kept up on the details of the conversion process. Jon ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
Re: [OSM-talk-nl] [OSM-dev] conversion AND data
On Sat, 2007-07-28 at 12:06 +0200, Marc Kessels wrote: I have updated the source, since it was experiencing a stack overflow, due to too many nodes with similar names. as a side effect, it is now also much faster (only 7 minutes to convert the compete dataset of The Netherlands). see http://wiki.openstreetmap.org/index.php/AND_Data/Import_Marc greetz, Marc I had a go at converting the data using your tool and then loading it into my local Mapnik renderer to generate some maps. At low zooms the data looks great however getting in close shows up several flaws. See the screenshot http://www.jburgess.uklinux.net/osm-and-water2.png * It looks like the aggregation of a road either side of a junction into a single way is not working properly. Below is the OSM data for part of Tweede Helmersstraat near the bottom left of my screenshot where the two ways should be joined at AND NODE 215559 but for some reason they appear as 2 distinct ways. The problem may be with nodes -748116 and -748858 which appear at the same location. Shouldn't the code detect this and re-use the node? way id=-247934 tag k=AND_FROM_NODE v=215559 / tag k=AND_TO_NODE v=215358 / tag k=external-ID v=AND=15216313 / tag k=oneway v=+1 / tag k=highway v=unclassified / tag k=name v=Tweede Helmersstraat / seg id=-500511 / seg id=-500512 / /way way id=-248207 tag k=AND_FROM_NODE v=215795 / tag k=AND_TO_NODE v=215559 / tag k=external-ID v=AND=15216586 / tag k=oneway v=+1 / tag k=highway v=unclassified / tag k=name v=Tweede Helmersstraat / seg id=-500979 / /way segment id=-500511 from=-748116 to=-748117 / segment id=-500512 from=-748117 to=-748118 / segment id=-500979 from=-748857 to=-748858 / node id=-748116 lat=52.36465 lon=4.87714 node id=-748117 lat=52.36499 lon=4.87851 node id=-748118 lat=52.36504 lon=4.87863 node id=-748857 lat=52.36426 lon=4.87596 node id=-748858 lat=52.36465 lon=4.87714 * Some of the water areas look like they are not correct and often have a triangular section where two distant points connect. I think similar artifacts are in the osmarender output on the web page so I think this is a genuine problem. This could be triggered by the previous problem if areas are not being output as a single complete way. Jon ___ Talk-nl mailing list Talk-nl@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk-nl
