Re: [OSM-dev] Renderd problems.
Am 28.09.2010 16:58, schrieb Samir Faci (Dev): When I tried to use render_list to regenerate all the usual tiles (that I care about 0-13). It gets to up to lvl 10 then almost dies. When I tried to run just level 10 it dies almost immediately. renderd is not able to handle such big queues. In our tests it started to die at 1000 metatiles in queue. z0-13 for the whole planet are 21848 metatiles, z0-13 would be 1398104 metatiles -- so much more than renderd can handle. We used tirex instead and it's rendering queues of 25000 tiles and more in the background while still handling live traffic. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Renderd problems.
On Wed, 2010-09-29 at 08:20 +0200, Peter Körner wrote: Am 28.09.2010 16:58, schrieb Samir Faci (Dev): When I tried to use render_list to regenerate all the usual tiles (that I care about 0-13). It gets to up to lvl 10 then almost dies. When I tried to run just level 10 it dies almost immediately. renderd is not able to handle such big queues. In our tests it started to die at 1000 metatiles in queue. z0-13 for the whole planet are 21848 metatiles, z0-13 would be 1398104 metatiles -- so much more than renderd can handle. I'm not sure what went wrong for you but the main OpenStreetMap server regularly runs with the queue full of 1000 requests. I'm not aware of any fundamental problems in renderd with handling this level of load. I know that some sites have problems when trying to scale up to handle hundreds of simultaneous styles but this is something which it was not originally designed to handle. Loading styles on-demand is an obvious solution but there is an efficiency trade-off. Jon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Renderd problems.
Am 29.09.2010 10:10, schrieb Jon Burgess: I'm not sure what went wrong for you but the main OpenStreetMap server regularly runs with the queue full of 1000 requests. I'm not aware of any fundamental problems in renderd with handling this level of load. We started with a very small number of pre-rendedered tiles on disk when we routed (for some minutes) a lot of traffic to the server. mod_tile tried to push all tiles in the renderd queue but renderd had a hard limit of 1000 tiles in the queue. This limit was reached filled after seconds and from that point on the render processes tarted to deadlock themselves anyhow and the throughput got drastically down. I did not check which part of the whole process was the reason for all this but tirex does not have this hard-1000 limit and works much better with the huge number of styles. I don't say renderd is somehow bad, it was just not the right tool for our inhomogeneous and fast-changing setup with a lot of different styles. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Problem in running checktouch.pl (QA script by Gray68)
Hi, I am having problems in running checktouch.pl, a touch check QA script by Gray68 I have added required modules. I get following errors use of uninitialized value $OSM::osm::line in pattern match (m//) at OSM/ osm.pm line 232, $file line 396942. use of uninitialized value $OSM::osm::line in pattern match (m//) at OSM/ osm.pm line 494, $file line 396942. use of uninitialized value $OSM::osm::line in pattern match (m//) at OSM/ osm.pm line 477, $file line 396942. use of uninitialized value $OSM::osm::line in pattern match (m//) at OSM/ osm.pm line 279, $file line 396942. . . . Please help me -- Regards M Naveed Akram ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Renderd problems.
Aside from this one occurrence, I haven't had any issues with renderd, at least not in regards to queue time. render_list does seem to have a bug where it ignores the -l, --max-load=LOAD value. Though I need to look at it a bit more to see why its not using the value, I'm passing. Now, I do have one question regarding the shapeindex and shape files (ie. processed_p.shp ). Does running or loading a planet file modify the shapeindex? or should it affect it in anyway? I have the world_boundaries, shorelines, and processed_p deployed as a Debian package. I was presuming these files were fairly static aside from when there's an update that needs to be fetched from the website. Does the shapeindex need to be updated with any frequency? -- Samir On Wed, Sep 29, 2010 at 3:18 AM, Peter Körner osm-li...@mazdermind.de wrote: Am 29.09.2010 10:10, schrieb Jon Burgess: I'm not sure what went wrong for you but the main OpenStreetMap server regularly runs with the queue full of 1000 requests. I'm not aware of any fundamental problems in renderd with handling this level of load. We started with a very small number of pre-rendedered tiles on disk when we routed (for some minutes) a lot of traffic to the server. mod_tile tried to push all tiles in the renderd queue but renderd had a hard limit of 1000 tiles in the queue. This limit was reached filled after seconds and from that point on the render processes tarted to deadlock themselves anyhow and the throughput got drastically down. I did not check which part of the whole process was the reason for all this but tirex does not have this hard-1000 limit and works much better with the huge number of styles. I don't say renderd is somehow bad, it was just not the right tool for our inhomogeneous and fast-changing setup with a lot of different styles. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- -- Samir Faci *insert title* fortune | cowsay -f /usr/share/cows/tux.cow ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Attributes info in GPX track to upload to OSM?
Hi Potlatch devs, I sent the forwarded message to OSM devs yesterday, but now realize (thanks Shaun) that this is more a request for the editors than for OSM. The details are in the forwarded mail, but the big line is: Is there an agreed-upon way to put OSM-compliant tags in a GPX track so that when opened with an editor (such as Potlatch) the POIs would contain the right tags instead of just a name tag? I was thinking about maybe creating a custom GPX extension where OSM-compliant tags could be stored by the application. Those could then be understood by the different OSM editors. I located a spot in Potlatch 1's code [1] where such an adaptation could be done, but looking at Potlatch 2 [2] it's not completely clear to me which of the GPX nodes are retrieved/displayed in Potlatch. Could such a feature be added to Potlatch? It would greatly improve the workflow for people tagging POIs in their GPS logging applications (OSMTracker for Android in my case, but this could be supported by the other applications out there). Thanks, +Emilien [1] http://trac.openstreetmap.org/browser/applications/editors/potlatch/gps.as#L63 [2] http://trac.openstreetmap.org/browser/applications/editors/potlatch2/net/systemeD/potlatch2/utils/GpxImporter.as -- Forwarded message -- From: Emilien Klein emilien+...@klein.st Date: 2010/9/28 Subject: Attributes info in GPX track to upload to OSM? To: dev@openstreetmap.org Hi devs, I am a happy user of OSMTracker for Android to record my GPS tracks. However, once the GPX tracks are uploaded to OSM, the POIs show up with the interesting part in the name attribute, like this: ele: number name: Post box This happens because the info is saved like this in the GPX file: wpt lat=LAT lon=LON eleELEVATION/ele timeTIMEDATE/time name![CDATA[Post box]]/name satNUMBER/sat /wpt I would love OSMTracker for Android to generate a GPX file that once uploaded to OSM would already contain the right osm-compliant tags (osm-compliant as in defined on the Map Features wiki page). Quickly sifting through the GPX documentation [1], it seems to me that such a thing is not directly supported (no custom-named XML node). My question is if there's a way to make such a thing work. Some ideas: - Is there any OSM-specific extension defined to pass such information? - Or is there an agreed-upon way of putting that information in for instance the desc element? I could imagine some kind of CSV-formatted way of putting that info in the desc, like this: descamenity: post_box,OTHER_ATTRIBUTE: ATTRIBUTE_VALUE[...]/desc In case such a mechanism does not already exist, would it be a possible feature? This would come in handy for OSM-specific applications (such as OSMTracker for Android. the original OSMTracker for Windows mobile, etc) and would be beneficial to the users that upload tracks to OSM. FYI the original bug was reported at [2]. Thanks, +Emilien [1] http://www.topografix.com/GPX/1/1/#type_wptType [2] http://code.google.com/p/osmtracker-android/issues/detail?id=55 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Renderd problems.
Samir Faci (Dev) wrote: render_list does seem to have a bug where it ignores the -l, --max-load=LOAD value. Though I need to look at it a bit more to see why its not using the value, I'm passing. My guess would be the check_load() call needs to happen in the dequeueing path rather than the enqueueing, as otherwise the rendering continues out of the queue above load, which is rapidly filled up again the instance the load dips below the allowed. I haven't verified this though, so I might be wrong. Depending on how you see it (bug or feature?), a similar issue I think occurs with mod_tile / renderd. The load checks again are on the enqueueing side (mod_tile) rather than dequeueing side of renderd, meaning it continues to render under heavy load, just stops queuing new requests, although that has its benefits as well. -- View this message in context: http://gis.638310.n2.nabble.com/Renderd-problems-tp5579792p5584157.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Problem in running checktouch.pl (QA script by Gray68)
ok, what is the operating system you are running on? what was the actual command with parameters you submitted? where did you get the osm file from? and is it bzipped? what is the version of osm.pm? cheers gerhard On Wed, 2010-09-29 at 15:41 +0500, M Naveed Akram wrote: Hi, I am having problems in running checktouch.pl, a touch check QA script by Gray68 I have added required modules. I get following errors use of uninitialized value $OSM::osm::line in pattern match (m//) at OSM/osm.pm line 232, $file line 396942. use of uninitialized value $OSM::osm::line in pattern match (m//) at OSM/osm.pm line 494, $file line 396942. use of uninitialized value $OSM::osm::line in pattern match (m//) at OSM/osm.pm line 477, $file line 396942. use of uninitialized value $OSM::osm::line in pattern match (m//) at OSM/osm.pm line 279, $file line 396942. . . . Please help me -- Regards M Naveed Akram ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Renderd problems.
On Wed, 2010-09-29 at 07:31 -0700, Kai Krueger wrote: Samir Faci (Dev) wrote: render_list does seem to have a bug where it ignores the -l, --max-load=LOAD value. Though I need to look at it a bit more to see why its not using the value, I'm passing. My guess would be the check_load() call needs to happen in the dequeueing path rather than the enqueueing, as otherwise the rendering continues out of the queue above load, which is rapidly filled up again the instance the load dips below the allowed. I haven't verified this though, so I might be wrong. I think you are right, the check_load() call should really be inside the while() loop of the thread_main() otherwise the render threads carry on and empty the queue when the system is overloaded. This became an issue when I added the extra queueing to allow the multi-threaded requests. I just committed this change into SVN. Depending on how you see it (bug or feature?), a similar issue I think occurs with mod_tile / renderd. The load checks again are on the enqueueing side (mod_tile) rather than dequeueing side of renderd, meaning it continues to render under heavy load, just stops queuing new requests, although that has its benefits as well. We want checking in mod_tile so it can know whether it is even worth trying to enqueue the request or not. When the code was first written the maximum queue size was small and just doing the checking during enqueue was good enough. Now that some servers queue up so many requests that the queue might need several hours to empty it probably makes sense to check in renderd as well. Jon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
Hi, Stefan de Konink wrote: When this tool is polished enough (must include some bbox'ing then) we can think about osm2pbf. Speaking of polished: The program currently produces invalid XML because and are not escaped, leading to lines like tag k=name v=Gasthaus Zum Eckenhaider Schloß/ tag k=operator v=DB Station Service AG/ Same for ampersand characters in user names. Other than that, it runs about twice as fast as Osmosis so that's a good sign. What does the README mean when it says: At the time of writing it can only handle dense nodes? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
On Thu, 30 Sep 2010, Frederik Ramm wrote: Stefan de Konink wrote: When this tool is polished enough (must include some bbox'ing then) we can think about osm2pbf. Speaking of polished: The program currently produces invalid XML because and are not escaped, leading to lines like Yes, Roeland pointed that out as well yesterday. We have discussed an escape table. Maybe first parsing the entire string table, alternatively doing it for each instance. Other than that, it runs about twice as fast as Osmosis so that's a good sign. What does the README mean when it says: At the time of writing it can only handle dense nodes? ...that the README is already outdated ;) But what I already discussed with Scott, we *need* a good 'has everything' PBF file. Something that can test a parser and has expected output. For example Osmosis only generates dense nodes, not the 'other notation'. Also Osmosis currently implements deflate, not bzip2/lzma. (At the time of writing I implemented bzip2 as well, but couldn't test it.) There are a few things on the todo before going for the osm2pbf myself: - xml escaping - mmap input - lzma - allow the use of the bbox + based on the index + based on geos like solution Roeland wanted to go for a library so other tools could call getNextNode() etc. without fiddling with the internal structures. My personal interest is going for output that support output that can be used to 'copy into' data into a database. And extend the current protocol for diff support. (Hence: replacing osmsucker-ykw) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
On Wed, Sep 29, 2010 at 6:47 PM, Stefan de Konink ste...@konink.de wrote: On Thu, 30 Sep 2010, Frederik Ramm wrote: Speaking of polished: The program currently produces invalid XML because and are not escaped, leading to lines like Yes, Roeland pointed that out as well yesterday. We have discussed an escape table. Maybe first parsing the entire string table, alternatively doing it for each instance. In addition to and , you need to escape . planet.c also escapes . It uses character references for each (quot;, amp;, lt;, and gt;). planet.c also escapes carriage return, line feed, and tab, as #13;, #10;, and #9;. AFAICT it is legal to include these unescaped (though it would be nice to escape at least line feeds to make it easier on fast, non-XML-compliant parsers). Now, finally, there are characters in the db which cannot be represented in XML 1.0 (but can be represented in XML 1.1). Most significantly, control characters (ASCII less than 32) other than carriage return, line feed, and tab. Some versions of planet.c convert these into ?. Some versions omit them completely. At least one version converts them into #ASCII;, where ASCII is the ASCII code. I actually like the last version the best, though it is invalid in XML 1.0 (valid in XML 1.1). Personally I'd recommend producing XML 1.1, at least as an option, in order to include these characters. I don't believe there are any null characters in the database. These could not be represented in XML 1.0 nor XML 1.1. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
On Wed, 29 Sep 2010, Anthony wrote: In addition to and , you need to escape . planet.c also escapes . It uses character references for each (quot;, amp;, lt;, and gt;). planet.c also escapes carriage return, line feed, and tab, as #13;, #10;, and #9;. AFAICT it is legal to include these unescaped (though it would be nice to escape at least line feeds to make it easier on fast, non-XML-compliant parsers). What is currently in git escapes the first 5 cases. But doing the other few shouldn't be a big issue. I implemented an almost printf free version, but I have some problems getting good benchmarks between versions, need to find out where some massive overhead comes from. (Some person schedules bulk processing every night... while I'm coding... no clue why he can't schedule them when people are not coding... *g*) Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] pbf2osm development has started [code to test!]
But what I already discussed with Scott, we *need* a good 'has everything' PBF file. Something that can test a parser and has expected output. I know. I know. I need to hack a custom program that generates the entire protocol buffer, serializes it, hitting all of the interesting features and corner cases. The resulting file is the test-case. Doing that requires writing code that sets every field manually and it may be one or more weeks until I have a chance to do it. For example Osmosis only generates dense nodes, not the 'other notation'. Actually, it can also generate normal nodes with a command line flag. Also Osmosis currently implements deflate, not bzip2/lzma. (At the time of writing I implemented bzip2 as well, but couldn't test it.) If I can find a lzma/bzip2 java code, its ready to plug them in. Although, I'm thinking that with the cost of bzip2 decompression, the only interesting case is lzma. There are a few things on the todo before going for the osm2pbf myself: - xml escaping - mmap input - lzma The current planet XML format doesn't include any form of index. I think you could call osm2pbf complete, and offering significant advantages, without support for any indexing data structure at all. I would be happy to help in the design and algorithms of a geometric index, but I don't have the time to program it. Are you interested in coding this? If done properly, geometric sorting code can do much much more than make a planet.osm.pbf indexable. They can make very cheap tileservers. PrimitiveBlocks are independently decodable blobs; they can be stored in a database. Given a bbox query, select out all of the blobs that intersect it, concatenate them together, add a header, and thats a conforming *.osm.pbf file. Depending on the precision used and whether metadata is kept, that entire database can be cached into 6-10GB of RAM. I would like to see this happen. This may be a new way to distribute the planet: A set of PrimitiveBlock tiles sitting in an Sqlite database. Actually, now that I think of it, this may be the superior approach compared to using my hooks to hack an index into the *.osm.pbf format. - allow the use of the bbox What exactly do you mean by this: ? + based on the index And this: ? + based on geos like solution Roeland wanted to go for a library so other tools could call getNextNode() etc. without fiddling with the internal structures. My personal interest is going for output that support output that can be used to 'copy into' data into a database. And extend the current protocol for diff support. (Hence: replacing osmsucker-ykw) What kinds of extensions are you thinking of? Scott ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
