Re: [OSM-dev] 404 from Apache2/mod_tile
Stefan Elspaß wrote > I compiled mod_tile again with svn-r28921 containing today's change from > apmon which writes the actual load into error.log. The result is strange: > > [info] [client xx.xx.xx.xx] Load (1202590843) larger max_load_missing > (50). Return HTTP_NOT_FOUND., referer: http://..xx/ > > Quite some high load, isn't it? I hadn't time yet to dig deeper but will > do that the coming days. If in the meantime someone has an idea: feel free > to share :-) It looks like there was a bug in mod_tile. It was missing an include of sys_utils.h and so the compiler was interpreting the return value of get_load_avg() as an int whereas it was a double resulting in totally bogus load values. Please do another svn update and check if this is fixed now? Kai -- View this message in context: http://gis.19327.n5.nabble.com/404-from-Apache2-mod-tile-tp5734970p5735268.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
Hi, Am 10.11.2012 um 18:01 schrieb Kai Krueger : >> Do you know by chance what unit "ModTileMaxLoadMissing" uses? Is it a >> percentage? The settings for the log below were as follows: > > The load is in units of "number of processes that are wanting CPU time". It > is the load number you get from /proc/loadavg. So if you have a load average > of 500 on a 1 core system, then each process only gets 1/500th of a CPU and > will be dog slow. The load numbers should therefor typically be not too much > more than the number of CPU cores you have. Therefore unless something is > drastically wrong with your server, you really shouldn't be seeing a load of > 500. I compiled mod_tile again with svn-r28921 containing today's change from apmon which writes the actual load into error.log. The result is strange: [info] [client xx.xx.xx.xx] Load (1202590843) larger max_load_missing (50). Return HTTP_NOT_FOUND., referer: http://..xx/ Quite some high load, isn't it? I hadn't time yet to dig deeper but will do that the coming days. If in the meantime someone has an idea: feel free to share :-) Greetings, Stefan___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
Stefan Elspaß wrote > Hi Kai, > >> If I read your renderd log correctly, mod_tile is sending rendering >> requests >> with priority "dirty" rather than the "missing" priority. The dirty >> priority >> is a background priority in which mod_tile does not wait for the result >> and >> immediately returns the tile (or in case of a missing tile a 404). > > This is indeed the case, but sometimes - in these seldom cases when > rendering on the first request works - the "command RenderPrio" instead of > "command Dirty" appears in /var/log/syslog (see below). Does that indicate > that it might by a load-related decision from mod_tile? > >> So the question is why does mod_tile send the requests with the incorrect >> priority? This should only happen if the load average is above the >> configured max missing load. Your configurations do look fine though. > > Do you know by chance what unit "ModTileMaxLoadMissing" uses? Is it a > percentage? The settings for the log below were as follows: The load is in units of "number of processes that are wanting CPU time". It is the load number you get from /proc/loadavg. So if you have a load average of 500 on a 1 core system, then each process only gets 1/500th of a CPU and will be dog slow. The load numbers should therefor typically be not too much more than the number of CPU cores you have. Therefore unless something is drastically wrong with your server, you really shouldn't be seeing a load of 500. Stefan Elspaß wrote >> Can you increase the debug level of mod_tile to debug and then post your >> apache error.log again? The debug output unfortunately currently doesn't >> tell you what mod_tile thinks the load on the server is, but at least it >> would confirm if this is the cause and what the configuration is set to. > > In /etc/apache2/site-available/default I changed the log-value by changing > the line to "LogLevel debug". Is there any other way to increase the debug > level of mod_tile? No, that is the correct way to increase the debug level of apache. And setting it to debug, will include all of the other levels like info warning and error. However, apache does not log to syslog. It has its own error.log file. It can typically be found in /var/log/apache/error.log Kai -- View this message in context: http://gis.19327.n5.nabble.com/404-from-Apache2-mod-tile-tp5734970p5735220.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
On 09.11.2012 20:37, Stefan Elspaß wrote: This is indeed the case, but sometimes - in these seldom cases when rendering on the first request works - the "command RenderPrio" instead of "command Dirty" appears in /var/log/syslog (see below). Does that indicate that it might by a load-related decision from mod_tile? mod_tile does read system load: get_load_avg() If this is higher than config value "ModTileMaxLoadMissing" then 404 is returned. But you should see this in the logfile. Can you enable level info? ap_log_rerror(APLOG_MARK, APLOG_INFO, 0, r, "Load larger max_load_missing (%d). Return HTTP_NOT_FOUND.", scfg->max_load_missing); Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
Hi Kai, > If I read your renderd log correctly, mod_tile is sending rendering requests > with priority "dirty" rather than the "missing" priority. The dirty priority > is a background priority in which mod_tile does not wait for the result and > immediately returns the tile (or in case of a missing tile a 404). This is indeed the case, but sometimes - in these seldom cases when rendering on the first request works - the "command RenderPrio" instead of "command Dirty" appears in /var/log/syslog (see below). Does that indicate that it might by a load-related decision from mod_tile? > So the question is why does mod_tile send the requests with the incorrect > priority? This should only happen if the load average is above the > configured max missing load. Your configurations do look fine though. Do you know by chance what unit "ModTileMaxLoadMissing" uses? Is it a percentage? The settings for the log below were as follows: ModTileRequestTimeout 900 ModTileMissingRequestTimeout 300 ModTileMaxLoadMissing 500 > Can you increase the debug level of mod_tile to debug and then post your > apache error.log again? The debug output unfortunately currently doesn't > tell you what mod_tile thinks the load on the server is, but at least it > would confirm if this is the cause and what the configuration is set to. In /etc/apache2/site-available/default I changed the log-value by changing the line to "LogLevel debug". Is there any other way to increase the debug level of mod_tile? Greetings, Stefan /var/log/syslog - Nov 9 20:15:44 xx renderd[16476]: DEBUG: Got incoming connection, fd 11, number 1 Nov 9 20:15:44 xx renderd[16476]: DEBUG: Got incoming connection, fd 12, number 2 Nov 9 20:15:44 xx renderd[16476]: DEBUG: Got command RenderPrio fd(12) xml(default), z(17), x(69332), y(43350) Nov 9 20:15:44 xx renderd[16476]: DEBUG: Got incoming connection, fd 13, number 3 Nov 9 20:15:44 xx renderd[16476]: DEBUG: Got incoming connection, fd 14, number 4 Nov 9 20:15:44 xx renderd[16476]: DEBUG: Got command RenderPrio fd(14) xml(default), z(17), x(69331), y(43349) Nov 9 20:15:44 xx renderd[16476]: DEBUG: Got command RenderPrio fd(13) xml(default), z(17), x(69332), y(43348) Nov 9 20:15:44 xx renderd[16476]: DEBUG: Got command RenderPrio fd(11) xml(default), z(17), x(69332), y(43349) Nov 9 20:15:44 xx renderd[16476]: DEBUG: Got incoming connection, fd 16, number 5 Nov 9 20:15:44 xx renderd[16476]: DEBUG: Got command RenderPrio fd(16) xml(default), z(17), x(69331), y(43348) Nov 9 20:15:47 xx renderd[16476]: DEBUG: DONE TILE default 17 69328-69335 43344-43351 Nov 9 20:15:47 xx renderd[16476]: DEBUG: DONE TILE default 17 69328-69335 43344-43351 in 3.495 seconds Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 1, fd 12 closed, now 4 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 2, fd 14 closed, now 3 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 1, fd 13 closed, now 2 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 1, fd 16 closed, now 1 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 0, fd 11 closed, now 0 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got incoming connection, fd 11, number 1 Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got command Dirty fd(11) xml(default), z(18), x(138663), y(86698) Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 0, fd 11 closed, now 0 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got incoming connection, fd 11, number 1 Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got incoming connection, fd 12, number 2 Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got command Dirty fd(11) xml(default), z(18), x(138663), y(86699) Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got incoming connection, fd 13, number 3 Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 0, fd 11 closed, now 2 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got incoming connection, fd 11, number 3 Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got command Dirty fd(12) xml(default), z(18), x(138662), y(86698) Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got command Dirty fd(13) xml(default), z(18), x(138662), y(86697) Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 0, fd 12 closed, now 2 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got command Dirty fd(11) xml(default), z(18), x(138663), y(86697) Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 0, fd 13 closed, now 1 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 0, fd 11 closed, now 0 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got incoming connection, fd 11, number 1 Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got command Dirty fd(11) xml(default), z(18), x(138662), y(86699) Nov 9 20:15:47 xx renderd[16476]: DEBUG: Connection 0, fd 11 closed, now 0 left Nov 9 20:15:47 xx renderd[16476]: DEBUG: Got incoming connection,
Re: [OSM-dev] 404 from Apache2/mod_tile
If I read your renderd log correctly, mod_tile is sending rendering requests with priority "dirty" rather than the "missing" priority. The dirty priority is a background priority in which mod_tile does not wait for the result and immediately returns the tile (or in case of a missing tile a 404). So the question is why does mod_tile send the requests with the incorrect priority? This should only happen if the load average is above the configured max missing load. Your configurations do look fine though. Can you increase the debug level of mod_tile to debug and then post your apache error.log again? The debug output unfortunately currently doesn't tell you what mod_tile thinks the load on the server is, but at least it would confirm if this is the cause and what the configuration is set to. Kai -- View this message in context: http://gis.19327.n5.nabble.com/404-from-Apache2-mod-tile-tp5734970p5735098.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
Hi, Am 09.11.2012 um 15:54 schrieb Yves CAINAUD : > I may have been the same issue until now. > Adding the general apache directive 'TimeOut 10' in the same virtualhost > seems to help (if 10 seconds are enough). Can you confirm ? Unfortunately not :-/. Still the same problem: 404 after 30-50ms on first request, second request ok with 200. Greetings, Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
(Sorry, was meant to the list) I may have been the same issue until now. Adding the general apache directive 'TimeOut 10' in the same virtualhost seems to help (if 10 seconds are enough). Can you confirm ? Yves ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
Hi Lynn and Martin, >> unless something was changed recently I think the values in >> mod_tile.conf are required to be in seconds. See also the default >> values: >> http://svn.openstreetmap.org/applications/utils/mod_tile/mod_tile.conf > > Ah well, I needed that egg on my face for breakfast anyway. It was just a > guess given the similarity between his 50 value and an observed timeout of > 29-49msec. If I understand the following code extract from http://svn.openstreetmap.org/applications/utils/mod_tile/mod_tile.c (function request_tile) correctly struct timeval tv = {(renderImmediately > 1?scfg->request_timeout_priority:scfg->request_timeout), 0 }; [...] s = select(fd+1, &rx, NULL, NULL, &tv); only the seconds-part (tv_sec) of the timeval struct is being set, not the microseconds-part (tv_usec). So I seconds should be correct. But thanks anyway for you answer. Any other ideas? Greetings, Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
On 11/9/2012 6:38 AM, Martin Koppenhoefer wrote: 2012/11/9 Lynn W. Deffenbaugh (Mr) : Have you considered that these might be in milliseconds? And that ModTileMaxLoadMissing is the parameter that you might want to try increasing to something like 1 (10 seconds)? Or increase all of them to 1 and see if you get the effect you're after. unless something was changed recently I think the values in mod_tile.conf are required to be in seconds. See also the default values: http://svn.openstreetmap.org/applications/utils/mod_tile/mod_tile.conf Ah well, I needed that egg on my face for breakfast anyway. It was just a guess given the similarity between his 50 value and an observed timeout of 29-49msec. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
>I would like the server to answer in these 3 seconds, but with >the tiles and HTTP 200 and not with an 404. > >I tried several settings, such as > >ModTileRequestTimeout 500 >ModTileMissingRequestTimeout 500 >ModTileMaxLoadMissing 50 > >but still the same 404. Is there a possibility for find out why >mod_tile return a 404, which timeout it runs into? I'm using mod_tile from not so long ago and all those values are taken into account fine. To troubleshoot, check your are using the "notice" log severity and logs go to error.log (but I'm unsure if you'll get what you want) next thing might code hacking/debuging... ps:just in case : did your stop and then start apache ? -- sly -- sly -- sly___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
2012/11/9 Lynn W. Deffenbaugh (Mr) : > On 11/9/2012 6:12 AM, Stefan Elspaß wrote: > Have you considered that these might be in milliseconds? And that > ModTileMaxLoadMissing is the parameter that you might want to try increasing > to something like 1 (10 seconds)? Or increase all of them to 1 and > see if you get the effect you're after. unless something was changed recently I think the values in mod_tile.conf are required to be in seconds. See also the default values: http://svn.openstreetmap.org/applications/utils/mod_tile/mod_tile.conf cheers, Martin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
On 11/9/2012 6:12 AM, Stefan Elspaß wrote: I tried several settings, such as ModTileRequestTimeout 500 ModTileMissingRequestTimeout 500 ModTileMaxLoadMissing 50 but still the same 404. Is there a possibility for find out why mod_tile return a 404, which timeout it runs into? Have you considered that these might be in milliseconds? And that ModTileMaxLoadMissing is the parameter that you might want to try increasing to something like 1 (10 seconds)? Or increase all of them to 1 and see if you get the effect you're after. Remember to restart renderd and/or apache when tweaking these because I'm not sure when they are actually loaded. Lynn (D) - KJ4ERJ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
Hi Pavithran, thanks for your answer. Am 09.11.2012 um 03:04 schrieb pavithran : > I understand you have set out the timeout configs but why is 17 or 18 > level tile not getting rendered on the fly ? Is it because of a very > slow hardware or dense location ( which is obvious due to map quality > in germany) . But I think its the postgres which is slowing you down . > Try keeping the postgres on a different machine if possible . The hardware is not too bad. AMD Athlon II X4 (Quad Core), 2,3GHz, 8 GB RAM, 2x1TB SATA II (Software-RAID1, but only 5.400rpm). Dedicated, not virtual. Unfortunately putting postgres on a different machine is not an option at this time due to budget restrictions. > Also regarding your choice of mod_tile , mod_tile is a meta tile > caching mechanism where the meta tile (8x8) is rendered . mod_tile is > good for prerendering and also serving large postion of tiles helpful > in panning , but the first request for a new tile in new location is > going to take time . :( For me it's ok that the first request takes some time. What I am only wondering about is that the metatile seems to be rendered in 3 seconds (Syslog: "Nov 8 20:23:55 xx renderd[16476]: DEBUG: DONE TILE default 17 69448-69455 43336-43343 in 2.967 seconds") - which would be ok for me for the first request - but the server answers with an 404 in 29-49ms. I would like the server to answer in these 3 seconds, but with the tiles and HTTP 200 and not with an 404. I tried several settings, such as ModTileRequestTimeout 500 ModTileMissingRequestTimeout 500 ModTileMaxLoadMissing 50 but still the same 404. Is there a possibility for find out why mod_tile return a 404, which timeout it runs into? Greetings, Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 404 from Apache2/mod_tile
On 9 November 2012 01:32, Stefan Elspaß wrote: > I have set up (for the first time) an Ubuntu Server with Apache2 and > renderd/mod_tile - using the instructions from switch2osm.org. I did the > import to PostGIS/Postgres with osm2pgsql (hstore) and used the german style. > With render_list I prerendered Germany up to zoom level 16 for performance > reasons. Nearly everything is working fine, my Leaflet-page works. Except: > when first accessing the (not prerendered) tiles in zoomlevel 17 and 18 I > often receive quite quickly (29-49ms) a HTTP 404 response. When accessing the > same tiles again some seconds later they are sent without any problems (HTTP > 200). I understand you have set out the timeout configs but why is 17 or 18 level tile not getting rendered on the fly ? Is it because of a very slow hardware or dense location ( which is obvious due to map quality in germany) . But I think its the postgres which is slowing you down . Try keeping the postgres on a different machine if possible . Also regarding your choice of mod_tile , mod_tile is a meta tile caching mechanism where the meta tile (8x8) is rendered . mod_tile is good for prerendering and also serving large postion of tiles helpful in panning , but the first request for a new tile in new location is going to take time . :( Do an $iotop to see whats going on with respect to IO , remember virtual machines have bad IO , so also a better file system would increase performance . Regards, pavithran -- pavithran sakamuri http://look-pavi.blogspot.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev