Re: [OSM-dev] tirex and mapnik 3, make deb fails
On Tue, 2014-07-01 01:58:38 +0200, Stephan Knauss o...@stephans-server.de wrote: I try to compile tirex (mapnik backend) against Mapnik 3. changing this line in debian/rules allowed me to ignore the error: dh_shlibdeps --dpkg-shlibdeps-params=--ignore-missing-info Won't work: A dep is missing and the resulting program will only run IFF the user has the lib already installed. dh_shlibdeps dpkg-shlibdeps: error: no dependency information found for /usr/local/lib/libmapnik.so.3.0 (used by debian/tirex-backend-mapnik/usr/lib/tirex/backends/mapnik) ^^^ dh_shlibdeps: dpkg-shlibdeps -Tdebian/tirex-backend-mapnik.substvars debian/tirex-backend-mapnik/usr/lib/tirex/backends/mapnik returned The problem is that the newly built binary is using a _local_ mapnik3 installation. Thus, there's no information available with package would provide that library. I guess mapnik3 isn't yet packaged, so either package it as well, or (though discouraged) link statically. MfG, JBG -- Jan-Benedict Glaw jbg...@lug-owl.de +49-172-7608481 Signature of: God put me on earth to accomplish a certain number of the second :things. Right now I am so far behind I will never die. signature.asc Description: Digital signature ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex and mapnik 3, make deb fails
Jan-Benedict Glaw writes: On Tue, 2014-07-01 01:58:38 +0200, Stephan Knauss o...@stephans-server.de wrote: I try to compile tirex (mapnik backend) against Mapnik 3. changing this line in debian/rules allowed me to ignore the error: dh_shlibdeps --dpkg-shlibdeps-params=--ignore-missing-info Won't work: A dep is missing and the resulting program will only run IFF the user has the lib already installed. So in my case this is perfectly fine. As I'm installing for a local machine where mapnik is also build. Mapnik 3 is not yet released and based on mapnik mailing list advise it's better compiled directly from git instead of using some (outdated?) nightly build from a ppd. So no packages available. The reason of doing a make deb instead of make install is the update path. I once made the mistake in the past and upgraded tirex to a new version to have access to bugfixes. The make install was simply overwriting already existing configuration files. Frederik or Jochen recommended using the deb package method for updates as it should take care for changes in already existing config files. So I worry a bit about the version information in the package. Is a newer build automatically recognized as a newer package? Does the build process create this version information? Honestly I'm just interested in getting an upgradeable tirex to the new machine. Learning all this stuff about debian package creation and fix the tirex build environment is certainly also an interesting topic but not the task I wanted to do now. For someone experienced in debian packaging it is certainly just a few minutes to check this and suggest a proper fix to the build environment. Stephan ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex and mapnik 3, make deb fails
Stephan Knauss o...@stephans-server.de wrote: Is a newer build automatically recognized as a newer package? No, use dch -i Regards Sven -- Der wichtigste Aspekt, den Sie vor der Entscheidung für ein Open Source-Betriebssystem bedenken sollten, ist, dass Sie kein Windows-Betriebssystem erhalten. (von http://www.dell.de/ubuntu) /me is giggls@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex and mapnik 3, make deb fails
Sven Geggus writes: Stephan Knauss o...@stephans-server.de wrote: Is a newer build automatically recognized as a newer package? No, use dch -i The last entry in tirex's debian/changelog is more than two years old. So its quite safe to assume that upstream changes will not result in new debian/ubuntu versions. Could the build-process be adapted to always run dch -i before building a package? Then the workflow to get updated tirex would be svn up, build package, install package Stephan ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] tirex and mapnik 3, make deb fails
I try to compile tirex (mapnik backend) against Mapnik 3. All done on ubuntu 14.04. Unfortunately it fails. make runs fine, but doing make deb after that produces errors. First problem was that it complained to not find mapnik-config. After patching the Makefile to contain the results of the mapnik-config call it successfully continued to break at another step. I have no idea about debian package creation. Can someone suggest a patch to resolve this issue? Sounds like it does not like I did make make install on mapnik changing this line in debian/rules allowed me to ignore the error: dh_shlibdeps --dpkg-shlibdeps-params=--ignore-missing-info does it matter that the packages have this version id in the name? _0.4.1precise1_ Will it conflict with my ubuntu setup which is trusty? Log: dh_shlibdeps dpkg-shlibdeps: error: no dependency information found for /usr/local/lib/libmapnik.so.3.0 (used by debian/tirex-backend- mapnik/usr/lib/tirex/backends/mapnik) dh_shlibdeps: dpkg-shlibdeps -Tdebian/tirex-backend-mapnik.substvars debian/tirex-backend-mapnik/usr/lib/tirex/backends/mapnik returned exit code 2 make: *** [binary-arch] Error 2 dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2 debuild: fatal error at line 1364: dpkg-buildpackage -rfakeroot -D -us -uc -I failed make: *** [deb] Error 29 ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] tirex: Can't locate mapscript.pm
Hello, where can I find the mapscript.pm? Greetings, Marcus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex: Can't locate mapscript.pm
Marcus Kruse kr...@vivai.de wrote: where can I find the mapscript.pm? Hm, are you trying to manually build tirex? You should really build a package instead using dpkg-buildpackage (at least on Debian and Ubuntu). The required perl dependencies can be found in debian/control back to your question. mapscript.pm is the mapserver perl Interface thus this is only necessary for the mapserver backend. On Debian/Ubuntu the package which contains mapscript.pm is called libmapscript-perl Regards Sven -- If you can spend five minutes on the Internet and do not run Linux, you're a genius. (Dirk Hohndel) /me is giggls@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex Installation
Hi, On 02/20/2013 05:12 PM, Gero Kriependorf wrote: Tiles without fonts are okay and rendered. But tiles with fonts I get the following error: Are the fonts installed properly at all? Can you render maps with e.g. nik2img? Try python -c from mapnik import FontEngine as e;print '\n'.join(e.instance().face_names()) and see if the resulting list includes the fonts that you are trying to use. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Tirex Installation
Hello everybody, I have problems getting Tirex up and running. Rendering maps with mapnik and render works fine. I use tirex from svn and mapnik version 2.1.0. I described the problem in detail on the irc channel: http://pastebin.com/Zn2KMZQw Even if you can't help me with the errors perhaps you can say me which versions you run and which you recommend. I am pleased for every help. I tried a lot until now but the errors remain. Just mapnik0.7 makes a difference, errors disappear but rendering is still not possible. The Wiki pages are so old and deprecated. Greetings Gero ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex Installation
Hello Martin, thank you for your answer. I tried setting both parameters, but nothing changed. I'm sure i set it to the right directory. It's confusing because I don't know if the font error or the python.input error is more worse. Greetings Gero On 20.02.2013 13:39, Martin Tesar wrote: Hi Gero, make sure you have edited configuration file /etc/tirex/renderer/mapnik.conf according to your mapnik installation. If the problem with fonts remains, try to add attribute font-directory=/path/to/your/fonts to your Map element in your mapnik style. Cheers, Martin 2013/2/20 Gero Kriependorf gkriepend...@umidev.de Hello everybody, I have problems getting Tirex up and running. Rendering maps with mapnik and render works fine. I use tirex from svn and mapnik version 2.1.0. I described the problem in detail on the irc channel: http://pastebin.com/Zn2KMZQw Even if you can't help me with the errors perhaps you can say me which versions you run and which you recommend. I am pleased for every help. I tried a lot until now but the errors remain. Just mapnik0.7 makes a difference, errors disappear but rendering is still not possible. The Wiki pages are so old and deprecated. Greetings Gero ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex Installation
Hello everybody, now I have tirex (SVN latest) and mapnik (v2.0.2 out of ppa from mapnik team) up and running. But rendering does still not work correctly. Tiles without fonts are okay and rendered. But tiles with fonts I get the following error: -- SNIP -- tirex-backend-mapnik[4062]: read: id=1361370596_16039968 map=osm prio=1 type=metatile_render_request x=1056 y=696 z=11 tirex-backend-mapnik[4062]: NetworkMessage::parse tirex-backend-mapnik[4062]: NetworkMessage::parse tirex-backend-mapnik[4062]: MetatileHandler::handleRequest tirex-backend-mapnik[4062]: MetatileHandler::render tirex-backend-mapnik[4062]: rendering area 626172.135712,6261721.357122 - 782715.169640,6418264.391050 in SRS 3857 to 2048x2048 px tirex-backend-mapnik[4062]: width: 2048, height:2048 tirex-backend-mapnik[4062]: Mapnik config error: Unable to find specified font face '' tirex-backend-mapnik[4062]: MetatileHandler::render tirex-backend-mapnik[4062]: NetworkMessage::build tirex-backend-mapnik[4062]: NetworkMessage::build tirex-backend-mapnik[4062]: sending: errmsg=renderer internal error id=1361370596_16039968 result=error type=metatile_render_request -- SNAP -- I don't know what could be wrong. I checked the font things a thousand times. Rendering via render works without problems, using the same style. Does anyone know if tirex needs something special? I tried every font directory i can imagine and checked it twice with recursive on and off. Greetings Gero On 20.02.2013 14:23, Martin Tesar wrote: I have the same python.input error. Setting font-directory=/usr/share/fonts/truetype/ttf-dejavu worked. Hope this helps, Martin 2013/2/20 Gero Kriependorf gkriepend...@umidev.de Hello Martin, thank you for your answer. I tried setting both parameters, but nothing changed. I'm sure i set it to the right directory. It's confusing because I don't know if the font error or the python.input error is more worse. Greetings Gero On 20.02.2013 13:39, Martin Tesar wrote: Hi Gero, make sure you have edited configuration file /etc/tirex/renderer/mapnik.conf according to your mapnik installation. If the problem with fonts remains, try to add attribute font-directory=/path/to/your/fonts to your Map element in your mapnik style. Cheers, Martin 2013/2/20 Gero Kriependorf gkriepend...@umidev.de Hello everybody, I have problems getting Tirex up and running. Rendering maps with mapnik and render works fine. I use tirex from svn and mapnik version 2.1.0. I described the problem in detail on the irc channel: http://pastebin.com/Zn2KMZQw Even if you can't help me with the errors perhaps you can say me which versions you run and which you recommend. I am pleased for every help. I tried a lot until now but the errors remain. Just mapnik0.7 makes a difference, errors disappear but rendering is still not possible. The Wiki pages are so old and deprecated. Greetings Gero ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Tirex-backend-manager mapnik problem
Hi, I have mapnik 2.1.0 and tirex installed on Debian Squeeze, boost1.47. Tirex is patched with this: https://gist.github.com/3899892 Mapnik installation works well, but when I'm trying to render simple map (just processed_p.shp layer) with mapnik + tirex-batch, tirex-backend-manager in debug mode outputs following backtrace and disables mapnik renderer. Anyone knows what might be a problem? Thanks, Martin *** glibc detected *** /usr/lib/tirex/backends/mapnik: double free or corruption (out): 0x7fd4ae23b0c0 *** === Backtrace: = /lib/libc.so.6(+0x71e16)[0x7fd4adf4ee16] /lib/libc.so.6(cfree+0x6c)[0x7fd4adf53b8c] /usr/lib/libstdc++.so.6(_ZNSs6assignERKSs+0x85)[0x7fd4ae77ffd5] /usr/lib/tirex/backends/mapnik[0x40c890] /usr/lib/tirex/backends/mapnik[0x40e274] /usr/lib/tirex/backends/mapnik[0x408de9] /lib/libc.so.6(__libc_start_main+0xfd)[0x7fd4adefbc8d] /usr/lib/tirex/backends/mapnik[0x407459] === Memory map: 0040-00415000 r-xp 09:00 592174 /usr/lib/tirex/backends/mapnik 00614000-00615000 rw-p 00014000 09:00 592174 /usr/lib/tirex/backends/mapnik 02527000-0278c000 rw-p 00:00 0 [heap] 7fd47c00-7fd47c021000 rw-p 00:00 0 7fd47c021000-7fd48000 ---p 00:00 0 7fd48308b000-7fd4832f2000 r--s 09:32 4949200 /mnt/nfs4/mtbmap-czechrep/Data/shp/world_boundaries/processed_p.index 7fd4832f2000-7fd483837000 r--s 09:32 4953143 /mnt/nfs4/mtbmap-czechrep/Data/shp/world_boundaries/processed_p.dbf 7fd483837000-7fd4a1832000 r--s 09:32 4949201 /mnt/nfs4/mtbmap-czechrep/Data/shp/world_boundaries/processed_p.shp 7fd4a290d000-7fd4a2938000 r-xp 09:00 560043 /usr/local/lib/mapnik/input/osm.input 7fd4a2938000-7fd4a2b37000 ---p 0002b000 09:00 560043 /usr/local/lib/mapnik/input/osm.input 7fd4a2b37000-7fd4a2b39000 rw-p 0002a000 09:00 560043 /usr/local/lib/mapnik/input/osm.input 7fd4a2b39000-7fd4a2b59000 r-xp 09:00 560041 /usr/local/lib/mapnik/input/geojson.input 7fd4a2b59000-7fd4a2d58000 ---p 0002 09:00 560041 /usr/local/lib/mapnik/input/geojson.input 7fd4a2d58000-7fd4a2d5a000 rw-p 0001f000 09:00 560041 /usr/local/lib/mapnik/input/geojson.input 7fd4a2d5a000-7fd4a2d9 r-xp 09:00 560048 /usr/local/lib/mapnik/input/sqlite.input 7fd4a2d9-7fd4a2f9 ---p 00036000 09:00 560048 /usr/local/lib/mapnik/input/sqlite.input 7fd4a2f9-7fd4a2f92000 rw-p 00036000 09:00 560048 /usr/local/lib/mapnik/input/sqlite.input 7fd4a2f92000-7fd4a2fcb000 r-xp 09:00 560046 /usr/local/lib/mapnik/input/raster.input 7fd4a2fcb000-7fd4a31ca000 ---p 00039000 09:00 560046 /usr/local/lib/mapnik/input/raster.input 7fd4a31ca000-7fd4a31cd000 rw-p 00038000 09:00 560046 /usr/local/lib/mapnik/input/raster.input 7fd4a31cd000-7fd4a320d000 r-xp 09:00 560044 /usr/local/lib/mapnik/input/postgis.input 7fd4a320d000-7fd4a340c000 ---p 0004 09:00 560044 /usr/local/lib/mapnik/input/postgis.input 7fd4a340c000-7fd4a340f000 rw-p 0003f000 09:00 560044 /usr/local/lib/mapnik/input/postgis.input 7fd4a340f000-7fd4a3436000 r-xp 09:00 560042 /usr/local/lib/mapnik/input/ogr.input 7fd4a3436000-7fd4a3636000 ---p 00027000 09:00 560042 /usr/local/lib/mapnik/input/ogr.input 7fd4a3636000-7fd4a3638000 rw-p 00027000 09:00 560042 /usr/local/lib/mapnik/input/ogr.input 7fd4a3638000-7fd4a3671000 r-xp 09:00 560039 /usr/local/lib/mapnik/input/csv.input 7fd4a3671000-7fd4a3871000 ---p 00039000 09:00 560039 /usr/local/lib/mapnik/input/csv.input 7fd4a3871000-7fd4a3873000 rw-p 00039000 09:00 560039 /usr/local/lib/mapnik/input/csv.input 7fd4a3873000-7fd4a38b8000 r-xp 09:00 560047 /usr/local/lib/mapnik/input/shape.input 7fd4a38b8000-7fd4a3ab8000 ---p 00045000 09:00 560047 /usr/local/lib/mapnik/input/shape.input 7fd4a3ab8000-7fd4a3abb000 rw-p 00045000 09:00 560047 /usr/local/lib/mapnik/input/shape.input 7fd4a3abb000-7fd4a3abd000 r-xp 09:00 425655 /lib/libutil-2.11.3.so 7fd4a3abd000-7fd4a3cbc000 ---p 2000 09:00 425655 /lib/libutil-2.11.3.so 7fd4a3cbc000-7fd4a3cbd000 r--p 1000 09:00 425655 /lib/libutil-2.11.3.so 7fd4a3cbd000-7fd4a3cbe000 rw-p 2000 09:00 425655 /lib/libutil-2.11.3.so 7fd4a3cbe000-7fd4a3d0e000 r-xp 09:00 564409 /usr/local/lib/libboost_python.so.1.47.0 7fd4a3d0e000-7fd4a3f0d000 ---p 0005 09:00 564409 /usr/local/lib/libboost_python.so.1.47.0 7fd4a3f0d000-7fd4a3f11000 rw-p 0004f000 09:00 564409 /usr/local/lib/libboost_python.so.1.47.0 7fd4a3f11000-7fd4a415 r-xp 09:00 582672 /usr/lib/libpython2.6.so.1.0 7fd4a415-7fd4a435 ---p 0023f000 09:00 582672 /usr/lib/libpython2.6.so.1.0 7fd4a435-7fd4a43b4000 rw-p 0023f000 09:00 582672 /usr/lib/libpython2.6.so.1.0 7fd4a43b4000-7fd4a43c2000 rw-p 00:00 0 7fd4a43c2000-7fd4a43de000 r-xp 09:00 560045 /usr/local/lib/mapnik/input/python.input 7fd4a43de000-7fd4a45dd000 ---p 0001c000 09:00 560045 /usr/local/lib/mapnik/input/python.input 7fd4a45dd000-7fd4a45df000 rw-p 0001b000 09:00 560045
[OSM-dev] tirex-batch tiles are not used by mod_tile
hello, i have a working mod_tile/tirex setup. to save load on the tileserver i pre-created all tiles for europe up to zoomlevel 15. tirex-batch map=terrain bbox=-12,34,48,60 z=5-15 this took about 5days of rendering time, but it seems, that this tiles are not used by mod_tile. here are some lines from tirex/jobs.log 2012-03-22T21:27:55 id=1332438509_61930816 map=terrain x=4536 y=2768 z=13 prio=99 request_time=1332438509 expire= sources= render_time=2877 success=1 2012-03-27T17:33:32 id=1332862409_46017048 map=terrain x=4536 y=2768 z=13 prio=2 request_time=1332862409 expire= sources= render_time=2646 success=1 you can see, the above tile is rendered twice. and if you request this tile, you have to wait until the tile is rendered (i.e. the old tile in filesystem is not delivered to the client). after the tiles is displayed in the browser the mod_tile cache works perfekt. please can anyone help me? thx and greetings from vienna. max. ps: you can look to the work at http://mtk2.toursprung.com/v8 pps: here is the output of tirex-tiledir-stats --Tiles--- --Age-- --Size- -Blocks Block Map Zoom CountMax Empty Exist% Empty% Min Avg Max Min Avg Max Min Avg Max Eff % --- terrain 4 2 4 0 50.00 0.002d22h 2d22h 2d22h 162338 186201 210065 320 368 416 99 terrain 5 2 16 0 12.50 0.001d 8h 1d 8h 1d 8h 189194 360664 532134 376 7081040 99 terrain 6 7 64 0 10.94 0.001d 0h 2d13h 5d 3h 125988 363637 1268928 248 7142480 99 terrain 7 12256 0 4.69 0.004h40m 2d 0h 5d 1h 188731 678146 1605715 37613293144 100 terrain 8 28 1024 2 2.73 0.204h41m 2d19h 5d 1h 7124 973329 1873490 1619053664 100 terrain 9 96 4096 6 2.34 0.152h 3m 4d 2h 5d 1h 7124 1188840 2939793 1623255744 100 terrain10 345 16384 30 2.11 0.18 16m40s 4d12h 5d 0h 7124 1143165 3046570 1622365952 100 terrain111286 65536145 1.96 0.22 16m51s 4d13h 5d 0h 7124 941342 2744138 1618425360 100 terrain124959 262144695 1.89 0.27 16m38s 4d20h 5d 1h 7124 708337 2525823 1613874936 100 terrain13 194361048576 3477 1.85 0.33 16m35s 4d18h 4d21h 7124 556441 2332295 1610904560 100 terrain14 771754194304 17314 1.84 0.41 24m13s 3d23h 4d 8h 7124 364792 2402824 16 7164696 100 terrain15 306432 16777216 82938 1.83 0.491h29m 1d20h 3d 8h 7124 245844 2057779 16 4834024 99 terrain16 86 67108864 2 0.00 0.00 31m 7s 1d16h 5d 1h 7124 804035 1503334 1615742944 100 terrain17 61 268435456 0 0.00 0.00 34m48s 2d13h 5d 1h 178119 698415 1053389 35213682064 100 --- ALL ALL 409927 357913940 104609 0.11 0.03 16m35s 2d10h 5d 3h 7124 291966 3046570 16 5735952 99 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] tirex-batch tiles are not used by mod_tile
hello, i have a working mod_tile/tirex setup. to save load on the tileserver i pre-created all tiles for europe up to zoomlevel 15. tirex-batch map=terrain bbox=-12,34,48,60 z=5-15 this took about 5days of rendering time, but it seems, that this tiles are not used by mod_tile. here are some lines from tirex/jobs.log 2012-03-22T21:27:55 id=1332438509_61930816 map=terrain x=4536 y=2768 z=13 prio=99 request_time=1332438509 expire= sources= render_time=2877 success=1 2012-03-27T17:33:32 id=1332862409_46017048 map=terrain x=4536 y=2768 z=13 prio=2 request_time=1332862409 expire= sources= render_time=2646 success=1 you can see, the above tile is rendered twice. and if you request this tile, you have to wait until the tile is rendered (i.e. the old tile in filesystem is not delivered to the client). after the tiles is displayed in the browser the mod_tile cache works perfekt. please can anyone help me? thx and greetings from vienna. max. ps: you can look to the work at http://mtk2.toursprung.com/v8 pps: here is the output of tirex-tiledir-stats --Tiles--- --Age-- --Size- -Blocks Block Map Zoom CountMax Empty Exist% Empty% Min Avg Max Min Avg Max Min Avg Max Eff % --- terrain 4 2 4 0 50.00 0.002d22h 2d22h 2d22h 162338 186201 210065 320 368 416 99 terrain 5 2 16 0 12.50 0.001d 8h 1d 8h 1d 8h 189194 360664 532134 376 7081040 99 terrain 6 7 64 0 10.94 0.001d 0h 2d13h 5d 3h 125988 363637 1268928 248 7142480 99 terrain 7 12256 0 4.69 0.004h40m 2d 0h 5d 1h 188731 678146 1605715 37613293144 100 terrain 8 28 1024 2 2.73 0.204h41m 2d19h 5d 1h 7124 973329 1873490 1619053664 100 terrain 9 96 4096 6 2.34 0.152h 3m 4d 2h 5d 1h 7124 1188840 2939793 1623255744 100 terrain10 345 16384 30 2.11 0.18 16m40s 4d12h 5d 0h 7124 1143165 3046570 1622365952 100 terrain111286 65536145 1.96 0.22 16m51s 4d13h 5d 0h 7124 941342 2744138 1618425360 100 terrain124959 262144695 1.89 0.27 16m38s 4d20h 5d 1h 7124 708337 2525823 1613874936 100 terrain13 194361048576 3477 1.85 0.33 16m35s 4d18h 4d21h 7124 556441 2332295 1610904560 100 terrain14 771754194304 17314 1.84 0.41 24m13s 3d23h 4d 8h 7124 364792 2402824 16 7164696 100 terrain15 306432 16777216 82938 1.83 0.491h29m 1d20h 3d 8h 7124 245844 2057779 16 4834024 99 terrain16 86 67108864 2 0.00 0.00 31m 7s 1d16h 5d 1h 7124 804035 1503334 1615742944100 terrain17 61 268435456 0 0.00 0.00 34m48s 2d13h 5d 1h 178119 698415 1053389 35213682064 100 --- ALL ALL 409927 357913940 104609 0.11 0.03 16m35s 2d10h 5d 3h 7124 291966 3046570 16 573 5952 99 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex-batch tiles are not used by mod_tile
Max Wukits writes: this took about 5days of rendering time, but it seems, that this tiles are not used by mod_tile. here are some lines from tirex/jobs.log I'm not sure about your log output. Is it possible that your tiles are considered outdated? Can you compare the timestamps with planet-import-complete? Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Tirex compilation bug
Hi,
I just try to compile Tirex from sources and face next error:
*user@optiplex:~/mapnik2/tirex$ make*
*cd backend-mapnik; make*
*make[1]: Entering directory `/home/user/mapnik2/tirex/backend-mapnik'*
*g++ -g -O2 -Wall -Wextra -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I
/usr/include/freetype2/ -DBOOST_FILESYSTEM_VERSION=2-c -o renderd.o
renderd.cc*
*renderd.cc: In member function ‘bool RenderDaemon::loadMapnikWrapper(const
char*)’:*
*renderd.cc:123: error: expected type-specifier*
*renderd.cc:123: error: expected unqualified-id before ‘cfgerr’*
*renderd.cc:123: error: expected ‘)’ before ‘cfgerr’*
*renderd.cc:123: error: expected ‘{’ before ‘cfgerr’*
*renderd.cc:123: error: ‘cfgerr’ was not declared in this scope*
*renderd.cc:123: error: expected ‘;’ before ‘)’ token*
*make[1]: *** [renderd.o] Error 1*
*make[1]: Leaving directory `/home/user/mapnik2/tirex/backend-mapnik'*
*make: *** [build] Error 2*
Installed under Ubuntu 10.04 LTS, Mapnik 2 (from sources)
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex compilation bug
https://github.com/mapnik/mapnik/issues/1139
On 21 March 2012 14:31, Valery N. cat_cr...@tut.by wrote:
Hi,
I just try to compile Tirex from sources and face next error:
*user@optiplex:~/mapnik2/tirex$ make*
*cd backend-mapnik; make*
*make[1]: Entering directory `/home/user/mapnik2/tirex/backend-mapnik'*
*g++ -g -O2 -Wall -Wextra -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I
/usr/include/freetype2/ -DBOOST_FILESYSTEM_VERSION=2-c -o renderd.o
renderd.cc*
*renderd.cc: In member function ‘bool
RenderDaemon::loadMapnikWrapper(const char*)’:*
*renderd.cc:123: error: expected type-specifier*
*renderd.cc:123: error: expected unqualified-id before ‘cfgerr’*
*renderd.cc:123: error: expected ‘)’ before ‘cfgerr’*
*renderd.cc:123: error: expected ‘{’ before ‘cfgerr’*
*renderd.cc:123: error: ‘cfgerr’ was not declared in this scope*
*renderd.cc:123: error: expected ‘;’ before ‘)’ token*
*make[1]: *** [renderd.o] Error 1*
*make[1]: Leaving directory `/home/user/mapnik2/tirex/backend-mapnik'*
*make: *** [build] Error 2*
Installed under Ubuntu 10.04 LTS, Mapnik 2 (from sources)
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex timeout - could not send data to client: Broken pipe
Hi, On 28.11.2011 12:36, Frederik Ramm wrote: Do you have messages like sending HUP to worker 'mapnik' with pid xxx (due to timeout) in your /var/log/daemon.log? If yes, then the mapnik backend has timed out, probably because you forgot to set backend_manager_alive_timeout to a similarly high value (55 would be good in your case). You were right. Problem was in parameter backend_manager_alive_timeout, that I forgot to change. Thank you! Regards Petr ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] tirex timeout - could not send data to client: Broken pipe
Hello everyone! We created own OSM server – unfortunately there is problem when we render metatiles in levels 6-10. Roughly one half of metatiles is not rendered due to tirex time-out. The second half and also tiles in level 11-18 are rendered correctly. We tried increase tirex timeout parameter (master_rendering_timeout=60), but it didn’t help. After that we found out problem in postgresql log: 2011-11-28 10:56:56 CET LOG: unexpected EOF on client connection 2011-11-28 10:56:56 CET LOG: could not send data to client: Broken pipe 2011-11-28 10:56:56 CET STATEMENT: SELECT AsBinary(way) AS geom,landuse,natural,waterway from Our environment: -PostgreSQL 8.4 -PostGIS 1.5 -Mapnik 0.7.1 -Tirex 0.2.0 We use standard OSM style xml : http://svn.openstreetmap.org/applications/rendering/mapnik/osm.xml DB user tirex is authenticate based on ident authentication Does anybody know what causes this issue? Thanks for you help Best Regards Petr Voldan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex timeout - could not send data to client: Broken pipe
Hi, On 11/28/11 11:15, Petr Voldán wrote: We created own OSM server – unfortunately there is problem when we render metatiles in levels 6-10. Roughly one half of metatiles is not rendered due to tirex time-out. The second half and also tiles in level 11-18 are rendered correctly. We tried increase tirex timeout parameter (master_rendering_timeout=60), but it didn’t help. After that we found out problem in postgresql log: 2011-11-28 10:56:56 CET LOG: unexpected EOF on client connection 2011-11-28 10:56:56 CET LOG: could not send data to client: Broken pipe 2011-11-28 10:56:56 CET STATEMENT: SELECT AsBinary(way) AS geom,landuse,natural,waterway from Do you have messages like sending HUP to worker 'mapnik' with pid xxx (due to timeout) in your /var/log/daemon.log? If yes, then the mapnik backend has timed out, probably because you forgot to set backend_manager_alive_timeout to a similarly high value (55 would be good in your case). Also, try and make sure that when you pre-render stuff on critical zoom levels, you only have one rendering thread (i.e. enqueue your requests with a suitable priority like 99). Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex-backend-manager doesn't work
Hi, I have now switched from Tirex to mod_tile/Renderd and everything seems be working well. However, the processing of tiles take very long, how would you evaluate this or why is that? Nov 16 23:24:17 osm renderd[8237]: DEBUG: DONE TILE default 6 24-31 16-23 Nov 16 23:24:17 osm renderd[8237]: DEBUG: DONE TILE default 6 24-31 16-23 in 4191.478 seconds Nov 17 04:54:18 osm renderd[8237]: DEBUG: DONE TILE default 10 528-535 352-359 Nov 17 04:54:18 osm renderd[8237]: DEBUG: DONE TILE default 10 528-535 352-359 in 23835.356 seconds Nov 17 05:01:58 osm renderd[8237]: DEBUG: DONE TILE default 7 56-63 40-47 Nov 17 05:01:58 osm renderd[8237]: DEBUG: DONE TILE default 7 56-63 40-47 in 18617.803 seconds Nov 17 05:18:45 osm renderd[8237]: DEBUG: DONE TILE default 7 56-63 32-39 Nov 17 05:18:45 osm renderd[8237]: DEBUG: DONE TILE default 7 56-63 32-39 in 1465.579 seconds Nov 17 05:32:08 osm renderd[8237]: DEBUG: DONE TILE default 7 56-63 48-55 Nov 17 05:32:08 osm renderd[8237]: DEBUG: DONE TILE default 7 56-63 48-55 in 1809.016 seconds Nov 17 05:38:59 osm renderd[8237]: DEBUG: DONE TILE default 7 64-71 32-39 Nov 17 05:38:59 osm renderd[8237]: DEBUG: DONE TILE default 7 64-71 32-39 in 18667.911 seconds Nov 17 05:44:44 osm renderd[8237]: DEBUG: DONE TILE default 7 64-71 40-47 Nov 17 05:44:45 osm renderd[8237]: DEBUG: DONE TILE default 7 64-71 40-47 in 21184.779 seconds Nov 17 13:47:22 osm renderd[8237]: DEBUG: DONE TILE default 9 272-279 168-175 Nov 17 13:47:22 osm renderd[8237]: DEBUG: DONE TILE default 9 272-279 168-175 in 11664.584 seconds Bye Chris Am 11.11.2011 00:23, schrieb Christian Chelius: Hi, Am 10.11.2011 22:37, schrieb Frederik Ramm: Hi, Are you sure that the daemon is really running under the user tirex (chack ps auxw or so)? osm@osm:~$ ps auxw |grep tirex tirex21023 0.4 13.4 712448 545344 ? SNov08 14:41 mapnik: rendering z=8 x=128 y=88 map=osm tirex21028 0.4 11.5 636832 469984 ? SNov08 13:25 mapnik: rendering z=8 x=136 y=88 map=osm tirex24483 0.1 0.3 54668 13604 ?Ss Nov10 0:05 /usr/bin/perl /usr/bin/tirex-master osm24992 0.0 0.0 7544 836 pts/3D+ 00:29 0:00 grep tirex Looks good. Tirex-master is running under the user tirex. I have also terminated the processes 21023 and 21028. After that I was able to start tirex-backend-manager. osm@osm:~$ ps auxw |grep tirex tirex24483 0.1 0.3 54668 13604 ?Ss Nov10 0:05 /usr/bin/perl /usr/bin/tirex-master tirex25016 1.5 0.2 43820 10852 ?Ss 00:30 0:00 /usr/bin/perl /usr/bin/tirex-backend-manager tirex25017 1.1 0.1 143592 4696 ?D00:30 0:00 mapnik: initializing tirex25018 1.3 0.1 143592 4688 ?D00:30 0:00 mapnik: initializing tirex25019 1.1 0.1 143592 4692 ?D00:30 0:00 mapnik: initializing tirex25020 3.0 0.2 75312 9876 ?S00:30 0:00 test: idle tirex25021 3.4 0.2 75312 9876 ?S00:30 0:00 test: idle osm 25023 0.0 0.0 9608 888 pts/3R+ 00:31 0:00 grep tirex Then I opened the browser and zoomed on the mapbut tirex-backend-manager was killed immediately. If yes, then to be absolutely sure that the permissions of everything are ok, do a su tirex and touch /var/run/tirex/foo. If that works too, then you should strace the daemon startup to find out exactly why it thinks it cannot access the socket. I was asked for a password, so I created one for user tirex (I hope that was right) and did a touch /var/run/tirex/foo. tirex@osm:~$ ls -l /var/run/tirex/ insgesamt 8 -rw-r--r-- 1 tirex tirex 0 11. Nov 01:27 foo srwxrwxrwx 1 tirex tirex 0 11. Nov 01:09 master.sock -rw-r--r-- 1 tirex tirex 6 11. Nov 01:10 tirex-backend-manager.pid -rw-r--r-- 1 tirex tirex 6 11. Nov 01:09 tirex-master.pid Another thing I want you to ask is about thehowto. This isfor 32 bit Debian 5 and installs libboost-program-options1.34.1. I use 64 bit Debian 6 and libboost-program-options1.42.0. Can this explain the error? Bye Chris ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] tirex-backend-manager doesn't work
Hello everybody,
I'm currently trying to install an osm wms server on debian squeeze 64
bit. I already setup postgis, mapnik, mod_tile and apache based on a
good howto. Unfortunately, the tirex-backend-manager will not start. The
process is not displayed by htop. I assume that for this reason only
those tiles are displayed, which have been calculated in advance.
/osm@osm:~$ sudo /etc/init.d/tirex-backend-manager start
osm@osm:~$ sudo /etc/init.d/tirex-backend-manager restart
Restarting Mapnik rendering daemon for Tirex:
tirex-backend-managerstart-stop-daemon: warning: failed to kill 22244:
No such process/
I took Tirex from SVN and built debian packages again. Then the packages
tirex-backend-mapnik_0.2.0_amd64.deb, tirex-core_0.2.0_amd64.deb, and
tirex-example-map_0.2.0_amd64.deb were installed.
Maybe, this output is useful:
/osm@osm:~$ tirex-master -d
tirex-master[22344]: tirex-master started with cmd line options: -d
tirex-master[22344]: Config bucket=[{maxload=20 maxproc=4 minprio=1
name=live},{maxload=8 maxproc=3 minprio=10 name=important},{maxload=4
maxproc=2 minprio=20 name=background}]
tirex-master[22344]: Renderer mapnik: port=9331 procs=3
path=/usr/lib/tirex/backends/mapnik syslog_facility=daemon debug=1
fontdir=/usr/lib64/mapnik/fonts fontdir_recurse=1
plugindir=/usr/lib64/mapnik/input
tirex-master[22344]: Map example: renderer=mapnik
tiledir=/var/lib/tirex/tiles/example zoom=0-4
mapfile=/usr/share/tirex/example-map/example.xml
tirex-master[22344]: Map osm: renderer=mapnik
tiledir=/var/lib/tirex/tiles/osm zoom=0-18
mapfile=/usr/share/osm-mapnik/osm.xml
tirex-master[22344]: Map surveillance: renderer=mapnik
tiledir=/var/lib/tirex/tiles/surveillance zoom=0-18
mapfile=/usr/share/osm-styles/surveillance/common.xml
tirex-master[22344]: Renderer test: port=9330 procs=2
path=/usr/lib/tirex/backends/test syslog_facility=daemon debug=0
tirex-master[22344]: Map test: renderer=test
tiledir=/var/lib/tirex/tiles/test zoom=0-10
Cannot open master UNIX domain socket: Keine Berechtigung/
--
/
osm@osm:~$ tirex-backend-manager -d
tirex-backend-manager[22356]: tirex-backend-manager started with cmd
line options: -d
tirex-backend-manager[22356]: Config bucket=[{maxload=20 maxproc=4
minprio=1 name=live},{maxload=8 maxproc=3 minprio=10
name=important},{maxload=4 maxproc=2 minprio=20 name=background}]
tirex-backend-manager[22356]: Can't open pidfile
'/var/run/tirex/tirex-backend-manager.pid' for writing: Keine Berechtigung
tirex-backend-manager[22356]: Found config for renderers: mapnik test
tirex-backend-manager[22356]: opened port 9331 for renderer 'mapnik'
tirex-backend-manager[22356]: opened port 9330 for renderer 'test'
tirex-backend-manager[22356]: renderer mapnik started with pid 22357
tirex-backend-manager[22356]: renderer mapnik started with pid 22358
tirex-backend-manager[22356]: renderer mapnik started with pid 22359
tirex-backend-mapnik[22357]: Renderer started (name=mapnik)
tirex-backend-manager[22356]: renderer test started with pid 22360
tirex-backend-mapnik[22358]: Renderer started (name=mapnik)
tirex-backend-manager[22356]: renderer test started with pid 22361
tirex-backend-mapnik[22359]: Renderer started (name=mapnik)
tirex-backend-mapnik[22357]: cannot add
/etc/tirex/renderer/mapnik/example.conf: tile directory
'/var/lib/tirex/tiles/example' not accessible
tirex-backend-mapnik[22357]: cannot add
/etc/tirex/renderer/mapnik/osm.conf: tile directory
'/var/lib/tirex/tiles/osm' not accessible
tirex-backend-mapnik[22357]: cannot add
/etc/tirex/renderer/mapnik/surveillance.conf: tile directory
'/var/lib/tirex/tiles/surveillance' not accessible
Cannot load any Mapnik styles
tirex-backend-mapnik[22358]: cannot add
/etc/tirex/renderer/mapnik/example.conf: tile directory
'/var/lib/tirex/tiles/example' not accessible
tirex-backend-manager[22356]: child 22357 terminated (exit_code=1, signal=0)
Use of uninitialized value $name in hash element at
/usr/share/perl5/Tirex/Renderer.pm line 68.
Undefined subroutine main::debug called at
/usr/bin/tirex-backend-manager line 368.
tirex-backend-mapnik[22358]: cannot add
/etc/tirex/renderer/mapnik/osm.conf: tile directory
'/var/lib/tirex/tiles/osm' not accessible
tirex-backend-mapnik[22358]: cannot add
/etc/tirex/renderer/mapnik/surveillance.conf: tile directory
'/var/lib/tirex/tiles/surveillance' not accessible
Cannot load any Mapnik styles
tirex-backend-mapnik[22359]: cannot add
/etc/tirex/renderer/mapnik/example.conf: tile directory
'/var/lib/tirex/tiles/example' not accessible
tirex-backend-mapnik[22359]: cannot add
/etc/tirex/renderer/mapnik/osm.conf: tile directory
'/var/lib/tirex/tiles/osm' not accessible
tirex-backend-mapnik[22359]: cannot add
/etc/tirex/renderer/mapnik/surveillance.conf: tile directory
'/var/lib/tirex/tiles/surveillance' not accessible
Cannot load any Mapnik styles
osm@osm:~$ tirex-backend-test[22360]: Renderer started (name=test)
tirex-backend-test[22360]: map
Re: [OSM-dev] tirex-backend-manager doesn't work
Hi, On 11/10/2011 07:30 PM, Christian Chelius wrote: Maybe, this output is useful: It is indeed: Cannot open master UNIX domain socket: Keine Berechtigung/ tirex-backend-manager[22356]: Can't open pidfile '/var/run/tirex/tirex-backend-manager.pid' for writing: Keine Berechtigung tirex-backend-mapnik[22357]: cannot add /etc/tirex/renderer/mapnik/example.conf: tile directory '/var/lib/tirex/tiles/example' not accessible If the tirex master cannot open the socket, there's no way it can receive rendering commands. If the backend cannot open the tile directory, there's no way it can store tiles. Assuming you are using the default config, make sure that the following directories exist and are writable for the tirex user: /var/lib/tirex /var/lib/tirex/tiles /var/run/tirex This should have been ensured by the debian install process but maybe something went wrong there. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex-backend-manager doesn't work
Hi, thanks for answering. Assuming you are using the default config, The configuration /etc/tirex/tirex.conf has not changed. make sure that the following directories exist and are writable for the tirex user: /var/lib/tirex /var/lib/tirex/tiles /var/run/tirex This should have been ensured by the debian install process but maybe something went wrong there. The directories exist and the permissions should be correct. / /osm@osm:~$ ls -l /var/lib/tirex/ insgesamt 8 srw-rw-rw- 1 tirex tirex0 10. Nov 22:25 modtile.sock drwxr-xr-x 2 tirex tirex 4096 3. Nov 00:41 stats drwxr-xr-x 6 tirex tirex 4096 9. Nov 02:10 tiles/ / osm@osm:~$ ls -l /var/lib/tirex/tiles/ insgesamt 16 drwxr-xr-x 7 tirex tirex 4096 9. Nov 02:30 example drwxr-xr-x 20 tirex tirex 4096 9. Nov 04:38 osm -rw-r--r-- 1 tirex tirex0 9. Nov 04:16 planet-import-complete drwxr-xr-x 9 tirex tirex 4096 3. Nov 08:36 surveillance drwxr-xr-x 3 tirex tirex 4096 4. Nov 01:32 test osm@osm:~$ ls -l /var/run/tirex/ insgesamt 8 srwxrwxrwx 1 tirex tirex 0 10. Nov 22:25 master.sock -rw-r--r-- 1 tirex tirex 6 10. Nov 22:24 tirex-backend-manager.pid -rw-r--r-- 1 tirex tirex 6 10. Nov 22:25 tirex-master.pid But the result is still the same: osm@osm:~$ ps ax |grep tirex 23668 ?Ss 0:00 /usr/bin/perl /usr/bin/tirex-master 23771 pts/3S+ 0:00 grep tirex Bye Chris Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex-backend-manager doesn't work
Hi, Am 10.11.2011 22:37, schrieb Frederik Ramm: Hi, Are you sure that the daemon is really running under the user tirex (chack ps auxw or so)? osm@osm:~$ ps auxw |grep tirex tirex21023 0.4 13.4 712448 545344 ? SNov08 14:41 mapnik: rendering z=8 x=128 y=88 map=osm tirex21028 0.4 11.5 636832 469984 ? SNov08 13:25 mapnik: rendering z=8 x=136 y=88 map=osm tirex24483 0.1 0.3 54668 13604 ?Ss Nov10 0:05 /usr/bin/perl /usr/bin/tirex-master osm24992 0.0 0.0 7544 836 pts/3D+ 00:29 0:00 grep tirex Looks good. Tirex-master is running under the user tirex. I have also terminated the processes 21023 and 21028. After that I was able to start tirex-backend-manager. osm@osm:~$ ps auxw |grep tirex tirex24483 0.1 0.3 54668 13604 ?Ss Nov10 0:05 /usr/bin/perl /usr/bin/tirex-master tirex25016 1.5 0.2 43820 10852 ?Ss 00:30 0:00 /usr/bin/perl /usr/bin/tirex-backend-manager tirex25017 1.1 0.1 143592 4696 ?D00:30 0:00 mapnik: initializing tirex25018 1.3 0.1 143592 4688 ?D00:30 0:00 mapnik: initializing tirex25019 1.1 0.1 143592 4692 ?D00:30 0:00 mapnik: initializing tirex25020 3.0 0.2 75312 9876 ?S00:30 0:00 test: idle tirex25021 3.4 0.2 75312 9876 ?S00:30 0:00 test: idle osm 25023 0.0 0.0 9608 888 pts/3R+ 00:31 0:00 grep tirex Then I opened the browser and zoomed on the mapbut tirex-backend-manager was killed immediately. If yes, then to be absolutely sure that the permissions of everything are ok, do a su tirex and touch /var/run/tirex/foo. If that works too, then you should strace the daemon startup to find out exactly why it thinks it cannot access the socket. I was asked for a password, so I created one for user tirex (I hope that was right) and did a touch /var/run/tirex/foo. tirex@osm:~$ ls -l /var/run/tirex/ insgesamt 8 -rw-r--r-- 1 tirex tirex 0 11. Nov 01:27 foo srwxrwxrwx 1 tirex tirex 0 11. Nov 01:09 master.sock -rw-r--r-- 1 tirex tirex 6 11. Nov 01:10 tirex-backend-manager.pid -rw-r--r-- 1 tirex tirex 6 11. Nov 01:09 tirex-master.pid Another thing I want you to ask is about thehowto. This isfor 32 bit Debian 5 and installs libboost-program-options1.34.1. I use 64 bit Debian 6 and libboost-program-options1.42.0. Can this explain the error? Bye Chris ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex and python rendering backend
It's good to know that it should be basically possible. However, my knowledge of linux communication mechanisms, pearl and python is not good enough to build something similar from scratch, let alone setting up python in an efficient way. So I'd still be looking for an existing python skeleton or some help in setting it up. bye Nop -- View this message in context: http://gis.638310.n2.nabble.com/Tirex-and-python-rendering-backend-tp6667407p6684706.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex and python rendering backend
Am 10.08.2011 00:58, schrieb Philipp Borgers: Perhaps you can take a look at the code http://svn.openstreetmap.org/applications/utils/tirex/lib/Tirex/Backend/Test.pm And don't forget to look at the parent class: http://svn.openstreetmap.org/applications/utils/tirex/lib/Tirex/Backend.pm and the caller: http://svn.openstreetmap.org/applications/utils/tirex/backends/test You'll see that the configuration comes from ENV and the communication is done via an UDP socket, so I too guess it should be possible to use any language with udp and file-io possibilities. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Tirex and python rendering backend
Hi! My server is using tirex and a fairly standard mapnik rendering backend to create map tiles. There are also tiles with elevation contour lines which are pre-rendered offline and uploaded. Not exactly state of the art. I would like to put tirex in control of rendering those contours also. Naturally, they will never expire, but this would make nice hill shading and contours available for all zoom levels, not just those where pre-rendering is feasible. However, the contour tiles cannot be created by mapnik directly as they need certain alpha channel techniques that mapnik does not provide. The rendering in this case is done by a python script with mapnik binding and PIL postprocessing. Is it possible to connect tirex to a python rendering backend? Has anybody experience with this? How would the skeleton of such a python script need to look like? bye Nop -- View this message in context: http://gis.638310.n2.nabble.com/Tirex-and-python-rendering-backend-tp6667407p6667407.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex and python rendering backend
On Tue, 2011-08-09 at 01:56 -0700, NopMap wrote: Hi! My server is using tirex and a fairly standard mapnik rendering backend to create map tiles. There are also tiles with elevation contour lines which are pre-rendered offline and uploaded. Not exactly state of the art. I would like to put tirex in control of rendering those contours also. Naturally, they will never expire, but this would make nice hill shading and contours available for all zoom levels, not just those where pre-rendering is feasible. However, the contour tiles cannot be created by mapnik directly as they need certain alpha channel techniques that mapnik does not provide. The rendering in this case is done by a python script with mapnik binding and PIL postprocessing. Is it possible to connect tirex to a python rendering backend? Has anybody experience with this? How would the skeleton of such a python script need to look like? Perhaps you can take a look at the code http://svn.openstreetmap.org/applications/utils/tirex/lib/Tirex/Backend/Test.pm I would argue it's possible :) bye Nop -- View this message in context: http://gis.638310.n2.nabble.com/Tirex-and-python-rendering-backend-tp6667407p6667407.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] tirex, output from mapnik-rendering backend different from mapnik for GDAL raster source with opacity
In my stylesheet I use some raster images which I combine with OSM data with the screen and multiply method and with opacity values 1. When rendered with tirex and the mapnik rendering backend they do not show up while the exact same stylesheet-file does behave as expected with the normal mapnik (tested with 0.7.1 and 0.7.2 and generate_image / tile). I suspect the reason that they are not rendered lies in the opacity-parameter, because I have other rasters with multiply (and same scaling) and default opacity of 1 that do render well. this is the style: Style name=bathymetry_m Rule MinScaleDenominator1/MinScaleDenominator MaxScaleDenominator2500/MaxScaleDenominator RasterSymbolizer CssParameter name=modemultiply/CssParameter CssParameter name=scalingbilinear/CssParameter CssParameter name=opacity0.3/CssParameter /RasterSymbolizer /Rule /Style and the layer: Layer name=bathymetry_m srs=+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs +over StyleNamebathymetry_m/StyleName Datasource Parameter name=fileOB_LR/OB_LR_multiply_.tif/Parameter Parameter name=formattiff/Parameter Parameter name=typegdal/Parameter /Datasource /Layer cheers, Martin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex Bucket Configuration
Frederik Ramm wrote: I'm surprised that could happen; even if the master was frozen, then the most that should happen is that every mod_tile attempt to speak to the master would fail, but normal Apache operations (not talking to master) should continue normally. I have no idea how it happens, but the apache server becomes completely unresponsive to any request (even simple html pages) until both tirex and apache are restarted. bye Nop -- View this message in context: http://gis.638310.n2.nabble.com/Tirex-Bucket-Configuration-tp6477485p6482208.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Tirex Bucket Configuration
We've been using mod_tile/renderd for approx. a year serving an australasian tile set, we will only be serving this one tile set for the foreseeable future. We're now looking at rebuilding our tile servers and I have set up a test installation running Tirex. I'm wondering if anybody is able to provide me with any guidelines on how to configure the buckets. Is there a relationship between the maxproc setting and the number of CPUs available on the host for instance? Cheers Adam ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex Bucket Configuration
Adam, Adam Ratcliffe wrote: I'm wondering if anybody is able to provide me with any guidelines on how to configure the buckets. Is there a relationship between the maxproc setting and the number of CPUs available on the host for instance? The ideal setting depends on a number of factors. If the server is used for anything else in addition to serving tiles, you'd probably want the maxproc setting in the highest priority class to be a little lower than the number of cores, so you still have a spare core to do other things even if the server is under full load rendering tiles. The lower priority classes depend on what mechanism you want to use to update your tiles. I don't normally bother doing tile expiry; I just have a cronjob running that stuffs the oldes 10k tiles into the queue for re-rendering once per hour. This means that my low-priority queue is *always* full, so tirex will *always* keep maxproc CPUs of the lowest priority queue busy. This means that I don't usually set that to anything more than 4 even on a 16-core machine. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex Bucket Configuration
On Wed, Jun 15, 2011 at 06:09:13PM +1200, Adam Ratcliffe wrote: We've been using mod_tile/renderd for approx. a year serving an australasian tile set, we will only be serving this one tile set for the foreseeable future. We're now looking at rebuilding our tile servers and I have set up a test installation running Tirex. I'm wondering if anybody is able to provide me with any guidelines on how to configure the buckets. Is there a relationship between the maxproc setting and the number of CPUs available on the host for instance? All of this of course depends on the details of your hardware, the map data you have, the number of tiles, the access pattern etc. But here are some points to get you started: * You probably don't want to have too many rendering processes even if you have many CPUs, because then your bottleneck will be the database disk access. * You also, of course, don't want to few rendering processes, because you don't want an idle machine. * You should not use all capacities in the low priority buckets, so that there are always capacities available if high priorities come in. * For the highest priority bucket, your maxproc should probably be slightly higher than the number of CPUs you have. Slightly higher because some processes will be waiting for IO. Maxload about twice that (1 for the renderer process, 1 one for the datatabase process). * For all other buckets use less. Numbers depend on your ratio of interactively rendered tiles vs. background-rendered tiles. If you want snappier access for live rendering give low-priority buckets smaller maxproc/maxload, but background-rendering will be slower. Start with those ideas and then watch your system and change settings accordingly. Look at the jobs.log logfile for rendering times and success rate. Experiment with different settings. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex Bucket Configuration
Thanks Jochen and Frederik for these tips - that should be plenty to get me started :) Cheers Adam On Wed, Jun 15, 2011 at 7:33 PM, Jochen Topf joc...@remote.org wrote: On Wed, Jun 15, 2011 at 06:09:13PM +1200, Adam Ratcliffe wrote: We've been using mod_tile/renderd for approx. a year serving an australasian tile set, we will only be serving this one tile set for the foreseeable future. We're now looking at rebuilding our tile servers and I have set up a test installation running Tirex. I'm wondering if anybody is able to provide me with any guidelines on how to configure the buckets. Is there a relationship between the maxproc setting and the number of CPUs available on the host for instance? All of this of course depends on the details of your hardware, the map data you have, the number of tiles, the access pattern etc. But here are some points to get you started: * You probably don't want to have too many rendering processes even if you have many CPUs, because then your bottleneck will be the database disk access. * You also, of course, don't want to few rendering processes, because you don't want an idle machine. * You should not use all capacities in the low priority buckets, so that there are always capacities available if high priorities come in. * For the highest priority bucket, your maxproc should probably be slightly higher than the number of CPUs you have. Slightly higher because some processes will be waiting for IO. Maxload about twice that (1 for the renderer process, 1 one for the datatabase process). * For all other buckets use less. Numbers depend on your ratio of interactively rendered tiles vs. background-rendered tiles. If you want snappier access for live rendering give low-priority buckets smaller maxproc/maxload, but background-rendering will be slower. Start with those ideas and then watch your system and change settings accordingly. Look at the jobs.log logfile for rendering times and success rate. Experiment with different settings. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex Bucket Configuration
Frederik Ramm wrote: I don't normally bother doing tile expiry; I just have a cronjob running that stuffs the oldes 10k tiles into the queue for re-rendering once per hour. This means that my low-priority queue is *always* full, so tirex will *always* keep maxproc CPUs of the lowest priority queue busy. On a side note: I have always had the problem that tirex on my server crashes on occasion (about once a month) while adding a few 1000 new tiles for rendering to a non-empty queue. If you keep your low-prio queue usually full, do you encounter such crashes, too, or was there a fix at some time? I'm still looking for a solution as the crash is pretty bad, taking down the whole apache server. bye Nop -- View this message in context: http://gis.638310.n2.nabble.com/Tirex-Bucket-Configuration-tp6477485p6479551.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex Bucket Configuration
Hi, On 06/15/11 18:18, NopMap wrote: On a side note: I have always had the problem that tirex on my server crashes on occasion (about once a month) while adding a few 1000 new tiles for rendering to a non-empty queue. I had that earlier; for some reason tirex-batch and the master process managed to get stuck talking to each other. r23241 fixed it for me but I guess you already have that (September 2010). I'm still looking for a solution as the crash is pretty bad, taking down the whole apache server. I'm surprised that could happen; even if the master was frozen, then the most that should happen is that every mod_tile attempt to speak to the master would fail, but normal Apache operations (not talking to master) should continue normally. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex with custom tileschema
On 4/11/2011 7:22 PM, Dane Springmeyer wrote: It works now, except that it is in the wrong projection :) I've to look for a different solution :( Let us know what you come up with! Now I'm using MapProxy with the new (experimental) Mapnik backend. And run a seed which refreshes tiles older than a week *), combined with the usual replicating database as per http://wiki.openstreetmap.org/wiki/Minutely_Mapnik The result is not a perfect minutely updating tileserver, but a reasonable up to date server which is fine for my needs. By only rendering .nl the total rendertime to refresh everything is workable. Steven *) That's initial, I've to play a bit with it to see what works best. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] tirex with custom tileschema
Hi all, Does Tirex support different projections and tileschema's? I'm trying to get Tirex running with the dutch tiling schema*). I've got Mapnik and TileCache working together to display my tiles, so the osm.xml is correct. However since TileCache doesn't have any update strategy and I do want to keep the tiles reasonably current so I'm looking into Tirex. I'm using a modified tileserver.js on node.js which I've gotten from Peter/ /Körner to access Tirex. However when I request tiles I get a HTTP 500 Internal server error and the X-Error header says 'render error'. I can't seem to find any more detailed information on this error. The only thing I've seen is from tileserver.js: tirex nodets-32069-1210 tirex nodets-32069-1210 GET /tiles/osm/2/3/1.png - 500 Internal Server Error (render error) And in /var/log/tirex/jobs.log 2011-04-11T12:14:09 id=1302516849_33668080 map=osm x=0 y=0 z=2 prio=10 request_time=1302516849 expire= sources=C render_time=0 success= Which to me means that tileserver.js does send the request to Tirex, but Tirex is unable to render it, which brings me to my initial question: can Tirex render non spherical mercator? Regards, Steven *) tilecache config: [osm_nl] type=Mapnik mapfile=/home/administrator/osm/mapnik/osm.xml spherical_mercator=false resolutions=3440.640,1720.320,860.160,430.080,215.040,107.520,53.760,26.880,13.440,6.720,3.360,1.680,0.840,0.420,0.210 metatile=yes bbox=-285401.920, 22598.080, 595401.92, 903401.920 srs=EPSG:28992 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex with custom tileschema
Hi Steven, Am 11.04.2011 12:24, schrieb Steven M. Ottens: I'm using a modified tileserver.js on node.js which I've gotten from Peter//Körner to access Tirex. However when I request tiles I get a HTTP 500 Internal server error and the X-Error header says 'render error'. really nice to hear somebody using the Tileserver :) To get debug-info from tirex it's best to start it in debug mode: first disable the running serive using /etc/init.d/tirex-backend-manager stop then start your own debug instance sudo -u tirex tirex-backend-manager --debug then restart the master (use another shell session) /etc/init.d/tirex-master restart now take a look at the running debug instance while requesting tiles. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex with custom tileschema
On Mon, Apr 11, 2011 at 12:24:38PM +0200, Steven M. Ottens wrote: Does Tirex support different projections and tileschema's? I'm No. The Google projection and tiling schema and the metatile schema used in OSM is very much hard-baked into Tirex. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex with custom tileschema
On Mon, Apr 11, 2011 at 02:31:19PM +0200, Steven M. Ottens wrote: On 4/11/2011 2:01 PM, Jochen Topf wrote: On Mon, Apr 11, 2011 at 12:24:38PM +0200, Steven M. Ottens wrote: Does Tirex support different projections and tileschema's? I'm No. The Google projection and tiling schema and the metatile schema used in OSM is very much hard-baked into Tirex. Thanks, that explains why I get 'google' tiles, even when osm.xml is in a different projection. I hadn't looked into this behaviour since I was still struggling with this error: tirex-backend-mapnik[2309]: Mapnik config error: Unable to find specified font face '' Is the tirex-backend-mapnik somehow using different settings files than mapnik? Since both osm.xml and fontset-settings.xml.inc are correct for the mapnik/tilecache setup. It uses the config file configured in /etc/tirex/mapnik/... or whatever you told it to use. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex with custom tileschema
Am 11.04.2011 14:31, schrieb Steven M. Ottens: On 4/11/2011 2:01 PM, Jochen Topf wrote: On Mon, Apr 11, 2011 at 12:24:38PM +0200, Steven M. Ottens wrote: Does Tirex support different projections and tileschema's? I'm No. The Google projection and tiling schema and the metatile schema used in OSM is very much hard-baked into Tirex. Thanks, that explains why I get 'google' tiles, even when osm.xml is in a different projection. I hadn't looked into this behaviour since I was still struggling with this error: tirex-backend-mapnik[2309]: Mapnik config error: Unable to find specified font face '' The font-path configured in the tirex-renderer-config-file (/etc/tirex/mapnik.conf) is not your default system path. Change the config file to point to the font-path of your system (search for ttf-files ;) Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex with custom tileschema
On 4/11/2011 4:50 PM, Peter Körner wrote: Am 11.04.2011 14:31, schrieb Steven M. Ottens: still struggling with this error: tirex-backend-mapnik[2309]: Mapnik config error: Unable to find specified font face '' The font-path configured in the tirex-renderer-config-file (/etc/tirex/mapnik.conf) is not your default system path. Change the config file to point to the font-path of your system (search for ttf-files ;) Ah thanks, the problem was indeed that in /etc/tirex/renderer/mapnik.conf 'fontdir' was pointing to the wrong directory and 'fontdir_recurse' was not set to 1 It works now, except that it is in the wrong projection :) I've to look for a different solution :( Steven ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex with custom tileschema
On Apr 11, 2011, at 8:23 AM, Steven M. Ottens wrote: On 4/11/2011 4:50 PM, Peter Körner wrote: Am 11.04.2011 14:31, schrieb Steven M. Ottens: still struggling with this error: tirex-backend-mapnik[2309]: Mapnik config error: Unable to find specified font face '' btw, sorry about ^^, I consider this poor error a mapnik bug and have flagged to fix before the next release: http://trac.mapnik.org/ticket/596#comment:1 The font-path configured in the tirex-renderer-config-file (/etc/tirex/mapnik.conf) is not your default system path. Change the config file to point to the font-path of your system (search for ttf-files ;) Ah thanks, the problem was indeed that in /etc/tirex/renderer/mapnik.conf 'fontdir' was pointing to the wrong directory and 'fontdir_recurse' was not set to 1 Also, minor detail, but since Tirex was released I have added a native recursive font registration function, `register_fonts`, to Mapnik2, and backported to Mapnik 0.7.2: http://trac.mapnik.org/browser/trunk/src/font_engine_freetype.cpp#L88 It works now, except that it is in the wrong projection :) I've to look for a different solution :( Let us know what you come up with! Steven ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex freezes when adding many tiles with tirex-batch
Hi, On 01/28/11 18:58, NopMap wrote: I am using tirex-batch to add large numbers of tiles (15000) to the rendering queue for pre-rendering of tiles. Usually this works nicely, but occasionally tirex freezes completely. There's no rendering being done and tirex-status shows unchanging numbers. The freeze always happens while tirex-batch is running and adding tiles. If it does not freeze during creation of render requests, then all requests are executed without a problem. I had that once and found it was a problem where somehow the master could not message back to tirex-batch that it had completed a tile. I thought I had fixed the problem in r23241. I believe it mostly happened in a scenario where you would input one batch of tiles with tirex-batch, then when that has terminated and before the queue is processed you start a second batch. Maybe, but I'm unsure with that, the problem occurs if the second batch contains a tiles that was already in the first or so. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex freezes when adding many tiles with tirex-batch
Hi! Frederik Ramm wrote: I had that once and found it was a problem where somehow the master could not message back to tirex-batch that it had completed a tile. I thought I had fixed the problem in r23241. I believe it mostly happened in a scenario where you would input one batch of tiles with tirex-batch, then when that has terminated and before the queue is processed you start a second batch. Maybe, but I'm unsure with that, the problem occurs if the second batch contains a tiles that was already in the first or so. I am using r24508, so that change should be included. It is well possible it is the same problem, there are multiple calls to tirex-batch in a script file. Do you think putting a small sleep between them might work around the problem? bye Nop -- View this message in context: http://gis.638310.n2.nabble.com/Tirex-freezes-when-adding-many-tiles-with-tirex-batch-tp5970695p5971201.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex freezes when adding many tiles with tirex-batch
Hi, NopMap wrote: I am using r24508, so that change should be included. It is well possible it is the same problem, there are multiple calls to tirex-batch in a script file. Do you think putting a small sleep between them might work around the problem? If the problem is still what it was back then, then two things are likely to solve it: (1) collect all input and run only one tirex-batch job that feeds *everything* into the queue. or (2) wait with the next tirex-batch call until the queue size is zero (relatively easy to determine with tirex-status). Ideally of course we'd isolate and fix the bug but as you said, it is elusive. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - Rendering jobs stucked in the queue
I'm not sure about a better place for this so I put it here: http://wiki.openstreetmap.org/wiki/Tirex/Logging -- Ing. Radek Bartoň Faculty of Information Technology Department of Computer Graphics and Multimedia Brno University of Technology E-mail: black...@post.cz Web: http://blackhex.no-ip.org Jabber: black...@jabber.cz ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - Rendering jobs stucked in the queue
Any other thoughts what may cause it or how to debug where is the problem, please? -- Ing. Radek Bartoň Faculty of Information Technology Department of Computer Graphics and Multimedia Brno University of Technology E-mail: black...@post.cz Web: http://blackhex.no-ip.org Jabber: black...@jabber.cz ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - Rendering jobs stucked in the queue
On 16.01.2011 23:15, Radek Bartoň wrote: Any other thoughts what may cause it or how to debug where is the problem, please? You don't have renderd running in parallel to tirex, right? Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - Rendering jobs stucked in the queue
On Sun, 2011-01-16 at 23:15 +0100, Radek Bartoň wrote: Any other thoughts what may cause it or how to debug where is the problem, please? Are you sure it isn't a tile which is hitting the timeout? There was a bug some time ago where the Agg code can get stuck in an infinite loop rendering some tiles but it only occurs in very rare circumstances. Have you tried starting up a fresh instance and just feeding in the tile co-ordinates of that one stuck request? Jon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - Rendering jobs stucked in the queue
You don't have renderd running in parallel to tirex, right? No, there is just mod_tile installed. Are you sure it isn't a tile which is hitting the timeout? There was a bug some time ago where the Agg code can get stuck in an infinite loop rendering some tiles but it only occurs in very rare circumstances. Have you tried starting up a fresh instance and just feeding in the tile co-ordinates of that one stuck request? I tired to stop apache2, tirex-master, tirex-backend-manager services, then delete all meta tiles and then start the services in order: tirex-backend- manager, tirex-master, apache2 (I suppose this is the correct one) and then I requested a single tile from a high level (14-17) sevral times with different tiles. From what I've seen while debugging it it doesn't matter on the tile requested. It stucks anytime and even there are some success=1 lines in the jobs.log, the metatiles are all empty. Isn't it possible that all this is caused by some problem with mod_deflate presence since it is the only difference between my working server and this new one, I'm aware of, except of hardware? -- Ing. Radek Bartoň Faculty of Information Technology Department of Computer Graphics and Multimedia Brno University of Technology E-mail: black...@post.cz Web: http://blackhex.no-ip.org Jabber: black...@jabber.cz ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - Rendering jobs stucked in the queue
Hi, On 01/16/11 23:57, Radek Bartoň wrote: From what I've seen while debugging it it doesn't matter on the tile requested. It stucks anytime and even there are some success=1 lines in the jobs.log, the metatiles are all empty. I suggest you configure the tirex test backend which should render tiles with a test pattern on them. That will allow you to determine if your problem is somewhere on the Mapnik side, or on the tile producing side. Also check if you have the symlink: $ ls -l /var/lib/mod_tile lrwxrwxrwx 1 root root 21 2010-08-29 13:52 /var/lib/mod_tile - /var/lib/tirex/tiles/ Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Tirex - Rendering jobs stucked in the queue
Hello everyone. I have a working setup of mod_tile+Tirex+Mapnik rendering server on one machine and recently I've tryied to setup another one on much more powerfull machine. I'm alost sure I've done exactly same configuration (except some paths are different) and I've checked if all the files/directories needed are existing and with proper permissions and that Mapnik is able to render the tiles with my styles from generate_tiles.py. The problem is that rendering jobs comming from mod_tile are enqeued in tirex- master rendering qeueu (tirex-status -eo shows them) but they are not processed and after they timeout after a long time period, the cached tile is empty. IMHO the problem is somewhere in tirex-master - tirex-backend-manager communication. I'm racking my brain with this almost a week so please give me some thoughts what can be wrong? Thank you. --- tirex-status -oe: Tirex Master Status (updated=2011-01-14 10:57:39) Master server: started=2011-01-13 16:08:28 pid=27222 Statistics: count_error=0 count_expired=0 count_rendered[tracks]=0, 0, 0, 0, 0, 0, 0, 6 count_requested=334 count_timeouted=324 sum_render_time[tracks]=0, 0, 0, 0, 0, 0, 0, 527 Queue: Prio Size Maxsize Age 1 17 95 53:24-75:41 all 17 95 53:24-75:41 Buckets: (load=0) Name Priority Rendering MaxRend Maxload Active Can Queued Age live 1- 9 44 20yes no 17 53:24-75:41 important 10- 19 038yes no 0 background20- 024yes no 0 Currently rendering: (num=4) Map X Y Z Prio Age tiles 136 88 81 371 tracks 1104696 111 371 tracks 552344 101 371 tiles 136 80 81 191 Renderers: NamePort Procs Path mapnik 9331 3 /usr/lib/tirex/backends/mapnik test9330 2 /usr/lib/tirex/backends/test wms 9332 2 /usr/lib/tirex/backends/wms Maps: Name RendererZoom Tiledir demowms wms 0-10 /var/lib/tirex/tiles/demowms example mapnik 0- 4 /var/lib/tirex/tiles/example test test0-10 /var/lib/tirex/tiles/test tilesmapnik 0-20 /var/lib/tirex/tiles/tiles tracks mapnik 0-20 /var/lib/tirex/tiles/tracks tracks-debug mapnik 0-20 /var/lib/tirex/tiles/tracks-debug --- /var/log/apache2/error.log: [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_translate: uri(/tracks/7/64/40.png), referer: http://opentrackmap.no-ip.org/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_storage_hook: handler(tile_serve), uri(/tracks/7/65/40.png), filename(/var/lib/mod_tile/tracks/7/0/0/0/66/8.meta), path_info((null)), referer: http://opentrackmap.no-ip.org/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_translate: baseuri(/test/) name(test), referer: http://opentrackmap.no-ip.org/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_translate: baseuri(/tiles/) name(tiles), referer: http://opentrackmap.no-ip.oqrg/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_translate: baseuri(/tracks/) name(tracks), referer: http://opentrackmap.no-ip.org/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_translate: op(tile_serve) xml(tracks) z(7) x(64) y(40), referer: http://opentrackmap.no- ip.org/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_storage_hook: handler(tile_serve), uri(/tracks/7/64/40.png), filename(/var/lib/mod_tile/tracks/7/0/0/0/66/8.meta), path_info((null)), referer: http://opentrackmap.no-ip.org/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_translate: uri(/tracks/7/63/40.png), referer: http://opentrackmap.no-ip.org/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_translate: baseuri(/test/) name(test), referer: http://opentrackmap.no-ip.org/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_translate: baseuri(/tiles/) name(tiles), referer: http://opentrackmap.no-ip.org/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_translate: baseuri(/tracks/) name(tracks), referer: http://opentrackmap.no-ip.org/ [Thu Jan 13 16:08:48 2011] [info] [client 147.229.13.140] tile_translate: op(tile_serve) xml(tracks) z(7) x(63) y(40), referer: http://opentrackmap.no- ip.org/ [Thu Jan 13 16:08:48 2011] [info]
Re: [OSM-dev] Tirex - Rendering jobs stucked in the queue
On Fri, Jan 14, 2011 at 11:14:30AM +0100, Radek Bartoň wrote: I have a working setup of mod_tile+Tirex+Mapnik rendering server on one machine and recently I've tryied to setup another one on much more powerfull machine. I'm alost sure I've done exactly same configuration (except some paths are different) and I've checked if all the files/directories needed are existing and with proper permissions and that Mapnik is able to render the tiles with my styles from generate_tiles.py. The problem is that rendering jobs comming from mod_tile are enqeued in tirex- master rendering qeueu (tirex-status -eo shows them) but they are not processed and after they timeout after a long time period, the cached tile is empty. IMHO the problem is somewhere in tirex-master - tirex-backend-manager communication. I'm racking my brain with this almost a week so please give me some thoughts what can be wrong? Sometimes there are tiles that take longer to render than the timeout for the renderer. Then the renderer will be killed before the tile is rendered and the tile will remain in the queue. If the slow rendering was because of some temporary reason, it will render faster the next time and everything is ok. But if the rendering is always slow it will try again and again and always fail. You can change the rendering timeout in the configuration, but it probably means your system is to slow. I ahven't looked at your logs, but if this happens you should get a message in your log about the timeout. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex problem creating Debian/Ubuntu packages under Ubuntu 10.04
Am 21.11.2010 12:36, schrieb Frederik Ramm: Hi, Peter Körner wrote: If you want a brand-new breaking-edge mapnik svn version, check this HowTo (german): http://wiki.openstreetmap.org/wiki/DE:HowTo_Mapnik_%26_Tirex This is a very good howto but you shouldn't claim that 0.7.1 is a brand new bleeding edge mapnik svn version - that would be 0.8.0 a.k.a. Mapnik2, and a hell of a lot more difficult to get to work on older systems. Sorry, you're right. I'd love to see a version of this how to with mapnik 0.8.0 on a debian unstable or testing system. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex problem creating Debian/Ubuntu packages under Ubuntu 10.04
Peter Körner osm-li...@mazdermind.de wrote: Sorry, you're right. I'd love to see a version of this how to with mapnik 0.8.0 on a debian unstable or testing system. 0.8 has not yet been releasead AFAIK and would require major changes in our styles anyway. Debian testing has 0.7.1 Sven -- The only thing we have to fear is fear itself (Franklin D. Roosevelt) /me is gig...@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex problem creating Debian/Ubuntu packages under Ubuntu 10.04
Hi, Peter Körner wrote: If you want a brand-new breaking-edge mapnik svn version, check this HowTo (german): http://wiki.openstreetmap.org/wiki/DE:HowTo_Mapnik_%26_Tirex This is a very good howto but you shouldn't claim that 0.7.1 is a brand new bleeding edge mapnik svn version - that would be 0.8.0 a.k.a. Mapnik2, and a hell of a lot more difficult to get to work on older systems. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex problem creating Debian/Ubuntu packages under Ubuntu 10.04
Am 19.11.2010 09:38, schrieb Sven Geggus: M∡rtin Koppenhoeferdieterdre...@gmail.com wrote: yes, I did this because I don't know how to get a proper package for mapnik, thank you. This should be as simple as apt-get install libmapnik-dev If you want a brand-new breaking-edge mapnik svn version, check this HowTo (german): http://wiki.openstreetmap.org/wiki/DE:HowTo_Mapnik_%26_Tirex Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex problem creating Debian/Ubuntu packages under Ubuntu 10.04
M∡rtin Koppenhoefer dieterdre...@gmail.com wrote: yes, I did this because I don't know how to get a proper package for mapnik, thank you. This should be as simple as apt-get install libmapnik-dev Sven -- Der normale Bürger ist nicht an der TU Dresden und schreibt auch nicht mit mutt. (Ulli Kuhnle in de.comp.os.unix.discussion) /me is gig...@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] tirex problem creating Debian/Ubuntu packages under Ubuntu 10.04
I tried to install tirex, but make deb stops at a certain point: ... dh_link dh_strip dh_compress dh_fixperms dh_installdeb dh_shlibdeps dpkg-shlibdeps: Fehler: Keine Abhängigkeitsinformationen für /usr/local/lib/libmapnik.so.0.7 (verwendet von debian/tirex-backend-mapnik/usr/lib/tirex/backends/mapnik) gefunden. dh_shlibdeps: dpkg-shlibdeps -Tdebian/tirex-backend-mapnik.substvars debian/tirex-backend-mapnik/usr/lib/tirex/backends/mapnik returned exit code 2 make: *** [binary-arch] Fehler 9 dpkg-buildpackage: Fehler: fakeroot debian/rules binary gab Fehler-Exitstatus 2 debuild: fatal error at line 1340: dpkg-buildpackage -rfakeroot -D -us -uc -I failed make: *** [deb] Fehler 29 there is a link /usr/local/lib/libmapnik.so.0.7 pointing to libmapnik.so.0.7.1 mo...@acermartin:~$ ls -l /usr/local/lib/ lrwxrwxrwx 1 root root 16 2010-10-11 19:09 libmapnik.so - libmapnik.so.0.7 lrwxrwxrwx 1 root root 18 2010-10-11 19:09 libmapnik.so.0.7 - libmapnik.so.0.7.1 -rwxr-xr-x 1 root root 2033792 2010-10-11 18:59 libmapnik.so.0.7.1 Do you know what could be the problem? cheers, Martin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex problem creating Debian/Ubuntu packages under Ubuntu 10.04
Hi, M∡rtin Koppenhoefer wrote: Do you know what could be the problem? Next time you post a problem to an international list, it will certainly help to do a export LANG=C before you run the command, that would make the error messages appear in English. Your probelm seems to be that you are trying to build a Debian package for Tirex, but you have installed Mapnik outside the Debian packaging mechanism (i.e. with scons install or something like that). Either install Tirex in the same way (make install), or remove your Mapnik installation and replace it by one that comes from a proper package. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] tirex problem creating Debian/Ubuntu packages under Ubuntu 10.04
2010/11/18 Frederik Ramm frede...@remote.org: thank you for your help. Next time you post a problem to an international list, it will certainly help to do a export LANG=C before you run the command, that would make the error messages appear in English. Thanks, I was aware that German error messages wouldn't be too helpful for many of the readers here, but I wasn't aware of how to change this (I am using Linux for roughly a year now, so I'd say I'm still almost a newbie). Your probelm seems to be that you are trying to build a Debian package for Tirex, but you have installed Mapnik outside the Debian packaging mechanism (i.e. with scons install or something like that). yes, I did that some time ago... Either install Tirex in the same way (make install) yes, I did this because I don't know how to get a proper package for mapnik, thank you. Maybe not the best way, but as soon as I learned more about package management, I'll probably change this. Cheers, Martin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Tirex
Hi! I just finished my slides for the Tirex lightning talk that I hope to give at SOTM. Not everybody is coming to SOTM, so I thought I might highlight some things here: Tirex is a tile rendering system. So its sits behind a web server and makes sure map tiles are rendered when they are needed. It is a much more flexible replacement for renderd. It uses the same metatile file format and filesystem layout as renderd so its easy to switch. Tirex is very configurable. You can have as many different map styles as you want and different backends to render these maps. The most often used backend is, of course, Mapnik, but there is also a test backend provided (helps to verify that your installation works without having to install all the Mapnik stuff). There is also a new WMS backend that will leave the rendering to a WMS server. With this you can use Tirex as an alternative to TileCache. You can configure how many rendering processes of the different kinds you want and how they should be used for higher priority live requests and lower priority background requests. This allows you to keep your machine working on re-rendering old tiles while not blocking live requests. Tirex comes with some nifty tools such as the tirex-status program which allows you to see what Tirex is currently doing, how large the queues are etc. There is a batch rendering tool that you can use to (re-)render tiles for some map, zoom range, and bbox with a specified priority. You can tell it to only render non-existing tiles, or only tiles older than a certain age etc. There are several scripts to create statistics from log files and the tile directory and even a script to create nice maps showing your where you have tiles or how old they are. There are plugins for Munin and Nagios. Tirex has been used for a while now on the tile.openstreetmap.de tile server and some others and I consider it quite stable. The biggest missing point is the link between the web server and Tirex. Currently the mod_tile module developed for use with renderd is also used for Tirex. Tirex has some special backwards compatibility code for that. But we are already working on a mod_tirex that will make this part more flexible, too. There is extensive documentation at http://wiki.openstreetmap.org/wiki/Tirex and there are man pages for all the commands. Of course all of this is Open Source, you can find the source in the OSM subversion repository. If you are running your own tile server or plan to do it, I encourage you to give it a try. Questions, bug reports and patches are welcome, be it on this list or at SOTM. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex vs Renderd
Hello, I am part of the Geopard project adverstised here some time ago. This project is about indoor mapping + indoor wifi geopositioning on any devices, while relying on the OSM stack. During the dev of a first prototype, I had 2 major issues related to mod_tile/renderd current designimplementation : 1) mod_tile renderd's max zoom level is hardcoded at 17, while we want ~22 or maybe more. I would like to have this number 1) not harcoded, or codable only in a single config.h-like file or a single .conf file loaded at startup. Or even more flexible, have the max zoom level settable with unix/internet socket packets. In discussions here about this max zoom level, I had raised and a developer agreed that mod_tile's current tile-expiring management was implemented with a hacky C bit array instead where 1 bit = 1 tile, r/w with a clever calculation. That array would be 2GB wide for max_zoom=22, so we concluded that a graph-based implementation would be better for Geopard. I am spotting two conversations on the max zoom level here : http://www.mail-archive.com/t...@openstreetmap.org/msg23769.html (in which I did not take part) and there : http://www.mail-archive.com/dev@openstreetmap.org/msg10266.html (which I had started). 2) in indoor tiles rendering, we have a bunch of vectors tagged with building=yes, name=uniqueBuildingName and level=INTEGER. When I want to render only floor N of building uniqueBuildingName, I would successfuly HTTP the TMS scheme http://domain/tiles/uniqueBuildingName_levelN/z/x/y.png(which is ugly). In the background, mod_tile would see... ok you want layer uniqueBuildingName_levelN and use the appropriate single-file stylesheet for handing renderd its jobs. Let me stress, that as for the wikipedia project multi-lingual issue with non-dynamic/on-the-fly stylesheet customization, this would force in an indoor mapping use case to have ~ NumberOfExistingBuildings*AverageNumberOfLevelsPerBuilding hand-generated stylesheets. In our case, we need a per-building-per-level stylesheet to have some of its PostgreSQL vector selection query changed dynamically out of a single template (select walls from lines where building=yes and name=[dynamically set by renderd] and level=[dynamically set by renderd]). In the case of our project (indoor mapping indoor wifi geopositionning), we have not got far yet but this is obvious that : a) we'd need patching and keeping up with upstream changes (=forking...) both both: - mod_tile (max zoom level dynamic non-hardcoding, tile-expiry graph implementation, nicer/customizable URL scheme) and - renderd (max zoom level dynamic coding, dynamic stylesheet generation) b) why not, creating a mapnik plug-in for the dynamic stylesheet generation issue c) using implementations of mod_tile renderd -like applications written in dynamic languages (thus Tirex, and renderd's python implementation) to prototype things more quickly and easily. So.. If I can sum up my yet unmet requirements regarding mod_tile/renderd, those are : - easy on the fly max zoom level change without any application restart - easy URL scheme customization - easy way to implement middleware code that allows pre-rendering stylesheet transformations thanks to input parameters received such as URL scheme parameters and other disk files/database stored info As far as indoor mapping is concerned, speed is not a big concern for me, so not staying with C-coded applications is not an issue for me. Thanks very much for our thoughts and decision to love ourselves and each other better, Jonathan-David Schröder Geopard Team http://geoparden.wordpress.com On Sat, Apr 17, 2010 at 8:35 PM, Frederik Ramm frede...@remote.org wrote: Hi, Richard Weait wrote: On Sat, Apr 17, 2010 at 7:59 AM, Jochen Topf joc...@remote.org wrote: There are many ways we could approach this: One would be to implement something in Mapnik so that it can allow variables in style files. Then you would only have one style file for all languages and somehow feed the language in separately. That sounds like a great idea. Please see the New Expressions Engine portion of the Mapnik2 announcement. You might like it. ;-) http://mapnik.org/news/2009/dec/ That could be a good basis for what Jochen suggested; the missing link would be the ability to inject variables from the outside when rendering, i.e. before you call the renderer's apply() method in Mapnik you would have to do something like render.setVariable(key, value) and then you could have a rule in the style file that, dending on the value, would create a different rendering outcome. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex vs Renderd
On Mon, Apr 19, 2010 at 05:26:18PM +0200, Jonathan-David SCHRODER wrote: So.. If I can sum up my yet unmet requirements regarding mod_tile/renderd, those are : - easy on the fly max zoom level change without any application restart Tirex currently allows any zoom level, if you ask for zoom level 3172 it will do that. I am working on making the zoom level range configurable, so Tirex will check this in the future. There is a limit of what can be done sensibly with the current metatile filename scheme that needs to be addressed at some point. - easy URL scheme customization Can be done with mod_rewrite or so. - easy way to implement middleware code that allows pre-rendering stylesheet transformations thanks to input parameters received such as URL scheme parameters and other disk files/database stored info Thats going to be difficult. I am not sure the whole tile rendering and caching stuff makes sense for an environment with possibly hundreds of map styles. Maybe you can just use UMN mapserver and WMS with a small PHP mapscript wrapper. That can do all that. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Tirex vs Renderd
Hello, with the recent appearance of tirex there are now at least two, with the python version of renderd 3, but probably more implementations of a rendering backend. All attempt to roughly do the same, i.e. take requests from a webserver or other program and render them. So the question is what is the future outlook for each and the relation between them? I.e. is tirex supposed to replace renderd and tile.osm.org will eventually switch? Or was tirex just something geofabrik threw together for a particular client and renderd should remain the default? Are they intended to solve the same problem and compete for the same market or does each have their own niche for which they are optimized for? This is less of a question though regarding using renderd vs tirex but more of a question of where the future lies and thus for which to develop new features. It would be a shame and rather inefficient to put in effort to develop features for something that turns out to be a dead end and my concern is to fracture the not directly large pool of developers for renderd/tirex. Yes, sometimes competition can be a good thing and thanks to it all being opensource one can fork and merge at will, but I would be interested to hear how others involved see the situation and help potential contributors to make a informed decision to which project best to contribute to. Kai ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex vs Renderd
Kai, Kai Krueger wrote: This is less of a question though regarding using renderd vs tirex but more of a question of where the future lies and thus for which to develop new features. It would be a shame and rather inefficient to put in effort to develop features for something that turns out to be a dead end What particular feature did you want to work on? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex vs Renderd
Hi! On Sat, Apr 17, 2010 at 10:21:31AM +0100, Kai Krueger wrote: with the recent appearance of tirex there are now at least two, with the python version of renderd 3, but probably more implementations of a rendering backend. All attempt to roughly do the same, i.e. take requests from a webserver or other program and render them. So the question is what is the future outlook for each and the relation between them? I.e. is tirex supposed to replace renderd and tile.osm.org will eventually switch? Or was tirex just something geofabrik threw together for a particular client and renderd should remain the default? Are they intended to solve the same problem and compete for the same market or does each have their own niche for which they are optimized for? This is less of a question though regarding using renderd vs tirex but more of a question of where the future lies and thus for which to develop new features. It would be a shame and rather inefficient to put in effort to develop features for something that turns out to be a dead end and my concern is to fracture the not directly large pool of developers for renderd/tirex. Yes, sometimes competition can be a good thing and thanks to it all being opensource one can fork and merge at will, but I would be interested to hear how others involved see the situation and help potential contributors to make a informed decision to which project best to contribute to. As one of the two primary developers of Tirex, I am - of course - biased, so take this with a grain of salt: I think that for most uses Tirex is the better and more flexible platform. Of course I would say that because we specifically started this prject because of the inflexibilities we saw in mod_tile/renderd that we wanted to address. Currently Tirex is in production use at tile.openstreetmap.de (Tile server mostly used on www.openstreetmap.de) and on the development server dev.openstreetmap.de (used for the topo.openstreetmap.de) and of course at Geofabrik. At least one other person is using it, as far as I know. This is still a very small user base. We are still actively working on Tirex (currently I am working on provisions to allow different rendering backends and make working with different styles easier), but we need more developers in the long run. Its by no means a stable Open Source project yet. There is quite a lot of documentation, but not enough yet and whats there is hard to find. Thats on my todo list, too. So while renderd is older and used in more places, Tirex is more flexible and in more active development currently (as far as I can see). Hard to tell where this is going, but I recommend going with Tirex. :-) Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex vs Renderd
On 04/17/2010 11:29 AM, Frederik Ramm wrote: Kai, Kai Krueger wrote: This is less of a question though regarding using renderd vs tirex but more of a question of where the future lies and thus for which to develop new features. It would be a shame and rather inefficient to put in effort to develop features for something that turns out to be a dead end What particular feature did you want to work on? It was somewhat intended as a more general question than a specific feature, but what sparked the question was the potential need to extend the rendering to support a lot of styles (200+) as is currently being discussed on the wikimedia maps list. At http://cassini.toolserver.org/tile-browse/ there are renderings for all the 278 languages wikipedia supports. With renderd each of the rendering threads has a copy of all the mapnik styles. As each style takes up about 5 - 6 Mb, this results in 278*8*5.5 = 12Gb of ram. Not directly desirable. So thoughts exist to implement some form of on demand loading and unloading of styles, as it is unlikely that all those styles will be needed all the time and/or share memory accross rendering threads/processes. So the question somewhat was where to do this potential development. But as it is only thoughts at the moment it will probably depend on who ever gets to actually implementing this. Hence my more general question of which of the two seems more recomended/future proof for potential developments like that. Kai Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex vs Renderd
On Sat, 2010-04-17 at 10:21 +0100, Kai Krueger wrote: Hello, with the recent appearance of tirex there are now at least two, with the python version of renderd 3, but probably more implementations of a rendering backend. All attempt to roughly do the same, i.e. take requests from a webserver or other program and render them. So the question is what is the future outlook for each and the relation between them? I.e. is tirex supposed to replace renderd and tile.osm.org will eventually switch? Or was tirex just something geofabrik threw together for a particular client and renderd should remain the default? Are they intended to solve the same problem and compete for the same market or does each have their own niche for which they are optimized for? I don't think tile.osm.org will be moving away from the mod_tile renderd in the near future but if tirex or something else demonstrates that it can do a better job then I could be persuaded to switch. This is less of a question though regarding using renderd vs tirex but more of a question of where the future lies and thus for which to develop new features. It would be a shame and rather inefficient to put in effort to develop features for something that turns out to be a dead end and my concern is to fracture the not directly large pool of developers for renderd/tirex. Yes, sometimes competition can be a good thing and thanks to it all being opensource one can fork and merge at will, but I would be interested to hear how others involved see the situation and help potential contributors to make a informed decision to which project best to contribute to. I see a place for both. Geofabrik have developed something which fits their needs better than mod_tile. On one level it is slightly wasteful to have multiple implementations, but on another it means that we may open up to a broader range of C/C++/Python/Perl developers. Any complex problem like map rendering is going to have several 'best' solutions depending on your priorities for determining merit. Even if a new feature was developed using one system does not mean that it won't be ported to run with another. For example, if I was working on enhancing the flexibility of the map rendering I might start experimenting with the python version and then move to a C implementation once I was happy with the approach. I would recommend you to pick whatever works best for you. The open source world is full of competing projects: Mapnik/Osmarender, JOSM/Potlatch, Linux/BSD, Gnome/KDE, Emacs/Vi, Bash/Csh, Python/Ruby, Debian/Ubuntu/Fedora. I believe this diversity encourages innovation and is probably a natural outcome of the bazaar style development model. Jon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex vs Renderd
On Sat, 2010-04-17 at 12:09 +0100, Kai Krueger wrote: On 04/17/2010 11:29 AM, Frederik Ramm wrote: Kai, Kai Krueger wrote: This is less of a question though regarding using renderd vs tirex but more of a question of where the future lies and thus for which to develop new features. It would be a shame and rather inefficient to put in effort to develop features for something that turns out to be a dead end What particular feature did you want to work on? It was somewhat intended as a more general question than a specific feature, but what sparked the question was the potential need to extend the rendering to support a lot of styles (200+) as is currently being discussed on the wikimedia maps list. At http://cassini.toolserver.org/tile-browse/ there are renderings for all the 278 languages wikipedia supports. With renderd each of the rendering threads has a copy of all the mapnik styles. As each style takes up about 5 - 6 Mb, this results in 278*8*5.5 = 12Gb of ram. Not directly desirable. So thoughts exist to implement some form of on demand loading and unloading of styles, as it is unlikely that all those styles will be needed all the time and/or share memory accross rendering threads/processes. So the question somewhat was where to do this potential development. But as it is only thoughts at the moment it will probably depend on who ever gets to actually implementing this. Hence my more general question of which of the two seems more recomended/future proof for potential developments like that. One thing to consider is whether the best place to focus the effort might be to reduce the memory usage by Mapnik. This then benefits both projects. I believe you are the first person to measure and highlight this memory usage. I would recommend you raise a ticket at trac.mapnik.org (if you have not already). For many uses 5MB per style is perfectly reasonable but not in your situation. Like many things there is a trade off between the time to reload the style vs the memory usage to pre-load everything. The most recent Mapnik code has moved to a new parser and is now supporting more flexible style rules. It might be possible that Mapnik would one day be able to render all your language variants using just a single style with some extra run-time parametrisation. It might also be possible to split the rendering so that language independent features could be rendered once and then the language specific layers rendered on top. Jon ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex vs Renderd
On Sat, Apr 17, 2010 at 12:09:52PM +0100, Kai Krueger wrote: On 04/17/2010 11:29 AM, Frederik Ramm wrote: Kai, Kai Krueger wrote: This is less of a question though regarding using renderd vs tirex but more of a question of where the future lies and thus for which to develop new features. It would be a shame and rather inefficient to put in effort to develop features for something that turns out to be a dead end What particular feature did you want to work on? It was somewhat intended as a more general question than a specific feature, but what sparked the question was the potential need to extend the rendering to support a lot of styles (200+) as is currently being discussed on the wikimedia maps list. At http://cassini.toolserver.org/tile-browse/ there are renderings for all the 278 languages wikipedia supports. With renderd each of the rendering threads has a copy of all the mapnik styles. As each style takes up about 5 - 6 Mb, this results in 278*8*5.5 = 12Gb of ram. Not directly desirable. So thoughts exist to implement some form of on demand loading and unloading of styles, as it is unlikely that all those styles will be needed all the time and/or share memory accross rendering threads/processes. So the question somewhat was where to do this potential development. But as it is only thoughts at the moment it will probably depend on who ever gets to actually implementing this. Hence my more general question of which of the two seems more recomended/future proof for potential developments like that. Having 278 styles that only differ in a few places where the different languages are involved certainly seems wasteful. We have already thought about the on-demand-loading of styles and it is probably something we want to implement for Tirex. There is a different use case that also needs that: When you are working on and testing a new style, you don't want to restart your renderer all the time to reload styles. Thats a different problem, but it would need a similiar solution, ie load styles on use. But in your case it might not solve the problem. Depending on how often tiles are asked for for the different languages, you might still end up with either most of the styles in memory or otherwise need to constantly reload the styles which takes time. There are many ways we could approach this: One would be to implement something in Mapnik so that it can allow variables in style files. Then you would only have one style file for all languages and somehow feed the language in separately. Another approach would be to render the map in one step and the labels in a second step and merge them. You still had one style file for each language, but they would be much smaller, because they contain only the labels. This merging could be done in the renderer. Bascially it would call Mapnik twice and merge the images. Or maybe we can modify Mapnik so that it can not only render into a new empty bitmap, but also into an existing one. Something like this. Most of these approaches are probably easier with Tirex, because we separated the queue management from the rendering. And, as I said in my other mail, I am currently working on making the rendering backend more configurable and flexible. The idea is that you can configure for any map, which rendering backend to call with which parameters. So it should be easy to experiment with a new backend. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex vs Renderd
On Sat, Apr 17, 2010 at 7:59 AM, Jochen Topf joc...@remote.org wrote: There are many ways we could approach this: One would be to implement something in Mapnik so that it can allow variables in style files. Then you would only have one style file for all languages and somehow feed the language in separately. That sounds like a great idea. Please see the New Expressions Engine portion of the Mapnik2 announcement. You might like it. ;-) http://mapnik.org/news/2009/dec/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex vs Renderd
Hi, Richard Weait wrote: On Sat, Apr 17, 2010 at 7:59 AM, Jochen Topf joc...@remote.org wrote: There are many ways we could approach this: One would be to implement something in Mapnik so that it can allow variables in style files. Then you would only have one style file for all languages and somehow feed the language in separately. That sounds like a great idea. Please see the New Expressions Engine portion of the Mapnik2 announcement. You might like it. ;-) http://mapnik.org/news/2009/dec/ That could be a good basis for what Jochen suggested; the missing link would be the ability to inject variables from the outside when rendering, i.e. before you call the renderer's apply() method in Mapnik you would have to do something like render.setVariable(key, value) and then you could have a rule in the style file that, dending on the value, would create a different rendering outcome. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - a renderd alternative
Op 09-03-10 14:07, Jochen Topf schreef: As long as you are on the same host and don´t fill up the buffer UDP is reliable. The whole thing works only on the same host anyway, so this should not be a problem. The same host thing is not what my future is. The idea of multiple render services that can be balanced by the webserver is. But since it is now also unknown if one could actually connect to a remote tile service, I would bias for TCP. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - a renderd alternative
Op 09-03-10 08:46, Frederik Ramm schreef: But Stefan, you are of course welcome to modify your mod_tile version to talk to Tirex directly. The protocol is really most simple, you just have to send one UDP message and wait for an answer. The wait for an answer part, how can your 'rely' on that with UDP? Or is your idea to that if your timeout expires, than serve a busy tile? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - a renderd alternative
Directly some other issue; since we also have modified Renderd (extensively), can I suggest some features to you too? - Meta tile rendering, but not storing it as meta tile. http://trac.openstreetmap.org/ticket/2653 - Per layer image settings http://trac.openstreetmap.org/ticket/2654 Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - a renderd alternative
Stefan, Stefan de Konink wrote: The same host thing is not what my future is. The idea of multiple render services that can be balanced by the webserver is. But since it is now also unknown if one could actually connect to a remote tile service, I would bias for TCP. I agree that distributed rendering is something that must be supported. There are several ways to do it; either run a classic load balancer setup with some intelligence so that it requests odd tiles from one server and even tiles from another etc., or build a mesh of queue managers that are aware of each other, or have one master queue manager (a.k.a. single point of failure) distributing requests across a range of render servers. If you were to attempt the latter, I would recommend a messaging deamon that takes UDP requests and transports them to the other side using TCP. That would allow you to have a steady TCP connection and monitor it properly. For localhost use, we decided against TCP because creating a new connection everytime you need one is too expensive, and re-using the existing one causes too much trouble. In theory, Tirex could also be expanded to use multiple messaging schemes, just as it uses UDP and Unix domain sockets (for backward compatibility) now. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - a renderd alternative
Op 09-03-10 17:19, Frederik Ramm schreef: I agree that distributed rendering is something that must be supported. There are several ways to do it; either run a classic load balancer setup with some intelligence so that it requests odd tiles from one server and even tiles from another etc., or build a mesh of queue managers that are aware of each other, or have one master queue manager (a.k.a. single point of failure) distributing requests across a range of render servers. I currently have a webserver that does this for me. If you were to attempt the latter, I would recommend a messaging deamon that takes UDP requests and transports them to the other side using TCP. That would allow you to have a steady TCP connection and monitor it properly. For localhost use, we decided against TCP because creating a new connection everytime you need one is too expensive, and re-using the existing one causes too much trouble. If I would go into this direction I would go by multicast such as mDNS, and make render clients bid for rendering the requests. More flexible approach. And gives guarantees someone picks it up. In theory, Tirex could also be expanded to use multiple messaging schemes, just as it uses UDP and Unix domain sockets (for backward compatibility) now. I already like it that it is decoupled from the webserver itself. The only thing that has to be taken into account with an 'cgi' approach is the fork cost. But personally... I think with a small script that only shoots in the coordinates in a packet, that is good enough for anyone that doesn't want to write 25 lines of C. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - a renderd alternative
Hi, Stefan de Konink wrote: I agree that distributed rendering is something that must be supported. There are several ways to do it; either run a classic load balancer setup with some intelligence so that it requests odd tiles from one server and even tiles from another etc., or build a mesh of queue managers that are aware of each other, or have one master queue manager (a.k.a. single point of failure) distributing requests across a range of render servers. I currently have a webserver that does this for me. A web server is sufficient for simple applications but if you want to pre-render a million tiles without impeding on live viewing performance then a more controlled approach is probably required. If I would go into this direction I would go by multicast such as mDNS, and make render clients bid for rendering the requests. More flexible approach. And gives guarantees someone picks it up. Sounds interesting. I don't understand enough about that but if anyone wants to try... Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - a renderd alternative
2010/3/8 Frederik Ramm frede...@remote.org: a few days ago I committed a first working version of Tirex to our svn (under applications/utils). Tirex is basically an alternative for renderd, it does the same things, just differently. Nice. It would be cool if it supported pluggable elaborate rendering flows like the one used in http://wiki.openstreetmap.org/wiki/TopOSM/Details. Cheers Colin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - a renderd alternative
Op 09-03-10 21:28, Frederik Ramm schreef: I agree that distributed rendering is something that must be supported. There are several ways to do it; either run a classic load balancer setup with some intelligence so that it requests odd tiles from one server and even tiles from another etc., or build a mesh of queue managers that are aware of each other, or have one master queue manager (a.k.a. single point of failure) distributing requests across a range of render servers. I currently have a webserver that does this for me. A web server is sufficient for simple applications but if you want to pre-render a million tiles without impeding on live viewing performance then a more controlled approach is probably required. Sure but in that case I would go for the 'nuke a zillion udp packets to somewhere'. If I would go into this direction I would go by multicast such as mDNS, and make render clients bid for rendering the requests. More flexible approach. And gives guarantees someone picks it up. Sounds interesting. I don't understand enough about that but if anyone wants to try... Basically it works like a negotiation strategy. But we can discuss it offline. Its trivial te implement it or to copy it from one of my other projects. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Tirex - a renderd alternative
Hi, a few days ago I committed a first working version of Tirex to our svn (under applications/utils). Tirex is basically an alternative for renderd, it does the same things, just differently. We (Geofabrik) had to roll out a tile server for one of our clients (a company named Enaikoon from Berlin - they do vehicle tracking and other telematics stuff). They wanted some functionality that would have required extensive changes to renderd, and we decided to write something new instead. Enaikoon likes OSM and open source, so the result is a fully open-source OSM project just as everything else which you can try out, run, and improve if you want. The main difference between the classic renderd and Tirex is that in Tirex, the actual rendering and the queue management are strictly separate. You have one daemon which only manages requests and queues and priorities and all that, and you have another set of daemons that do nothing but tile rendering. Communication between these is done using UDP, with a very simple protocol where you have one key=value pair per line. Tirex does not use multithreading at all - everything is either multiplexing inside one process, or multiprocessing. The bit that handles the queue is called the Tirex master. It is written in Perl to allow for easy manipulation of policy decisions. The Tirex master supports any number of rendering queues (here called buckets) with a high degree of configurability. For example, if you know that you have 16 backend rendering processes, then you can configure the master to use no more than 10 of these for low-priority requests (and only do so below a certain load) so that you always retain spare rendering capacity for interactive requests. Rendering requests can also have an expire time so that if the queue manager could not get them rendered within a specified time they are discarded. The bit that does the actual rendering is the Tirex rendering daemon or Tirex renderd. It is written in C++ but does very little - read rendering request, call Mapnik, create Metatile. The plan is to support other rendering backends as well. You will usually have a number of these running (and they are automatically re-started if they die e.g. due to a problem in the Mapnik library). In addition there's a number of helper programs, for example a nifty status console that lets you see what your queues are doing, munin plugins, and a tool to mass-enqueue requests (everything in the following bbox from z6 to z10 but only if the queue is not too long) and other helpful stuff. Further documentation is in SVN. A debian subdirectory is provided, so on reasonably current Debian and Ubuntu distributions you should be able to simply do a debuild to create a number of .deb packages. We do not have a replacement for mod_tile yet but are planning to have one in the future. At the moment you can simply use the existing mod_tile, unchanged, with Tirex. You can also use all of the existing utilities (render_list etc.) with Tirex. Tirex uses the same meta tiles and the same directory structure, making it easy to run it as a drop-in replacement for renderd if desired. We don't expect Tirex to quickly replace renderd - renderd is well established and does what it is expected to do. But we believe that the more modular structure of Tirex makes it easier for developers to play around and plug in their own little bits and pieces (just as it was for us). We do have some plans for Tirex ourselves and will surely continue to contribute, but we do not plan to keep this as our own, and we will work with the OSM svn version in the future just as everyone else. Tirex is in production on tile.openstreetmap.de since two weeks ago, and seems to work well. Feedback is welcome. There's a Wiki page named Tirex with about the same content as this. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tirex - a renderd alternative
Hi, Ævar Arnfjörð Bjarmason wrote: What are the plans for the mod_tile replacement? Hopefully something to decouple it from Apache, maybe using FastCGI? I spoke to JonB during Tirex development and that was one of the ideas he had as well. Either FastCGI, or at least put all the relevant stuff in a library so that when doing a native module for whatever web server you want to use, you just have to glue together a few library calls. But Stefan, you are of course welcome to modify your mod_tile version to talk to Tirex directly. The protocol is really most simple, you just have to send one UDP message and wait for an answer. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
