Re: [OSM-talk] TileMill performance
Hi again,
I've now moved to an 8-core VM with 32Gb of RAM, which has made
certain operations faster, but processing the stylesheets and
beginning rendering still seems to be very slow.
Some numbers: from the moment I hit 'save' at zoom 13 in the centre of
Melbourne:
1. at 1:05 , the spinning icon stops (I assume this means the
stylesheets have been saved and processed)
2. at 2:15, the first tile appears
3. at 2:35, all visible tiles are rendered
.
During this process, there's almost no database activity - all queries
are handled very quickly.
Turning off every layer (so just rendering background-color):
1 0:48, spinning icon stops (tilemill-ui at just over 100% until now)
2. 1:45, first tile appears, (tilemill-tile around 100% until now).
3. 1:46 all tiles
This cluster has had problems with slow file I/O in the past, so
wonder if that's the likely result?
By contrast, a simple project with simple stylesheet saves and renders
almost instantly (1s).
The styles are here: https://github.com/stevage/stevebikemap
What I'd love to know:
1) What is happening for the first 50 seconds? Does TileMill really
take that long to parse 5 smallish stylesheets and convert them into
mapnik styles? If it sounds like a bug, I'm happy to investigate
further...
2) What exactly is happening in step 2? I don't know Mapnik at all, so
I'm just guessing that it's doing pre-computation like loading all the
objects and their styles, computing z-orders, resolving conflicts
between labels etc. Still this seems slow?
3) Why is step 2 so slow even when every layer is invisible?
4) Is there any way to increase the number of threads/processes, to
get more advantage from the 8 cores?
Hmm, with further investigation, it seems almost all the time is going
into the 'areas.mss' style. Commenting it out reduces the whole
save/render cycle to just 7 seconds (from 155!)
As an example:
#areas[zoom13][leisure=pitch],
#areas[zoom13][leisure=golf_course],
#areas[zoom13][landuse=cemetery] {
line-color:darken(green,5%);
line-width:1;
line-opacity:0.2;
polygon-fill:#bdb;
}
Rendering that one rule takes an additional 7 seconds (ie, 14 seconds
instead of 7, including saving/processing time), even at zoom 13, when
it's not even being applied. Making the #areas layer invisible had no
effect on rendering time. I guess I can further decompose these db
queries (#landuse, #leisure...), but is there anything else I can do
to speed them up? Is this normal?
Thanks again for any information,
Steve
On Wed, Feb 27, 2013 at 10:03 PM, Steve Bennett stevag...@gmail.com wrote:
Hi,
(First - is this the right list to discuss using TileMill? I can
only find the MapBox support form, or gis.stackexchange.)
I'm having some problems with TileMill rendering very slowly.
Sometimes it seizes up altogether, until I restart it or reboot the
server. This seems to happen particularly when I frequently interrupt
rendering by saving the stylesheet again.
My setup is a 2-core, 8Gb Ubuntu VM running on an OpenStack cluster.
PostGIS (with Melbourne city data from bbbike.org), nginx for auth,
and one shapefile. Pretty vanilla.
I have a few questions about how to improve speed in TileMill/Mapnik:
1) In general, what kinds of rendering rules are slow? Does the way
you specify a rule affect the speed? (eg, is [zoom13] { #ways[...] }
slower/faster than #ways[...][zoom13] ?)
2) How does caching take place? It seems to me that when saving a
stylesheet with changes, there's a long delay before anything renders,
then subsequent small changes aren't too slow. So some layers are
computed once then reused?
3) Are there any easy tips for tuning the database?
4) Or tuning TileMill/Mapnik?
5) Watching 'top' during a render, it doesn't look like much memory is
being used. Is there a way to trade memory for speed?
6) Does setting a layer invisible definitely prevent it being
computed? Sometimes I think I'm going mad...
7) Lastly,will adding cores lead to a proportional increase in speed?
Thanks very much in advance,
Steve
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
On Mon, Mar 4, 2013 at 7:02 PM, Steve Bennett stevag...@gmail.com wrote: effect on rendering time. I guess I can further decompose these db queries (#landuse, #leisure...), but is there anything else I can do to speed them up? Is this normal? Ok, yes, that's apparently what you need to do - with 21 layers, I'm now under 7 seconds for a full render. I wish there was a more elegant way though :/ Steve ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
I have found some combinations of stylesheet rules that suddenly cause the carto to take a long time to generate, however I've not managed to find a very small version that exhibits the issue: https://github.com/mapbox/carto/issues/213 The generated mapnik XML often isn't as compact as it could be. Shaun On 4 Mar 2013, at 08:21, Steve Bennett stevag...@gmail.com wrote: On Mon, Mar 4, 2013 at 7:02 PM, Steve Bennett stevag...@gmail.com wrote: effect on rendering time. I guess I can further decompose these db queries (#landuse, #leisure...), but is there anything else I can do to speed them up? Is this normal? Ok, yes, that's apparently what you need to do - with 21 layers, I'm now under 7 seconds for a full render. I wish there was a more elegant way though :/ Steve ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
On Miércoles, 27 de febrero de 2013 12:03:30 Steve Bennett escribió: I'm having some problems with TileMill rendering very slowly. I suggest to have a look at Dane's slides about How to make slow maps: http://dbsgeo.com/foss4g2011/foss4g2011-lecture2-how-to-make-slow-maps.pdf Best, -- Iván Sánchez Ortega i...@sanchezortega.es i...@geonerd.org ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
On Thu, Feb 28, 2013 at 7:27 PM, Iván Sánchez Ortega i...@sanchezortega.es wrote: On Miércoles, 27 de febrero de 2013 12:03:30 Steve Bennett escribió: I'm having some problems with TileMill rendering very slowly. I suggest to have a look at Dane's slides about How to make slow maps: http://dbsgeo.com/foss4g2011/foss4g2011-lecture2-how-to-make-slow-maps.pdf Nice. Couple of questions about how to avoid certain of those traps: 1) How do you control multithreading? It looks like I'm getting access to some 8 core VMs. How should I configure TileMill/Postgres for best performance then? 2) How do I know whether reprojection is happening? At least I have fixed the query more than you render problem - that did indeed improve performance significantly. Steve ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
On Jueves, 28 de febrero de 2013 09:57:34 Steve Bennett escribió: Nice. Couple of questions about how to avoid certain of those traps: 1) How do you control multithreading? It looks like I'm getting access to some 8 core VMs. How should I configure TileMill/Postgres for best performance then? Tilemill is not a map server. Set up a propar map server which is able to configure proper multithreading. 2) How do I know whether reprojection is happening? Make sure your source data is in the very same projection as the display. If t isn't, mapnik will reproject on-the-fly. -- Iván Sánchez Ortega i...@sanchezortega.es i...@geonerd.org ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
On Thu, Feb 28, 2013 at 8:05 PM, Iván Sánchez Ortega i...@sanchezortega.es wrote: On Jueves, 28 de febrero de 2013 09:57:34 Steve Bennett escribió: Nice. Couple of questions about how to avoid certain of those traps: 1) How do you control multithreading? It looks like I'm getting access to some 8 core VMs. How should I configure TileMill/Postgres for best performance then? Tilemill is not a map server. Set up a propar map server which is able to configure proper multithreading. Again, I'm not trying to serve up static tiles to the public. I'm just trying to get quicker performance for styles under development. If that's not possible, then ok. Steve ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
Hi Steve,
TileMill is not designed for that kind of application (running as a live
server with no cache), though it will work 'a bit'.
So: it doesn't do caching - you'll want a cache. Look at CloudFront,
nginx's cache, varnish, squid, and so on.
Tuning the database: check that you have all possible indexes installed and
the data is in EPSG:900913.
The long-term answer is switching to something designed to be a live-server
(mod_tile, TileStache) or rendering your tiles and serving them from
MBTiles (with TileStache or TileStream).
Tom
On Wed, Feb 27, 2013 at 6:03 AM, Steve Bennett stevag...@gmail.com wrote:
Hi,
(First - is this the right list to discuss using TileMill? I can
only find the MapBox support form, or gis.stackexchange.)
I'm having some problems with TileMill rendering very slowly.
Sometimes it seizes up altogether, until I restart it or reboot the
server. This seems to happen particularly when I frequently interrupt
rendering by saving the stylesheet again.
My setup is a 2-core, 8Gb Ubuntu VM running on an OpenStack cluster.
PostGIS (with Melbourne city data from bbbike.org), nginx for auth,
and one shapefile. Pretty vanilla.
I have a few questions about how to improve speed in TileMill/Mapnik:
1) In general, what kinds of rendering rules are slow? Does the way
you specify a rule affect the speed? (eg, is [zoom13] { #ways[...] }
slower/faster than #ways[...][zoom13] ?)
2) How does caching take place? It seems to me that when saving a
stylesheet with changes, there's a long delay before anything renders,
then subsequent small changes aren't too slow. So some layers are
computed once then reused?
3) Are there any easy tips for tuning the database?
4) Or tuning TileMill/Mapnik?
5) Watching 'top' during a render, it doesn't look like much memory is
being used. Is there a way to trade memory for speed?
6) Does setting a layer invisible definitely prevent it being
computed? Sometimes I think I'm going mad...
7) Lastly,will adding cores lead to a proportional increase in speed?
Thanks very much in advance,
Steve
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
Hi Steve,
(eg, is [zoom13] { #ways[...] } slower/faster than #ways[...][zoom13] ?)
Does your layer setup actually looks like this? ie. one 'ways' layer
pulling in the entire planet_osm_ways table? If so this will be
problematic. Unlike MapCSS, filtering out objects with CartoCSS will not
prevent them from being loaded. With Mapnik/TileMill you should be querying
specific thematic subsets of the database for each layer.
The layer setup plays an important role in performance, so knowing what
that looks like in detail (number of layers, their contents, file/database
type database queries, projections, etc) would help track down potential
issues.
Does setting a layer invisible definitely prevent it being computed?
TileMill/Mapnik will still check that hidden layers are valid each time the
project is saved/loaded, however that is the only thing it will do. These
layers shouldn't affect
--
AJ Ashton
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
On Wed, Feb 27, 2013 at 12:09 PM, AJ Ashton aj.ash...@gmail.com wrote: These layers shouldn't affect ...rendering speed, is what I forgot to type. -- AJ Ashton ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
On Thu, Feb 28, 2013 at 12:27 AM, Tom MacWright t...@macwright.org wrote: TileMill is not designed for that kind of application (running as a live server with no cache), though it will work 'a bit'. So: it doesn't do caching - you'll want a cache. Look at CloudFront, nginx's cache, varnish, squid, and so on. I might have been a bit unclear: I'm not trying to serve up pre-computed tiles to the public. I'm just doing style development, so rendering tiles to see what it looks like. So I don't think caching of pre-rendered tiles is an issue? Tuning the database: check that you have all possible indexes installed and the data is in EPSG:900913. Can you elaborate on all possible indexes? :) Where can I read up on this stuff? The long-term answer is switching to something designed to be a live-server (mod_tile, TileStache) or rendering your tiles and serving them from MBTiles (with TileStache or TileStream). Steve ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] TileMill performance
On Thu, Feb 28, 2013 at 4:09 AM, AJ Ashton aj.ash...@gmail.com wrote:
(eg, is [zoom13] { #ways[...] } slower/faster than #ways[...][zoom13] ?)
Does your layer setup actually looks like this? ie. one 'ways' layer pulling
in the entire planet_osm_ways table? If so this will be problematic. Unlike
MapCSS, filtering out objects with CartoCSS will not prevent them from being
loaded. With Mapnik/TileMill you should be querying specific thematic
subsets of the database for each layer.
Ah. Yes, #ways is the whole table - although it's just one city.
That's probably my problem. So, I guess I should set up individual
queries (#bikepaths, #parks, ...) using PostGIS to do the filtering
rather than Mapnik. That makes sense, thanks!
Steve
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk
