Re: [mapserver-users] mapcache seed slow
We have 2 dedicated servers that we use to serve the full mapping system and when we update the map we switch the system to just one server and use the other one just to see the new map with a different URL. Sebastiano Laini Web Developer From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 22 February 2019 15:03 To: Sebastiano Laini ; 'mapserver-users@lists.osgeo.org' Subject: Re: mapcache seed slow Hi, Stupid question, but do you know how fast your WMS service is? How many GetMap requests/sec it can serve with the metatile size that you are seeding with? Have you some evidence that WMS is running idle while MapCache is busy with cutting the tiles and filling the cache? -Jukka Rahkonen- Lähettäjä: mapserver-users mailto:mapserver-users-boun...@lists.osgeo.org>> Puolesta Sebastiano Laini Lähetetty: perjantai 22. helmikuuta 2019 13.35 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Aihe: [mapserver-users] mapcache seed slow Hi all, I'm trying to seed the whole UK at all zoom lv with these resolutions 280 140 70 28 14 7 6.16 2.8 1.4 0.7 But after 13 days it arrives at the 5th level where before I kill the process it seed just 900MB on that level. This is my script su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c /usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 400" 2>&1> /var/www/vhosts/$SITE/scripts/cache_log.txt I know that in the documentation say that -n should be less or max the number of cpu in the server but is really slow if I put 2, so I tried with 100 and after 13 days that was the result so now I'm trying 400 Is there something else I can use to speed up the seed? I'm using the SQlite cache method Should I use the -p (| -nprocesses: number of parallel processes that should be used to request tiles from the WMS source) Instead of the -n parameter? Sebastiano Laini Web Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache seed slow
I don't know how fast is but I have a whole dedicated server just for do that with a development URL that I'm using just to develop, is still configured in HTTP/1, looking to make some test on a new server to setup HTTP/2 with NGINX and possibly WMTS. I'm seeding with a bash file and I can see (with TOP) that it's using about 60/80% CPU usage and other HTTPD requests make it from the APACHE user. Sebastiano Laini Web Developer From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 22 February 2019 15:03 To: Sebastiano Laini ; 'mapserver-users@lists.osgeo.org' Subject: Re: mapcache seed slow Hi, Stupid question, but do you know how fast your WMS service is? How many GetMap requests/sec it can serve with the metatile size that you are seeding with? Have you some evidence that WMS is running idle while MapCache is busy with cutting the tiles and filling the cache? -Jukka Rahkonen- Lähettäjä: mapserver-users mailto:mapserver-users-boun...@lists.osgeo.org>> Puolesta Sebastiano Laini Lähetetty: perjantai 22. helmikuuta 2019 13.35 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Aihe: [mapserver-users] mapcache seed slow Hi all, I'm trying to seed the whole UK at all zoom lv with these resolutions 280 140 70 28 14 7 6.16 2.8 1.4 0.7 But after 13 days it arrives at the 5th level where before I kill the process it seed just 900MB on that level. This is my script su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c /usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 400" 2>&1> /var/www/vhosts/$SITE/scripts/cache_log.txt I know that in the documentation say that -n should be less or max the number of cpu in the server but is really slow if I put 2, so I tried with 100 and after 13 days that was the result so now I'm trying 400 Is there something else I can use to speed up the seed? I'm using the SQlite cache method Should I use the -p (| -nprocesses: number of parallel processes that should be used to request tiles from the WMS source) Instead of the -n parameter? Sebastiano Laini Web Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache seed slow
Hi, Stupid question, but do you know how fast your WMS service is? How many GetMap requests/sec it can serve with the metatile size that you are seeding with? Have you some evidence that WMS is running idle while MapCache is busy with cutting the tiles and filling the cache? -Jukka Rahkonen- Lähettäjä: mapserver-users Puolesta Sebastiano Laini Lähetetty: perjantai 22. helmikuuta 2019 13.35 Vastaanottaja: 'mapserver-users@lists.osgeo.org' Aihe: [mapserver-users] mapcache seed slow Hi all, I'm trying to seed the whole UK at all zoom lv with these resolutions 280 140 70 28 14 7 6.16 2.8 1.4 0.7 But after 13 days it arrives at the 5th level where before I kill the process it seed just 900MB on that level. This is my script su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c /usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 400" 2>&1> /var/www/vhosts/$SITE/scripts/cache_log.txt I know that in the documentation say that -n should be less or max the number of cpu in the server but is really slow if I put 2, so I tried with 100 and after 13 days that was the result so now I'm trying 400 Is there something else I can use to speed up the seed? I'm using the SQlite cache method Should I use the -p (| -nprocesses: number of parallel processes that should be used to request tiles from the WMS source) Instead of the -n parameter? Sebastiano Laini Web Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache seed slow
I’m using this configuration
/var/www/vhosts/buchananmapping.co.uk/httpdocs/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3
1
1
1573741823
image/png
miniscale
…/maps/bcmaps.map
https://.../cgi-bin/mapserv?
bcmaps
cache_sqlite
bcgrid
PNG
4 4
0
2628000
BC custom grid
EPSG:27700
128 128
0 0 70 125
280 140 70 28 14 7 6.16 2.8 1.4 0.7
best
It supposed to have a limit that it’s the full EPSG extent
Sebastiano Laini
Web Developer
From: Travis Kirstine [mailto:traviskirst...@gmail.com]
Sent: 22 February 2019 13:57
To: Sebastiano Laini
Cc: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] mapcache seed slow
The time you have indicated seems excessive to seed the UK. My guess is that
the current process it seeding the extents of your grid (world?) but default.
I've seeded the province of ontario 1 million km2 z0-17 in about 2 weeks using
a sqlite cache
Ramping up the -n will only bottle neck the process more (same with the -p
flag) the limiting factor is SQLite and it limitation on concurrent writes (I
use -n 4). I've noticed that if you try to use other sqlite3 journal modes you
can degrade performance
(http://osgeo-org.1560.x6.nabble.com/mapcache-sqlite-cache-large-wal-files-and-blocked-checkpoints-when-seeding-td5377571.html).
I would just use most basic cache config as a starting point. BTW using SSD
should increase the SQLite writes as well (if you are lucky enough to have them)
./{tileset}.sqlite3
There are a few things that you could do when seeding to speed up the process:
Use the "restricted_extent" parameter in the tileset grid to limit the number
of tiles
Example
ontario_2018_wms
sqlite
gmaps
mixed
5 5
Use a polygon shapefile (or other ogr datasource) to limit the seeding using
the -d flag in mapcache_seed command (note the projection of the datasource
should be the same as the grid)
Use a larger metatile when seeding (-M flag override in command line).
Mapcache will "bulk" insert the resultant tiles based on the metatile - if
5 5 mapcache will insert 25 tiles at a time if 8
8 it will do 64 (I think this is the way it works). The limiting
factor here is trying to balance this with your source WMS performance. I use
8,8 when seeding
Regards
On Fri, 22 Feb 2019 at 06:35, Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>>
wrote:
Hi all,
I’m trying to seed the whole UK at all zoom lv with these resolutions
280 140 70 28 14 7 6.16 2.8 1.4 0.7
But after 13 days it arrives at the 5th level where before I kill the process
it seed just 900MB on that level.
This is my script
su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c
/usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 –n
400" 2>&1> /var/www/vhosts/$SITE/scripts/cache_log.txt
I know that in the documentation say that –n should be less or max the number
of cpu in the server but is really slow if I put 2, so I tried with 100 and
after 13 days that was the result so now I’m trying 400
Is there something else I can use to speed up the seed? I’m using the SQlite
cache method
Should I use the -p (| –nprocesses: number of parallel processes that should be
used to request tiles from the WMS source) Instead of the –n parameter?
Sebastiano Laini
Web Developer
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache seed slow
The time you have indicated seems excessive to seed the UK. My guess is
that the current process it seeding the extents of your grid (world?) but
default. I've seeded the province of ontario 1 million km2 z0-17 in about
2 weeks using a sqlite cache
Ramping up the -n will only bottle neck the process more (same with the -p
flag) the limiting factor is SQLite and it limitation on concurrent writes
(I use -n 4). I've noticed that if you try to use other sqlite3 journal
modes you can degrade performance (
http://osgeo-org.1560.x6.nabble.com/mapcache-sqlite-cache-large-wal-files-and-blocked-checkpoints-when-seeding-td5377571.html).
I would just use most basic cache config as a starting point. BTW using
SSD should increase the SQLite writes as well (if you are lucky enough to
have them)
./{tileset}.sqlite3
There are a few things that you could do when seeding to speed up the
process:
Use the "restricted_extent" parameter in the tileset grid to limit the
number of tiles
Example
ontario_2018_wms
sqlite
gmaps
mixed
5 5
Use a polygon shapefile (or other ogr datasource) to limit the seeding
using the -d flag in mapcache_seed command (note the projection of the
datasource should be the same as the grid)
Use a larger metatile when seeding (-M flag override in command line).
Mapcache will "bulk" insert the resultant tiles based on the metatile -
if 5 5 mapcache will insert 25 tiles at a time if
8 8 it will do 64 (I think this is the way it works).
The limiting factor here is trying to balance this with your source WMS
performance. I use 8,8 when seeding
Regards
On Fri, 22 Feb 2019 at 06:35, Sebastiano Laini <
sebastiano.la...@buchanancomputing.co.uk> wrote:
> Hi all,
>
>
>
> I’m trying to seed the whole UK at all zoom lv with these resolutions
>
>
>
> 280 140 70 28 14 7 6.16 2.8 1.4 0.7
>
>
>
> But after 13 days it arrives at the 5th level where before I kill the
> process it seed just 900MB on that level.
>
>
>
> This is my script
>
>
>
> su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c
> /usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z
> 5,6 –n 400" 2>&1> /var/www/vhosts/$SITE/scripts/cache_log.txt
>
>
>
> I know that in the documentation say that –n should be less or max the
> number of cpu in the server but is really slow if I put 2, so I tried with
> 100 and after 13 days that was the result so now I’m trying 400
>
>
>
> Is there something else I can use to speed up the seed? I’m using the
> SQlite cache method
>
>
>
> Should I use the -p (| –nprocesses: number of parallel processes that
> should be used to request tiles from the WMS source) Instead of the –n
> parameter?
>
>
>
> Sebastiano Laini
>
> Web Developer
>
>
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] mapcache seed slow
Hi all, I'm trying to seed the whole UK at all zoom lv with these resolutions 280 140 70 28 14 7 6.16 2.8 1.4 0.7 But after 13 days it arrives at the 5th level where before I kill the process it seed just 900MB on that level. This is my script su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c /usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 400" 2>&1> /var/www/vhosts/$SITE/scripts/cache_log.txt I know that in the documentation say that -n should be less or max the number of cpu in the server but is really slow if I put 2, so I tried with 100 and after 13 days that was the result so now I'm trying 400 Is there something else I can use to speed up the seed? I'm using the SQlite cache method Should I use the -p (| -nprocesses: number of parallel processes that should be used to request tiles from the WMS source) Instead of the -n parameter? Sebastiano Laini Web Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Configure a images pyramid in mapserver - gdal_retile command
Hi, have you found the documentation, https://mapserver.org/input/raster.html, specifically chapter "Raster and Tile indexing" I can try to help if you have a more precise question. /Erik On Mon, Feb 18, 2019 at 3:21 AM Herbert wrote: > > I have already created a pyramid of images with GDAL using the command > gdal_retile. > I currently use the geoserver, but I want to test the performance with the > mapserver. (The geoserver has already created the .shp and .proj files > inside each directory of the pyramid) > > --- > I already see it: > https://lists.osgeo.org/pipermail/mapserver-users/2010-August/066444.html > but i used gdal_retile and I want check for this possibility.. > > thanks > Herbert > ___ > mapserver-users mailing list > mapserver-users@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
