Re: [mapserver-users] mapcache seed speed optimization

2019-11-12 Thread Stephen Woodbridge 

Thank Lars.

You can now fetch it from https://github.com/woodbri/imaptools.com

Enjoy,
  -Steve W

On 11/11/2019 3:07 PM, lars.schylb...@blixtmail.se wrote:

Hi

I have had a similar situation as You with large national data sets in 
shape format. One of the biggest improvements for me has been to split 
large shape files into smaller and

then use vector tile indexes in Mapserver.
The tool shp2tile has been working well for me. Stephen Woodbridge has 
written that tool. It can be found at 
http://imaptools.com/download-software.html

I think that You had to compile yourself on linux, but that was easy.

I have an example where I have automated this in a script that runs 
through all shape file in a directory and splits shape files larger 
than a certain size and then creates the vector tile index files.

The code snippet that is part of larger import script can be found here:
https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359

The script snippet also shows the normal optimizations we do.

Hope this could give You some ideas how optimize and make Mapserver 
and Mapcache run faster. I also ran into the problem with to many inodes.

That got solved by using sqlite as backend in Mapcache.

Best regards / Lars Schylberg



11 november 2019 kl. 15:42, "Travis Kirstine" 
> 
skrev:


I may not be unrealistic for it take mapserver to 4-6 seconds to
generate 4096px image depending on the amount data being rendered
and the complexity of your styles / expressions / labels. If your
mapserver is able to generate a 1024x1024 in under a second that
is pretty good. Like Jukka and other suggest I would look at
translating the files to shp files with spatial indexes or postgis
tables with spatial and attribute indexes if necessary.
Preprocessing the data (simplification / prefiltering / point
thinning cluster) and using SCALETOKENs to reference different
source layers at different scales can lead to big performance
boosts at the mapserver end.
For us the biggest limiting factor has been the write blocking to
sqlite cache when seeding tiles, if you increase the number of
threads / process in mapcache seed those process will just end up
waiting to write to the cache. You can generally figure out what
the max number of processes the cache can handle by running a seed
on a test area and look at the number of tiles seeded per second,
as some point -n will have no effect which your process are just
waiting to write the cache. I would look at using this method to
test sqlite vs disk or other backends. The GeoTiff cache may be
promising as it appears to be non-blocking but experimental...
We use a riak cache with a riak leveldb backend which works well
for us but is a bit of a pain to manage as it difficult to delete
objects so it's not great for caches that need to be refreshed.
Finally you may want to test MapProxy as an alternative to
MapCache (we use both). MapProxy supports a non-blocking compact
cache that could solve your inode problem
Regards
On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>> wrote:

We don’t create the data, we rely on OS (ordnance survey) to
supply us the maps files and then we publish them in our
service but I assume that probably we will need to create some
flow to improve it or download it in other format that is faster.

Sebastiano Laini

Web Developer

Buchanan Computing

*From:*Fawcett, David (MNIT) [mailto:david.fawc...@state.mn.us
]
*Sent:* 08 November 2019 16:41
*To:* Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'Rahkonen
Jukka (MML)' mailto:jukka.rahko...@maanmittauslaitos.fi>>;
'mapserver-users@lists.osgeo.org
'
mailto:mapserver-users@lists.osgeo.org>>
*Subject:* RE: mapcache seed speed optimization

For data formats that are slower to read, would it add fit
your workflow to convert it to different data format before
creating the tiles?

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/mapserver-users 





___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed speed optimization

2019-11-12 Thread Sebastiano Laini 
I manage to seed at 300 tiles/s in the less complex layers, now I need to speak 
with the technical director and see how we are going to optimize our data.

I was trying to test MapProxy but I couldn’t install it in our Centos 7 
machine, I may give it a try again.
For your experience which one is the fastest one? Not just seeding but in 
delivering the requests also.

What tiles size are you using normally? Are you still using HTTP1 or HTTP2?

Regards,

Sebastiano Laini
Web Developer
Buchanan Computing

From: Travis Kirstine [mailto:traviskirst...@gmail.com]
Sent: 11 November 2019 14:42
To: Sebastiano Laini 
Cc: Fawcett, David (MNIT) ; Rahkonen Jukka (MML) 
; mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] mapcache seed speed optimization

I may not be unrealistic for it take mapserver to 4-6 seconds to generate 
4096px image depending on the amount data being rendered and the complexity of 
your styles / expressions / labels.  If your mapserver is able to generate a 
1024x1024 in under a second that is pretty good.  Like Jukka and other suggest 
I would look at translating the files to shp files with spatial indexes or 
postgis tables with spatial and attribute indexes if necessary.  Preprocessing 
the data (simplification / prefiltering / point thinning cluster) and using 
SCALETOKENs to reference different source layers at different scales can lead 
to big performance boosts at the mapserver end.

For us the biggest limiting factor has been the write blocking to sqlite cache 
when seeding tiles, if you increase the number of threads / process in mapcache 
seed those process will just end up waiting to write to the cache.  You can 
generally figure out what the max number of processes the cache can handle by 
running a seed on a test area and look at the number of tiles seeded per 
second, as some point -n will have no effect which your process are just 
waiting to write the cache.  I would look at using this method to test sqlite 
vs disk or other backends.  The GeoTiff cache may be promising as it appears to 
be non-blocking but experimental...

We use a riak cache with a riak leveldb backend which works well for us but is 
a bit of a pain to manage as it difficult to delete objects so it's not great 
for caches that need to be refreshed.

Finally you may want to test MapProxy as an alternative to MapCache (we use 
both).  MapProxy supports a non-blocking compact cache that could solve your 
inode problem

Regards









On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini 
mailto:sebastiano.la...@buchanancomputing.co.uk>>
 wrote:
We don’t create the data, we rely on OS (ordnance survey) to supply us the maps 
files and then we publish them in our service but I assume that probably we 
will need to create some flow to improve it or download it in other format that 
is faster.

Sebastiano Laini
Web Developer
Buchanan Computing

From: Fawcett, David (MNIT) 
[mailto:david.fawc...@state.mn.us]
Sent: 08 November 2019 16:41
To: Sebastiano Laini 
mailto:sebastiano.la...@buchanancomputing.co.uk>>;
 'Rahkonen Jukka (MML)' 
mailto:jukka.rahko...@maanmittauslaitos.fi>>;
 'mapserver-users@lists.osgeo.org' 
mailto:mapserver-users@lists.osgeo.org>>
Subject: RE: mapcache seed speed optimization

For data formats that are slower to read, would it add fit your workflow to 
convert it to different data format before creating the tiles?
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users