My experience with SQLite is that it is significantly slower than a disk
based cache. My guess is this is related to how mapcache structures the
transaction and inserts into the sqlitedb. It appears that the mapcache
creates a transaction based on the <metatile> defined in the config, if the
metatile is set to 5 5, mapcache will create a transaction with 25 inserts
per process. Problems seem to arise when multiple threads / processes
attempt to concurrently seed the cache and are backlogged / waiting due to
a transaction in progress. Disk cache appear to be only limited based on
disk io.
On Wed, 17 Oct 2018 at 08:47, Sebastiano Laini <
sebastiano.la...@buchanancomputing.co.uk> wrote:
> This is my cache setup
>
>
>
> <cache name="cache_sqlite" type="sqlite3">
>
>
> <dbfile>/var/www/path…./maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3</dbfile>
>
> <xcount>10000</xcount>
>
> <ycount>10000</ycount>
>
> <pragma name="max_page_count">1573741823</pragma>
>
> </cache>
>
>
>
> <grid name="bcgrid">
>
> <metadata>
>
> <title>this is a custom grid I made up for
> an example</title>
>
> </metadata>
>
> <srs>EPSG:27700</srs>
>
> <size>128 128</size>
>
> <extent>0 0 700000 1250000</extent>
>
> <resolutions>70 28 14 7 6.16 2.8 1.4 0.7 0.28 0.14 0.07</resolutions>
>
> </grid>
>
>
>
> <tileset name="bccache">
>
> <source>bcmaps</source>
>
> <cache>cache_sqlite</cache>
>
> <grid>bcgrid</grid>
>
> <format>JPEG</format>
>
> <metatile>4 4</metatile>
>
> <metabuffer>0</metabuffer>
>
> <expires>31536000</expires>
>
> </tileset>
>
>
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
>
> 227 Shepherds Bush Road London W6 7AS
>
> 020 8846 3220
>
>
>
> Web: http://www.BuchananComputing.co.uk
> <http://www.buchanancomputing.co.uk/>
>
> For general or urgent enquiries please write to:
> *supp...@buchanancomputing.co.uk
> <supp...@buchanancomputing.co.uk>*
>
>
>
> Buchanan Computing Ltd. Registered at the above office, no. 3274424
> (England & Wales)
>
>
>
> This message is intended solely for the individual/entity to whom it is
> addressed. It may contain confidential or legally privileged information.
> Any unauthorised disclosure or copying is prohibited and my be unlawful.
> If you have received this communication in error, please notify the sender
> immediately and delete it from your system.
>
>
>
> *From:* Sebastiano Laini
> *Sent:* 17 October 2018 13:39
> *To:* 'mapserver-users@lists.osgeo.org' <mapserver-users@lists.osgeo.org>
> *Cc:* 'Rahkonen Jukka (MML)' <jukka.rahko...@maanmittauslaitos.fi>
> *Subject:* RE: mapcache fastcgi
>
>
>
> Hi,
>
>
>
> It appears in both ways, when I’m seeding and when is requested from a
> pre-seeded cache, today I can’t because I need to move the dev domain in
> another server but tomorrow I will pass you the link to the 2 cache so you
> can see the difference.
>
>
>
> Tonight I will let it seed the whole area.
>
>
>
> The cache is on a normal HDD, I think all out servers are 10000 RPM, and
> the test was done on a brand new server so the disk is new.
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
>
> *From:* Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi
> <jukka.rahko...@maanmittauslaitos.fi>]
> *Sent:* 17 October 2018 13:35
> *To:* Sebastiano Laini <sebastiano.la...@buchanancomputing.co.uk>; '
> mapserver-users@lists.osgeo.org' <mapserver-users@lists.osgeo.org>
> *Subject:* RE: mapcache fastcgi
>
>
>
> Hi,
>
>
>
> In which way does the slowness appear? Is it when seeding or when serving
> from a pre-seeded cache? Can you give metrics about the difference in speed?
>
>
>
> In this comparison the differences were rather small
> http://blog.sogeo.services/blog/2016/07/01/mapcache-disk-vs-sqlite.html
>
>
>
> Also by the documentation the speed should be only “a bit slower”
>
>
>
> The SQLite based caches are a bit slower than the disk based caches, and
> may have write-locking issues at seed time if a high number of threads all
> try to insert new tiles concurrently.
>
>
>
> You wrote that SQLite cache is very slow for you which makes me think that
> there is something sub-optimal in your installation. First thing to check
> is if the SQLite database is on a fast disk.
> If SQLite cache is slow then MBTiles cache should be slow as well but I
> would make a test to be sure.
>
>
>
> -Jukka Rahkonen-
>
>
>
>
>
> *Lähettäjä:* Sebastiano Laini [
> mailto:sebastiano.la...@buchanancomputing.co.uk
> <sebastiano.la...@buchanancomputing.co.uk>]
> *Lähetetty:* 17. lokakuuta 2018 15:00
> *Vastaanottaja:* 'mapserver-users@lists.osgeo.org' <
> mapserver-users@lists.osgeo.org>
> *Kopio:* Rahkonen Jukka (MML) <jukka.rahko...@maanmittauslaitos.fi>
> *Aihe:* RE: mapcache fastcgi
>
>
>
> Is there a way to improve the speed and match the speed of the sqLite3
> cache with the disk one?
>
>
>
> We have 80 websites connected to MapCache and when I tried the sqLite3 I
> was testing it just on one website and on a complete different server with
> nothing in it.
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
>
> *From:* Sebastiano Laini
> *Sent:* 17 October 2018 12:43
> *To:* 'mapserver-users@lists.osgeo.org' <mapserver-users@lists.osgeo.org>
> *Cc:* 'Rahkonen Jukka (MML)' <jukka.rahko...@maanmittauslaitos.fi>
> *Subject:* RE: mapcache fastcgi
>
>
>
> Still it redirect you to that link, and btw it seems that mapcache can be
> used also as a fastCGI.
>
>
>
>
> https://mapserver.org/mapcache/install.html#cgi-fastcgi-specific-instructions
>
>
>
> Though is true that I’m reading now in the CMaker Builder section
>
>
>
> MapCache can run as a FastCGI executable. Note that the overhead of
> FastCGI is non-negligible with respect to the throughput you may obtain
> with a native Apache module. The FastCGI build is less tested, and may lag
> behind the Apache module version on some minor details. YMMV.
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
>
>
>
> *From:* Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi
> <jukka.rahko...@maanmittauslaitos.fi>]
> *Sent:* 17 October 2018 12:39
> *To:* Sebastiano Laini <sebastiano.la...@buchanancomputing.co.uk>; '
> mapserver-users@lists.osgeo.org' <mapserver-users@lists.osgeo.org>
> *Subject:* VS: mapcache fastcgi
>
>
>
> Hi,
>
>
>
> That documentation is about Mapserver, Mapcache is another software. It
> does belong to the same Mapserver but is not the same.
>
>
>
> MapCache is documented here:
> https://mapserver.org/mapcache/index.html#mapcache
>
>
>
> -Jukka Rahkonen-
>
>
>
> *Lähettäjä:* Sebastiano Laini [
> mailto:sebastiano.la...@buchanancomputing.co.uk
> <sebastiano.la...@buchanancomputing.co.uk>]
> *Lähetetty:* 17. lokakuuta 2018 14:33
> *Vastaanottaja:* 'mapserver-users@lists.osgeo.org' <
> mapserver-users@lists.osgeo.org>
> *Kopio:* Rahkonen Jukka (MML) <jukka.rahko...@maanmittauslaitos.fi>
> *Aihe:* RE: mapcache fastcgi
>
>
>
> Well, for what I can read in the documentation of mapserver:
>
> https://mapserver.org/optimization/fastcgi.html
>
> FastCGI is a protocol for keeping cgi-bin style web applications running
> as a daemon to take advantage of preserving memory caches, and amortizing
> other high startup costs (like heavy database connections) over many
> requests.
>
>
>
> It seems that using mapcache as fastCGI over apache modules it that should
> improve speed also.
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
>
>
>
> *From:* Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi
> <jukka.rahko...@maanmittauslaitos.fi>]
> *Sent:* 17 October 2018 12:30
> *To:* Sebastiano Laini <sebastiano.la...@buchanancomputing.co.uk>; '
> mapserver-users@lists.osgeo.org' <mapserver-users@lists.osgeo.org>
> *Subject:* Re: mapcache fastcgi
>
>
>
> Hi,
>
>
>
> Could you explain your plan with some more details? What I have understood
> is that you have experienced that SQLite cache is slower than disk cache in
> your environment.
>
>
>
> What I do not understand is what you are going to reach by running
> Mapcache as fastcgi instead of running it as Apache module which should be
> the faster option, and if this has some connection with the SQLite cache
> vs. disk cache comparison. Do you guess that fastcgi woud make SQLite cache
> faster?
>
>
>
> -Jukka Rahkonen-
>
>
>
>
>
> *Lähettäjä:* mapserver-users [
> mailto:mapserver-users-boun...@lists.osgeo.org
> <mapserver-users-boun...@lists.osgeo.org>] *Puolesta *Sebastiano Laini
> *Lähetetty:* 17. lokakuuta 2018 12:16
> *Vastaanottaja:* 'mapserver-users@lists.osgeo.org' <
> mapserver-users@lists.osgeo.org>
> *Aihe:* [mapserver-users] mapcache fastcgi
>
>
>
> Hi all,
>
>
>
> In the end I tried MapCache with sqlite3 cache but it’s very slow compared
> with the disk cache, so I want to setup MapCache with fastCGI on CentOS 7.
>
>
>
> I don’t understand what I need to install as the fastCGI link in the
> documentation is broken and I don’t know if is up to date.
>
>
>
> I already have apache fgcid installed from default in the httpd folder and
> I use PHP in fastCGI in all the domains, so I’m sure that fastCGI it works,
> though, I still need the fastCGI library? How can I install it in CentOS 7?
>
>
>
> In my apache conf I’ve already added the lines to load the mapcache module
> and the mapcache.xml file
>
>
>
> LoadModule mapcache_module modules/mod_mapcache.so
>
>
>
> <IfModule mapcache_module>
>
> <Directory /path/to/directory>
>
> Require all granted
>
> </Directory>
>
> MapCacheAlias /mapcache "/path/to/directory/mapcache.xml"
>
> </IfModule>
>
>
>
>
>
> I need to leave the previous configuration and add these new lines in the
> conf files?
>
>
>
> FcgidInitialEnv "MAPCACHE_CONFIG_FILE" "/path/to/mapcache/mapcache.xml
>
>
>
> <IfModule mod_fcgid.c>
>
> IPCCommTimeout 120
>
> MaxProcessCount 10
>
> FcgidInitialEnv "MAPCACHE_CONFIG_FILE" "/path/to/mapcache/mapcache.xml"
>
> <Location /map.fcgi>
>
> Order Allow,Deny
>
> Allow from all
>
> SetHandler fcgid-script
>
> </Location>
>
> ScriptAlias /map.fcgi "/path/to/mapcache/src/mapcache"
>
> </IfModule>
>
>
>
>
>
> And then instead of http://myserver/mapcache I need to use
> http://myserver/fcgi-bin to access mapcache fcgi?
>
>
>
> Regards,
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
> _______________________________________________
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
_______________________________________________
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
