[mapserver-users] mapserver cgi-bin security
Hi All,
We submitted our new system to a pen-test and apparently is pretty easy to be
infected by SQL-injection, anyone came across this before?
We run the feature layer through a cgi-bin request, see the script below
#! /bin/sh
MAPSERV="/var/.../cgi-bin/mapserv"
MAPFILE="/var/.../twfeat.map"
if [ "${REQUEST_METHOD}" = "GET" ]; then
if [ -z "${QUERY_STRING}" ]; then
QUERY_STRING="map=${MAPFILE}"
else
QUERY_STRING="map=${MAPFILE}&${QUERY_STRING}"
fi
exec ${MAPSERV}
else
echo "Sorry, I only understand GET requests."
fi
exit 1
how can you sanitize or implement security layers?
Kind Regards,
Sebastiano Laini
Web Developer
Buchanan Computing
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] MapCache clusters
Is not just the MapCache instances but also MapServer instances included so in theory you have X MapServers accepting request depending on the number of instances. Making tests today I discover that if we use just Vector and Raster layer I can reach 100 tiles/s in a VM with 8core and 8GB RAM even though the ram is under 1GB of use. A 4core reach the same speed, and I was using –p 5 and 10 respectively. Yesterday I came across this article https://www.maptiler.com/blog/2018/04/maps-with-docker.html and I was wondering if was something possible also with MapServer/MapCache. Again if we use just Vector and Raster I think I can seed in 15 days instead of more than 60 days but is still a lot because I have 4 cache to seed twice a year. Sebastiano Laini Web Developer Buchanan Computing From: Travis Kirstine [mailto:traviskirst...@gmail.com] Sent: 19 November 2019 16:11 To: Sebastiano Laini Cc: mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] MapCache clusters I don't think multiple instances of MapCache will help for seeding, in general most of the seeding time depends on the zoom level, with each higher zoom level taking 4 x the time as the proceeding zoom level (e.g. zoom 20 take 4 x the time to seed as zoom 19) . This can vary depending on your source service as there may be certain map scales / zoom levels that take longer per tile to render due to expression, labels, # of features etc... If you were to seed two levels for each mapcache instance you still would be waiting for the higher zoom levels to complete. Using multiple load balanced mapserver nodes across multiple VM may improve things a bit however if all the VMs are hosted on the same physical server this may be slower overall. For seeding I have heard you can use 2x the number of cores so you could set up a single mapcache server using -n 16 pointing at the load balanced source. Regards On Mon, 18 Nov 2019 at 10:47, Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>> wrote: Hi All, I was wondering if is possible to dockerise my server and then use multiple cluster to seed the cache? I know I can use multiple processes with the –n and –p flag though I’m wondering there is a limit on how much you can really increase those number. E.g. I dockerise the server and I spawn a 8core VM with 16GB ram, I can use sure –p 8 or maybe 10 but still I think that will require like 60 days to seed the whole UK with our data, can I spawn another machine and speed up the process? For example another 8core VM. We have 4 caches with different style that are updated twice a year so if it takes me 60 days for each by the time I finish it’s already time to start again. I was wondering maybe I can run multiple VM and seed one or maybe 2 levels on each VM or if I spawn a new VM the original process will take advantage of the new VM? In a VM I’m going to have MapServer and MapCache and I will use a subdomain under a load balancer just to seed while the domain will be use just for the seeded cache and in the same way will be under a load balancer. Regards, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org> https://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] MapCache clusters
Hi All, I was wondering if is possible to dockerise my server and then use multiple cluster to seed the cache? I know I can use multiple processes with the -n and -p flag though I'm wondering there is a limit on how much you can really increase those number. E.g. I dockerise the server and I spawn a 8core VM with 16GB ram, I can use sure -p 8 or maybe 10 but still I think that will require like 60 days to seed the whole UK with our data, can I spawn another machine and speed up the process? For example another 8core VM. We have 4 caches with different style that are updated twice a year so if it takes me 60 days for each by the time I finish it's already time to start again. I was wondering maybe I can run multiple VM and seed one or maybe 2 levels on each VM or if I spawn a new VM the original process will take advantage of the new VM? In a VM I'm going to have MapServer and MapCache and I will use a subdomain under a load balancer just to seed while the domain will be use just for the seeded cache and in the same way will be under a load balancer. Regards, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache seed speed optimization
I manage to seed at 300 tiles/s in the less complex layers, now I need to speak with the technical director and see how we are going to optimize our data. I was trying to test MapProxy but I couldn’t install it in our Centos 7 machine, I may give it a try again. For your experience which one is the fastest one? Not just seeding but in delivering the requests also. What tiles size are you using normally? Are you still using HTTP1 or HTTP2? Regards, Sebastiano Laini Web Developer Buchanan Computing From: Travis Kirstine [mailto:traviskirst...@gmail.com] Sent: 11 November 2019 14:42 To: Sebastiano Laini Cc: Fawcett, David (MNIT) ; Rahkonen Jukka (MML) ; mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] mapcache seed speed optimization I may not be unrealistic for it take mapserver to 4-6 seconds to generate 4096px image depending on the amount data being rendered and the complexity of your styles / expressions / labels. If your mapserver is able to generate a 1024x1024 in under a second that is pretty good. Like Jukka and other suggest I would look at translating the files to shp files with spatial indexes or postgis tables with spatial and attribute indexes if necessary. Preprocessing the data (simplification / prefiltering / point thinning cluster) and using SCALETOKENs to reference different source layers at different scales can lead to big performance boosts at the mapserver end. For us the biggest limiting factor has been the write blocking to sqlite cache when seeding tiles, if you increase the number of threads / process in mapcache seed those process will just end up waiting to write to the cache. You can generally figure out what the max number of processes the cache can handle by running a seed on a test area and look at the number of tiles seeded per second, as some point -n will have no effect which your process are just waiting to write the cache. I would look at using this method to test sqlite vs disk or other backends. The GeoTiff cache may be promising as it appears to be non-blocking but experimental... We use a riak cache with a riak leveldb backend which works well for us but is a bit of a pain to manage as it difficult to delete objects so it's not great for caches that need to be refreshed. Finally you may want to test MapProxy as an alternative to MapCache (we use both). MapProxy supports a non-blocking compact cache that could solve your inode problem Regards On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>> wrote: We don’t create the data, we rely on OS (ordnance survey) to supply us the maps files and then we publish them in our service but I assume that probably we will need to create some flow to improve it or download it in other format that is faster. Sebastiano Laini Web Developer Buchanan Computing From: Fawcett, David (MNIT) [mailto:david.fawc...@state.mn.us<mailto:david.fawc...@state.mn.us>] Sent: 08 November 2019 16:41 To: Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'Rahkonen Jukka (MML)' mailto:jukka.rahko...@maanmittauslaitos.fi>>; 'mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org>' mailto:mapserver-users@lists.osgeo.org>> Subject: RE: mapcache seed speed optimization For data formats that are slower to read, would it add fit your workflow to convert it to different data format before creating the tiles? ___ mapserver-users mailing list mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org> https://lists.osgeo.org/mailman/listinfo/mapserver-users ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache seed speed optimization
We don't create the data, we rely on OS (ordnance survey) to supply us the maps files and then we publish them in our service but I assume that probably we will need to create some flow to improve it or download it in other format that is faster. Sebastiano Laini Web Developer Buchanan Computing From: Fawcett, David (MNIT) [mailto:david.fawc...@state.mn.us] Sent: 08 November 2019 16:41 To: Sebastiano Laini ; 'Rahkonen Jukka (MML)' ; 'mapserver-users@lists.osgeo.org' Subject: RE: mapcache seed speed optimization For data formats that are slower to read, would it add fit your workflow to convert it to different data format before creating the tiles? ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache seed speed optimization
Thanks, I'll take a look at it :), What should be maxsize to seed? Before was 2084 with 8 8 metatiles now is 4096 with 16 16 metatiles I found some of the problematic layers, some of them take 4 to 6 seconds in TAB files, while the same layer in raster take 40ms, we just need to find another way to style it. Sebastiano Laini Web Developer Buchanan Computing From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 08 November 2019 15:03 To: Sebastiano Laini ; 'mapserver-users@lists.osgeo.org' Subject: VS: mapcache seed speed optimization Hi, You can see examples about how to debug and improve the rendering speed with Mapserver in the presentation of Lars Schylberg https://media.ccc.de/v/bucharest-271-mapserver-optimize-for-performance. Your maps look pretty and I hope you will find some bottleneck layers so you can make them faster. -Jukka- Lähettäjä: Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>> Lähetetty: perjantai 8. marraskuuta 2019 16.30 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Kopio: Rahkonen Jukka (MML) mailto:jukka.rahko...@maanmittauslaitos.fi>>; 'Yves Jacolin' mailto:yves.jaco...@camptocamp.com>> Aihe: RE: mapcache seed speed optimization My MapServer is on the same server as MapCache and my network is pretty fast. I was just trying to experiment with the -n and -p value and checking the CPU load and I didn't seen any difference, but now that I read the responses I'm trying -n 6 and the CPU and the RAM is more in use now and seems to seed faster than before. I reach that numbers because the last time I was seeding with -n 10 was incredibly slow so I started to increase the numbers. Now in HTOP I can actually see 6 mapcache processes and not just 2 Sadly to seed the whole UK in disk cache we run out of inodes, I would like to use SSD. About our data we use a mix of everything, here the final result https://www.buchananmapping.co.uk/ We use raster, vector and tab files; the reason I can't move out from mapserver are mainly the mapinfo tables (tab), possibly those tab files are the slowest layers Even though some of the vector layers are very huge, some of them the data is about 21GB. I'll try to debug layer by layer through MapServer and strip down to the minimum or replace some layers. Lastly, the last time I've tried to seed the whole UK it took me more than a month and I simply stop it because was unacceptable to seed 7 levels of resolutions 70, 28, 14, 7, 2.8, 1.4, 0.7 that correspond to these scales 2500, 5000, 1, 25000, 5, 10, 25, 50 Sebastiano Laini Web Developer From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 08 November 2019 14:02 To: Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Subject: Re: mapcache seed speed optimization Hi, When you seed tiles from WMS the slowest component is usually the WMS server. And what the WMS server is spending its time is usually rendering. And what makes rendering slow is usually that you render too much, and sometimes that WMS server gets the vector data too slow because the backend data source is slow, very often because it has poor indexes which do not for suit the queries that WMS server is sending. If you have a slow WMS you can't get more speed out of it by bombing it with very many parallel threads. I would have a try if using 2-4, perhaps 8 threads makes seeding faster but if there is no difference I would concentrate on the WMS configuration. BTW. why did you select to use 2 threads? I am not sure why you see just 1 core running on 100% load. I guess that you are now running WMS and MapCache on the same computer. Mapserver program is using only 1 core but http server should start more processes for parallel requests. That CPU load is at 100% is generally a good thing because it tells that CPU is working, instead of waiting data from a slow disk. Another thing is that CPU may be doing something heavy and stupid. Myself I would start by gathering some information about how Mapserver WMS is working be setting "DEBUG 2" on each layer or for the whole mapfile. By looking at the rendering times of each layer the slowest layers can be found and then it is time to think about how to make those layers faster. You wrote that you use SQLite cache so it would be good to know that SQLite database does not support parallel writes https://sqlite.org/lockingv3.html. Seeding with a few parallel threads may increase the speed if the WMS is the slowest part in your chain but when you have saturated the SQLite writing speed adding threads will do no good. If you have optimized your WMS so well that SQLite sets the limits then you should
Re: [mapserver-users] mapcache seed speed optimization
My MapServer is on the same server as MapCache and my network is pretty fast. I was just trying to experiment with the -n and -p value and checking the CPU load and I didn't seen any difference, but now that I read the responses I'm trying -n 6 and the CPU and the RAM is more in use now and seems to seed faster than before. I reach that numbers because the last time I was seeding with -n 10 was incredibly slow so I started to increase the numbers. Now in HTOP I can actually see 6 mapcache processes and not just 2 Sadly to seed the whole UK in disk cache we run out of inodes, I would like to use SSD. About our data we use a mix of everything, here the final result https://www.buchananmapping.co.uk/ We use raster, vector and tab files; the reason I can't move out from mapserver are mainly the mapinfo tables (tab), possibly those tab files are the slowest layers Even though some of the vector layers are very huge, some of them the data is about 21GB. I'll try to debug layer by layer through MapServer and strip down to the minimum or replace some layers. Lastly, the last time I've tried to seed the whole UK it took me more than a month and I simply stop it because was unacceptable to seed 7 levels of resolutions 70, 28, 14, 7, 2.8, 1.4, 0.7 that correspond to these scales 2500, 5000, 1, 25000, 5, 10, 25, 50 Sebastiano Laini Web Developer From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 08 November 2019 14:02 To: Sebastiano Laini ; 'mapserver-users@lists.osgeo.org' Subject: Re: mapcache seed speed optimization Hi, When you seed tiles from WMS the slowest component is usually the WMS server. And what the WMS server is spending its time is usually rendering. And what makes rendering slow is usually that you render too much, and sometimes that WMS server gets the vector data too slow because the backend data source is slow, very often because it has poor indexes which do not for suit the queries that WMS server is sending. If you have a slow WMS you can't get more speed out of it by bombing it with very many parallel threads. I would have a try if using 2-4, perhaps 8 threads makes seeding faster but if there is no difference I would concentrate on the WMS configuration. BTW. why did you select to use 2 threads? I am not sure why you see just 1 core running on 100% load. I guess that you are now running WMS and MapCache on the same computer. Mapserver program is using only 1 core but http server should start more processes for parallel requests. That CPU load is at 100% is generally a good thing because it tells that CPU is working, instead of waiting data from a slow disk. Another thing is that CPU may be doing something heavy and stupid. Myself I would start by gathering some information about how Mapserver WMS is working be setting "DEBUG 2" on each layer or for the whole mapfile. By looking at the rendering times of each layer the slowest layers can be found and then it is time to think about how to make those layers faster. You wrote that you use SQLite cache so it would be good to know that SQLite database does not support parallel writes https://sqlite.org/lockingv3.html. Seeding with a few parallel threads may increase the speed if the WMS is the slowest part in your chain but when you have saturated the SQLite writing speed adding threads will do no good. If you have optimized your WMS so well that SQLite sets the limits then you should test other cache alternatives. I guess that file cache would be faster to seed from multiple threads. When is comes to ticket https://github.com/mapserver/mapcache/issues/169 the latest comment is from a Mpaserver developer and their last words "I wouldn't mind helping to fix if there were interested parties..." feel rather polite to me. -Jukka Rahkonen- Lähettäjä: mapserver-users mailto:mapserver-users-boun...@lists.osgeo.org>> Puolesta Sebastiano Laini Lähetetty: perjantai 8. marraskuuta 2019 14.35 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Aihe: [mapserver-users] mapcache seed speed optimization Hi all, I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache due to the input source of our maps, so I'm trying to improve the speed of the mapserver/mapcache stack and what I cannot improve is the seed speed. The -n and -p parameters seems to be useless. -n | -nthreads: number of parallel threads that should be used to request tiles from the WMS source. The default is 1, but can be set higher if the WMS server can withstand parallel requests. (As a rule of thumb, the value chosen here should never be much higher than the number of CPUs on the WMS server.) -p | -nprocesses: number of parallel processes that should be used to request tiles from the WMS source. I've tried
[mapserver-users] mapcache seed speed optimization
Hi all,
I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache
due to the input source of our maps, so I'm trying to improve the speed of the
mapserver/mapcache stack and what I cannot improve is the seed speed.
The -n and -p parameters seems to be useless.
-n | -nthreads: number of parallel threads that should be used to request tiles
from the WMS source. The default is 1, but can be set higher if the WMS server
can withstand parallel requests. (As a rule of thumb, the value chosen here
should never be much higher than the number of CPUs on the WMS server.)
-p | -nprocesses: number of parallel processes that should be used to request
tiles from the WMS source.
I've tried to use -n 2 and -p 6 but for some reason the CPU load is max
100% in 1 core, no way to make it use more power and speed up the seed process.
Max memory used is 470MB between Centos 7 (with all the services running) and
the seed process.
I've setup apache to use HTT/2 and it make quite a difference when the cache is
already seeded but during the seed process doesn't seems to be affected.
My MapServer build is:
./configure \
--with-ogr=/usr/local/bin/gdal-config \
--with-gdal=/usr/local/bin/gdal-config \
--with-wfsclient \
--with-wmsclient \
--with-wfs \
--enable-debug \
--with-curl-config=/usr/bin/curl-config \
--with-proj \
--with-jpeg \
--with-freetype \
--with-postgis=/usr/pgsql-9.3/bin/pg_config \
--with-geos=/usr/local/bin/geos-config
I know there is mapserver 7 and yet I'm using 6 for the test.
While the MapCache is:
cmake -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \
-DWITH_SQLITE=1 \
-DWITH_BERKELEY_DB=0 \
-DWITH_TIFF=0 \
-DWITH_GEOTIFF=0 \
-DWITH_FCGI=0 \
-DWITH_PCRE=0 \
-DWITH_PIXMAN=1 \
-DWITH_OGR=1 \
-DWITH_GEOS=1 \
../
MapCache is the latest version and this is my mapcache.xml file, we use sqlite
as a cache method
/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3
1
1
1573741823
bcmaps
cache_sqlite
bcgrid
PNG
16 16
0
2628000
BC custom grid
EPSG:27700
256 256
0 0 70 125
70 28 14 7 2.8 1.4 0.7
best
assemble
bilinear
mypng
4096
I've also tried to use -DWITH_MAPSERVER but seems that the integration is not
working and they doen't care? https://github.com/mapserver/mapcache/issues/169
this issue was open 2 years ago and still open, not even a reply from the
developers.
How can I improve the seed process?
Should I increase the maxsize and metatile to requests more tiles and cache
faster?
Regards,
Sebastiano Laini
Web Developer
Buchanan Computing
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] join filter
Hi all,
Is possible to create a filter based on multiple tables?
This is my current filter:
DATA "wkb_geometry from london.ordersp"
FILTER "date_advert <= '%DTO%' and date_close >= '%DTO%' and
visible and scheme_id = %SCHEMEID%"
I have also a london.schemes table with the same column that I can modify
through an admin page so I can choose with SCHEME is visible and change the
dates but I'm updating just the london.schemes table and I'm not iterating
through all the possible tables with some data and modify the dates/visible for
each item in it.
Is there any way to let mapserver check the "date_advert" "date_close" and
visible in the london.scheme table where the scheme_id =%SCHEMEID% ?
This is an example of my data:
london.scheme
scheme_id: 2000
date_advert: 2019-01-01
date_close 2019-12-31
visible: true
london.ordersr
name: item2
scheme_id: 2000
date_advert: 2019-01-01
date_close 2019-12-31
visible: true
As you can see I have duplicated columns and ideally I should get rid of the
dates and visible column in the orders{r/l/p,etc} tables and check just the
scheme table
Sebastiano Laini
Web Developer
Buchanan Computing
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Cannot compile mapserver 7 with postgis
Hi Jonas, now it works with PostGis support. I suppose that the problems was in the -DCMAKE_PREFIX_PATH="/usr/lib64;/usr/pgsql-11;/usr/lib64/oracle/12.1/client64/sdk/include" \ I was using : without double quotes around the full string and you use double quotes and semicolons to split the paths. Thanks. Sebastiano Laini Web Developer Buchanan Computing From: Jonas Lund Nielsen [mailto:jo...@sdfe.dk] Sent: 19 June 2019 10:11 To: Sebastiano Laini ; 'mapserver-users@lists.osgeo.org' Subject: SV: [mapserver-users] Cannot compile mapserver 7 with postgis I have had success compiling MapServer 7.4 on RHEL7 as noted below. I guess it should be somewhat similar to CentOS. Just remove whatever Oracle-specific and other options you might not need. I don't think MapServer has PostGIS as a direct build-dependency. It is unclear to me from your post, whether you have a previous build working with an earlier version, and the problem just occurred with an update to proj 6? So please bear in mind that this build was relying on proj 5.2 and gdal 2.4. I have installed "postgresql11.x86_64", "postgresql11-devel.x86_64" and "postgresql11-libs.x86_64" from the postgres yum repository: https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-redhat11-11-2.noarch.rpm yum install pgdg-redhat11-11-2.noarch.rpm (installs the repository, not the above three packages) cmake3 -DINSTALL_LIB_DIR:PATH=/usr/lib64 \ -DCMAKE_PREFIX_PATH="/usr/lib64;/usr/pgsql-11;/usr/lib64/oracle/12.1/client64/sdk/include" \ -DWITH_WMS=ON \ -DWITH_WFS=ON \ -DWITH_WCS=ON \ -DWITH_SOS=ON \ -DWITH_KML=ON \ -DWITH_CLIENT_WFS=ON \ -DWITH_CLIENT_WMS=ON \ -DWITH_CURL=ON \ -DWITH_PROJ=ON \ -DWITH_GEOS=ON \ -DWITH_ICONV=ON \ -DWITH_LIBXML2=ON \ -DWITH_OGR=ON \ -DWITH_GDAL=ON \ -DWITH_GIF=ON \ -DWITH_CAIRO=OFF \ -DWITH_RSVG=OFF\ -DWITH_POSTGIS=ON \ -DWITH_ORACLESPATIAL=ON \ -DWITH_FCGI=ON \ -DWITH_THREAD_SAFETY=OFF \ -DWITH_PYTHON=OFF \ -DWITH_PERL=OFF \ -DWITH_JAVA=OFF \ -DWITH_PHP=OFF \ -DWITH_APACHE_MODULE=OFF \ -DWITH_FRIBIDI=OFF \ -DWITH_HARFBUZZ=OFF \ -DWITH_PROTOBUFC=ON .. Best regards /Jonas Fra: mapserver-users mailto:mapserver-users-boun...@lists.osgeo.org>> På vegne af Sebastiano Laini Sendt: 18. juni 2019 15:24 Til: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Emne: [mapserver-users] Cannot compile mapserver 7 with postgis For the version that I use I need Proj6 I don't really understand why I can compile other software correctly and MapServer no, I don't even understand what should I put https://github.com/Kitware/CMake/blob/master/Modules/FindPostgreSQL.cmake this is the first error that appear, Cmake Warning at cmake/FindPostgreSQL.cmake:21 (message) Pg_config not found, will try some defaults Call stacks (most recent call first): cMakeLists.txt:755 (find_package) and the second one is: cMake error at CmakeLists.txt:73 (messages) POSTGIS library/components/dependency could not be found. HINTS: -disable postgis... - add cmake_prefix_path... Call stack (most recent call firsts) CMakeLists.txt:590 (report_optional_not_found) Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Cannot compile mapserver 7 with postgis
For the version that I use I need Proj6 I don't really understand why I can compile other software correctly and MapServer no, I don't even understand what should I put https://github.com/Kitware/CMake/blob/master/Modules/FindPostgreSQL.cmake this is the first error that appear, Cmake Warning at cmake/FindPostgreSQL.cmake:21 (message) Pg_config not found, will try some defaults Call stacks (most recent call first): cMakeLists.txt:755 (find_package) and the second one is: cMake error at CmakeLists.txt:73 (messages) POSTGIS library/components/dependency could not be found. HINTS: -disable postgis... - add cmake_prefix_path... Call stack (most recent call firsts) CMakeLists.txt:590 (report_optional_not_found) Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Cannot compile mapserver 7 with postgis
And yes I have PostgreSQL 10 / PostgreSQL 10 server and PostgreSQL 10 devel installed, there is no such PostgreSQL 10 server devel for centos 7 Those comes directly from the PostgreSQL download page: postgresql-client libraries and client binaries postgresql-server core database server postgresql-contribadditional supplied modules postgresql-devel libraries and headers for C language development Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Cannot compile mapserver 7 with postgis
I've added the 2 lines in the cmake but nothing to do, my file is in /usr/pgsql-10/lib/libpq.so maybe because I'm on centos 7 cmake -DCMAKE_INSTALL_PREFIX=/opt \ -DCMAKE_PREFIX_PATH=/usr/pgsql-10:/usr/local/src/postgis-2.5.2:/usr/local:/opt:/usr/local/src \ -DWITH_CLIENT_WFS=ON \ -DWITH_CLIENT_WMS=ON \ -DPOSTGRESQL_LIBRARY=/usr/pgsql-10/lib/libpq.so \ -DWITH_POSTGIS=ON \ -DWITH_CURL=ON \ -DWITH_SOS=ON \ -DWITH_FCGI=0 \ -DWITH_PERL=0 \ -DWITH_RUBY=0 \ -DWITH_JAVA=0 \ -DWITH_CSHARP=0 \ -DWITH_PYTHON=0 \ -DWITH_SVGCAIRO=0 \ -DWITH_FRIBIDI=0 \ -DWITH_PROTOBUFC=0 \ -DWITH_CAIRO=0 \ -DWITH_HARFBUZZ=0 \ -DWITH_ORACLESPATIAL=0 \ -DWITH_MSSQL2008=0 \ ../ >../configure.out.txt Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Cannot compile mapserver 7 with postgis
Hi all, I'm trying to compile MapServer 7 with: Proj 6 Geos 3.7 Gdal 3.0 PostGis 2.5 Postgres 10 But for some reason it complaining about pg_config not found, I usually use /usr/pgsql-10/bin/pg_config to compile software that need the same file but with MapServer 7 I can't Here my cmake : cmake -DCMAKE_INSTALL_PREFIX=/opt \ -DCMAKE_PREFIX_PATH=/usr/pgsql-10/bin:/usr/local/src/postgis-2.5.2:/usr/local:/opt:/usr/local/src \ -DWITH_CLIENT_WFS=ON \ -DWITH_CLIENT_WMS=ON \ -DWITH_CURL=ON \ -DWITH_SOS=ON \ -DWITH_FCGI=0 \ -DWITH_PERL=0 \ -DWITH_RUBY=0 \ -DWITH_JAVA=0 \ -DWITH_CSHARP=0 \ -DWITH_PYTHON=0 \ -DWITH_SVGCAIRO=0 \ -DWITH_FRIBIDI=0 \ -DWITH_PROTOBUFC=0 \ -DWITH_CAIRO=0 \ -DWITH_HARFBUZZ=0 \ -DWITH_ORACLESPATIAL=0 \ -DWITH_MSSQL2008=0 \ ../ >../configure.out.txt How can I compile it with PostGis support? Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] mapserver 6.2 unexpected behavior or not?
Yesterday I came across this section https://mapserver.org/optimization/mapfile.html#layers and I don't understand the status behaviour and MIN/MAXSCALEDENOM it say: For every layer in a Map File that has a status of ON or DEFAULT, MapServer will load that layer and prepare it for display, even if that layer never gets displayed. Ok, so if I setup all layers to OFF and I use MIN/MAXSCALEDENOM they should be displayed at their zoom level? Because I tried but is not work as expected in a web. This is one of our layer LAYER NAME "YellowB" TYPE POLYGON MINSCALEDENOM 0 MAXSCALEDENOM 7500 STATUS DEFAULT CONNECTIONTYPE OGR CONNECTION "OpenData_Jan19/YellowBack.TAB" STYLEITEM "AUTO" CLASS NAME "YellowB" STYLE COLOR 0 0 0 END END PROJECTION "init=epsg:27700" END END Should I use the PROJECTION tag in each layer even if the layer itself it supposed to be in that projection? Sebastiano Laini Web Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] mapcache for website tips
Hi all, I'm curious to know what are your mapcache configurations for a use in a website. I'm using tiles of 128x128 and the maximum weight of a single tile can be 80KB, our map is counting 33 layers and a total of 60GB of data (whole UK EPSG:27700, Ireland is just the border and a white background). On first load with OpenLayer 4 and preload infinity (Preload. Load low-resolution tiles up to preload levels. 0 means no preloading) I load around 350 tiles and the load times is around 8 second (if the zoom layer is cached) I'm also seeding with metatile 4 4 that was the old standard used before I changed the tilesize, which are the best settings I should use for web (tile size and which should be the max file size) and seed caching? As you know in previous posts I'm having really big issue in seeding the whole projection and I'm trying to figure out how to improve the service. (bear in mind that we update the data 2 at year so I need to recache again every 6 months) Sebastiano Laini Web Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] mapserver/mapcache apache+nginx
Hi all, Following the maximum speed I can get from mapcache, I'm going to make some test in a new server with Nginx as a reverse proxy and Apache. How stable is mapcache 1.6.1 with Nginx and how should I setup mapcache. I know there are instruction but I don't understand some points. https://www.mapserver.org/mapcache/install.html#nginx-specific-instructions this are the instruction to build my actual setup cd /usr/local/src/ wget http://download.osgeo.org/mapserver/mapcache-1.6.1.tar.gz tar -xzvf mapcache-1.6.1.tar.gz rm -f mapcache-1.6.1.tar.gz cd mapcache-1.6.1 mkdir build cd build cmake -DCMAKE_PREFIX_PATH="usr/bin/sqlite3" \ -DWITH_SQLITE=1 \ -DWITH_BERKELEY_DB=0 \ -DWITH_TIFF=0 \ -DWITH_GEOTIFF=0 \ -DWITH_FCGI=0 \ -DWITH_PCRE=0 \ ../ Run make and then make install: make make install sudo ldconfig I add this to the httpd.conf: LoadModule mapcache_modulemodules/mod_mapcache.so Require all granted MapCacheAlias /mapcache "/usr/local/src/mapcache-1.6.1/build/mapcache.xml" And then I copy my mapcache.xml in the build directory and: sudo ldconfig service httpd restart If I want to use Nginx as a reverse proxy and speed up the cache I need to build it from scratch as in the instruction and change the configuration to use mapcache with Nginx? Or still you will use both of them? Can I use yum install nginx and then add the module? The part about The basic configuration without any proxying Where should be put? In the mapcache.xml? Is there any particular apache and Nginx config that I should use? My apache configuration is the default one in Plesk (at the moment mpm event) and in the test server (mpm prefork as they told me that it's better, though it's using the default configuration) Thanks, Sebastiano Laini Web Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache seed slow
We have 2 dedicated servers that we use to serve the full mapping system and when we update the map we switch the system to just one server and use the other one just to see the new map with a different URL. Sebastiano Laini Web Developer From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 22 February 2019 15:03 To: Sebastiano Laini ; 'mapserver-users@lists.osgeo.org' Subject: Re: mapcache seed slow Hi, Stupid question, but do you know how fast your WMS service is? How many GetMap requests/sec it can serve with the metatile size that you are seeding with? Have you some evidence that WMS is running idle while MapCache is busy with cutting the tiles and filling the cache? -Jukka Rahkonen- Lähettäjä: mapserver-users mailto:mapserver-users-boun...@lists.osgeo.org>> Puolesta Sebastiano Laini Lähetetty: perjantai 22. helmikuuta 2019 13.35 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Aihe: [mapserver-users] mapcache seed slow Hi all, I'm trying to seed the whole UK at all zoom lv with these resolutions 280 140 70 28 14 7 6.16 2.8 1.4 0.7 But after 13 days it arrives at the 5th level where before I kill the process it seed just 900MB on that level. This is my script su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c /usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 400" 2>&1> /var/www/vhosts/$SITE/scripts/cache_log.txt I know that in the documentation say that -n should be less or max the number of cpu in the server but is really slow if I put 2, so I tried with 100 and after 13 days that was the result so now I'm trying 400 Is there something else I can use to speed up the seed? I'm using the SQlite cache method Should I use the -p (| -nprocesses: number of parallel processes that should be used to request tiles from the WMS source) Instead of the -n parameter? Sebastiano Laini Web Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache seed slow
I don't know how fast is but I have a whole dedicated server just for do that with a development URL that I'm using just to develop, is still configured in HTTP/1, looking to make some test on a new server to setup HTTP/2 with NGINX and possibly WMTS. I'm seeding with a bash file and I can see (with TOP) that it's using about 60/80% CPU usage and other HTTPD requests make it from the APACHE user. Sebastiano Laini Web Developer From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 22 February 2019 15:03 To: Sebastiano Laini ; 'mapserver-users@lists.osgeo.org' Subject: Re: mapcache seed slow Hi, Stupid question, but do you know how fast your WMS service is? How many GetMap requests/sec it can serve with the metatile size that you are seeding with? Have you some evidence that WMS is running idle while MapCache is busy with cutting the tiles and filling the cache? -Jukka Rahkonen- Lähettäjä: mapserver-users mailto:mapserver-users-boun...@lists.osgeo.org>> Puolesta Sebastiano Laini Lähetetty: perjantai 22. helmikuuta 2019 13.35 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Aihe: [mapserver-users] mapcache seed slow Hi all, I'm trying to seed the whole UK at all zoom lv with these resolutions 280 140 70 28 14 7 6.16 2.8 1.4 0.7 But after 13 days it arrives at the 5th level where before I kill the process it seed just 900MB on that level. This is my script su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c /usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 400" 2>&1> /var/www/vhosts/$SITE/scripts/cache_log.txt I know that in the documentation say that -n should be less or max the number of cpu in the server but is really slow if I put 2, so I tried with 100 and after 13 days that was the result so now I'm trying 400 Is there something else I can use to speed up the seed? I'm using the SQlite cache method Should I use the -p (| -nprocesses: number of parallel processes that should be used to request tiles from the WMS source) Instead of the -n parameter? Sebastiano Laini Web Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache seed slow
I’m using this configuration
/var/www/vhosts/buchananmapping.co.uk/httpdocs/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3
1
1
1573741823
image/png
miniscale
…/maps/bcmaps.map
https://.../cgi-bin/mapserv?
bcmaps
cache_sqlite
bcgrid
PNG
4 4
0
2628000
BC custom grid
EPSG:27700
128 128
0 0 70 125
280 140 70 28 14 7 6.16 2.8 1.4 0.7
best
It supposed to have a limit that it’s the full EPSG extent
Sebastiano Laini
Web Developer
From: Travis Kirstine [mailto:traviskirst...@gmail.com]
Sent: 22 February 2019 13:57
To: Sebastiano Laini
Cc: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] mapcache seed slow
The time you have indicated seems excessive to seed the UK. My guess is that
the current process it seeding the extents of your grid (world?) but default.
I've seeded the province of ontario 1 million km2 z0-17 in about 2 weeks using
a sqlite cache
Ramping up the -n will only bottle neck the process more (same with the -p
flag) the limiting factor is SQLite and it limitation on concurrent writes (I
use -n 4). I've noticed that if you try to use other sqlite3 journal modes you
can degrade performance
(http://osgeo-org.1560.x6.nabble.com/mapcache-sqlite-cache-large-wal-files-and-blocked-checkpoints-when-seeding-td5377571.html).
I would just use most basic cache config as a starting point. BTW using SSD
should increase the SQLite writes as well (if you are lucky enough to have them)
./{tileset}.sqlite3
There are a few things that you could do when seeding to speed up the process:
Use the "restricted_extent" parameter in the tileset grid to limit the number
of tiles
Example
ontario_2018_wms
sqlite
gmaps
mixed
5 5
Use a polygon shapefile (or other ogr datasource) to limit the seeding using
the -d flag in mapcache_seed command (note the projection of the datasource
should be the same as the grid)
Use a larger metatile when seeding (-M flag override in command line).
Mapcache will "bulk" insert the resultant tiles based on the metatile - if
5 5 mapcache will insert 25 tiles at a time if 8
8 it will do 64 (I think this is the way it works). The limiting
factor here is trying to balance this with your source WMS performance. I use
8,8 when seeding
Regards
On Fri, 22 Feb 2019 at 06:35, Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>>
wrote:
Hi all,
I’m trying to seed the whole UK at all zoom lv with these resolutions
280 140 70 28 14 7 6.16 2.8 1.4 0.7
But after 13 days it arrives at the 5th level where before I kill the process
it seed just 900MB on that level.
This is my script
su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c
/usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 –n
400" 2>&1> /var/www/vhosts/$SITE/scripts/cache_log.txt
I know that in the documentation say that –n should be less or max the number
of cpu in the server but is really slow if I put 2, so I tried with 100 and
after 13 days that was the result so now I’m trying 400
Is there something else I can use to speed up the seed? I’m using the SQlite
cache method
Should I use the -p (| –nprocesses: number of parallel processes that should be
used to request tiles from the WMS source) Instead of the –n parameter?
Sebastiano Laini
Web Developer
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] mapcache seed slow
Hi all, I'm trying to seed the whole UK at all zoom lv with these resolutions 280 140 70 28 14 7 6.16 2.8 1.4 0.7 But after 13 days it arrives at the 5th level where before I kill the process it seed just 900MB on that level. This is my script su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c /usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 400" 2>&1> /var/www/vhosts/$SITE/scripts/cache_log.txt I know that in the documentation say that -n should be less or max the number of cpu in the server but is really slow if I put 2, so I tried with 100 and after 13 days that was the result so now I'm trying 400 Is there something else I can use to speed up the seed? I'm using the SQlite cache method Should I use the -p (| -nprocesses: number of parallel processes that should be used to request tiles from the WMS source) Instead of the -n parameter? Sebastiano Laini Web Developer ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] reserved words
Yes, I'm using brackets [fstatus] Thanks :) Sebastiano Laini Web Developer Buchanan Computing From: Sebastiano Laini Sent: 25 January 2019 08:34 To: 'mapserver-users@lists.osgeo.org' Subject: reserved words Hello, Is there some reserved words for the xml template that you can't use as a field? Yesterday in a XML template to have back data from the database there was a field called LEGEND, and that instead of giving back the column LEGEND from the database was populate with a linux path followed by an image that doesn't exist. I'm using Mapserver 6 with a .map file for the data layer and it calls an xml template to give back the info from the database. Regards, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] reserved words
Hello, Is there some reserved words for the xml template that you can't use as a field? Yesterday in a XML template to have back data from the database there was a field called LEGEND, and that instead of giving back the column LEGEND from the database was populate with a linux path followed by an image that doesn't exist. I'm using Mapserver 6 with a .map file for the data layer and it calls an xml template to give back the info from the database. Regards, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Mapserver limit watermark
Hi Richard, I saw it but is not enough, and we actually already have an “Attribution” copyright in the left bottom part (is a custom one). To fullfill the copyright statements we need the watermark, I just created a new layer with OL with bigger size and call it inside the app so I don’t need to add it on the cache. Cheers, Sebastiano Laini Web Developer Buchanan Computing From: Richard Greenwood [mailto:richard.greenw...@gmail.com] Sent: 28 November 2018 14:28 To: Sebastiano Laini Cc: mapserver Subject: Re: [mapserver-users] Mapserver limit watermark OpenLayers has "attribution" functions that allow you to place copyright information. So instead of putting the watermark on the images on the server you do it in the browser. On Tue, Nov 27, 2018 at 1:55 AM Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>> wrote: Thanks Steve, We do use mapcache but not for the watermark, that is on the mapserver request within the mastermap map. I will check properly if we do must need the watermark as part of the copyright or we can remove it, if we need it I will think about to insert it on the mapcache ☺ Cheers, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org> https://lists.osgeo.org/mailman/listinfo/mapserver-users -- Richard W. Greenwood, PLS www.greenwoodmap.com<http://www.greenwoodmap.com> ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Mapserver limit watermark
Thanks Steve, We do use mapcache but not for the watermark, that is on the mapserver request within the mastermap map. I will check properly if we do must need the watermark as part of the copyright or we can remove it, if we need it I will think about to insert it on the mapcache :) Cheers, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Mapserver limit watermark
Hi all, Since I rewritten the code and switch from OpenLayers 2 to OpenLayers 4 I also implemented a smaller raster tile request instead of the bigger one that they were using before, now the problem is that some clients use watermark and the map is full of watermarks as is present on each tile. Is there a way to limit the number of watermark presents in the map? This my watermark layer on the .MAP file LAYER NAME "credits" GROUP "g" STATUS DEFAULT TRANSFORM cc TRANSPARENCY 30 TYPE POINT FEATURE POINTS -0 -20 END END CLASS NAME 'oslabel' STYLE SYMBOL "copyright" COLOR 0 0 0 OUTLINECOLOR 0 0 0 END END END And here my copyright.sym SYMBOLSET SYMBOL NAME "copyright" TYPE PIXMAP IMAGE "watermark.gif" TRANSPARENT 8 END END Here a link where you can see the result: https://www.traffwebdemo.co.uk/parking/basic.html Regards, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] mapserver 6 mapcache 1.6 tileindex limit
Hi, Sorry for the delay, at the moment I can't make tests as one is a live system and the dev system is seeding the database cache, BTW we have a map file that depending on which zoom level you are we use one or another type of map, we have like 10 layers and in 2 zoom levels we use this raster that doesn't work BUT as I said if I call the map with QGIS it works perfectly, wasn't just working when was called through a web service. Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] mapserver 6 mapcache 1.6 tileindex limit
Hi All, Since I reimaged the server and installed CentOS 7 (previously was centOS 6) , mapserver 6.2.1 and Mapcache 1.6 (previously was v.1) I can't make work my .MAP file that call inside the Raster tileindex of the whole UK (created with GDAL). The only way to make it works it to create a new index with less images, and if I go over that limit mapserver doesn't load the layer. It's a settings that I can change within mapserver or is related with something else? Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache fastcgi
I seeded the first 4 zoom level of the sqlite cache. Here the sqlite version https://www.traffwebdev.uk/parking/ here the disk version http://www.buchananmapping.co.uk/ You can compare them from 5km to 200m (the scale in the bottom right), the rest is not cached. Bear in mind that the disk version is accessed over 80 website every day so there are people there all day while the sqlite you will be the only one as is setup just for test at the moment. You can see that the sqlite version is almost 4 times slower, the only difference is that the disk is served over http and the sqlite over https. Server setup: -CentOS 7 -MapServer 6 -mapCache 1.6 -sqlite3 (3.7) the 2 servers are exactly the same and I reimage them this august. Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache fastcgi
This is my cache setup
/var/www/path/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3
1
1
1573741823
this is a custom grid I made up for an
example
EPSG:27700
128 128
0 0 70 125
70 28 14 7 6.16 2.8 1.4 0.7 0.28 0.14 0.07
bcmaps
cache_sqlite
bcgrid
JPEG
4 4
0
31536000
Sebastiano Laini
Web Developer
Buchanan Computing
227 Shepherds Bush Road London W6 7AS
020 8846 3220
Web: http://www.BuchananComputing.co.uk<http://www.buchanancomputing.co.uk/>
For general or urgent enquiries please write to:
supp...@buchanancomputing.co.uk<mailto:supp...@buchanancomputing.co.uk>
Buchanan Computing Ltd. Registered at the above office, no. 3274424 (England &
Wales)
This message is intended solely for the individual/entity to whom it is
addressed. It may contain confidential or legally privileged information. Any
unauthorised disclosure or copying is prohibited and my be unlawful. If you
have received this communication in error, please notify the sender immediately
and delete it from your system.
From: Sebastiano Laini
Sent: 17 October 2018 13:39
To: 'mapserver-users@lists.osgeo.org'
Cc: 'Rahkonen Jukka (MML)'
Subject: RE: mapcache fastcgi
Hi,
It appears in both ways, when I'm seeding and when is requested from a
pre-seeded cache, today I can't because I need to move the dev domain in
another server but tomorrow I will pass you the link to the 2 cache so you can
see the difference.
Tonight I will let it seed the whole area.
The cache is on a normal HDD, I think all out servers are 1 RPM, and the
test was done on a brand new server so the disk is new.
Sebastiano Laini
Web Developer
Buchanan Computing
From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi]
Sent: 17 October 2018 13:35
To: Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>>;
'mapserver-users@lists.osgeo.org'
mailto:mapserver-users@lists.osgeo.org>>
Subject: RE: mapcache fastcgi
Hi,
In which way does the slowness appear? Is it when seeding or when serving from
a pre-seeded cache? Can you give metrics about the difference in speed?
In this comparison the differences were rather small
http://blog.sogeo.services/blog/2016/07/01/mapcache-disk-vs-sqlite.html
Also by the documentation the speed should be only "a bit slower"
The SQLite based caches are a bit slower than the disk based caches, and may
have write-locking issues at seed time if a high number of threads all try to
insert new tiles concurrently.
You wrote that SQLite cache is very slow for you which makes me think that
there is something sub-optimal in your installation. First thing to check is if
the SQLite database is on a fast disk.
If SQLite cache is slow then MBTiles cache should be slow as well but I would
make a test to be sure.
-Jukka Rahkonen-
Lähettäjä: Sebastiano Laini [mailto:sebastiano.la...@buchanancomputing.co.uk]
Lähetetty: 17. lokakuuta 2018 15:00
Vastaanottaja: 'mapserver-users@lists.osgeo.org'
mailto:mapserver-users@lists.osgeo.org>>
Kopio: Rahkonen Jukka (MML)
mailto:jukka.rahko...@maanmittauslaitos.fi>>
Aihe: RE: mapcache fastcgi
Is there a way to improve the speed and match the speed of the sqLite3 cache
with the disk one?
We have 80 websites connected to MapCache and when I tried the sqLite3 I was
testing it just on one website and on a complete different server with nothing
in it.
Sebastiano Laini
Web Developer
Buchanan Computing
From: Sebastiano Laini
Sent: 17 October 2018 12:43
To: 'mapserver-users@lists.osgeo.org'
mailto:mapserver-users@lists.osgeo.org>>
Cc: 'Rahkonen Jukka (MML)'
mailto:jukka.rahko...@maanmittauslaitos.fi>>
Subject: RE: mapcache fastcgi
Still it redirect you to that link, and btw it seems that mapcache can be used
also as a fastCGI.
https://mapserver.org/mapcache/install.html#cgi-fastcgi-specific-instructions
Though is true that I'm reading now in the CMaker Builder section
MapCache can run as a FastCGI executable. Note that the overhead of FastCGI is
non-negligible with respect to the throughput you may obtain with a native
Apache module. The FastCGI build is less tested, and may lag behind the Apache
module version on some minor details. YMMV.
Sebastiano Laini
Web Developer
Buchanan Computing
From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi]
Sent: 17 October 2018 12:39
To: Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>>;
'mapserver-users@lists.osgeo.org'
mailto:mapserver-users@lists.osgeo.org>>
Subject: VS: mapcache fastcgi
Hi,
That documentation is about Mapserver, Mapcache is another software. It does
belong to the same Mapserver but is not the same.
MapCache is documented here:
ht
Re: [mapserver-users] mapcache fastcgi
Hi, It appears in both ways, when I'm seeding and when is requested from a pre-seeded cache, today I can't because I need to move the dev domain in another server but tomorrow I will pass you the link to the 2 cache so you can see the difference. Tonight I will let it seed the whole area. The cache is on a normal HDD, I think all out servers are 1 RPM, and the test was done on a brand new server so the disk is new. Sebastiano Laini Web Developer Buchanan Computing From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 17 October 2018 13:35 To: Sebastiano Laini ; 'mapserver-users@lists.osgeo.org' Subject: RE: mapcache fastcgi Hi, In which way does the slowness appear? Is it when seeding or when serving from a pre-seeded cache? Can you give metrics about the difference in speed? In this comparison the differences were rather small http://blog.sogeo.services/blog/2016/07/01/mapcache-disk-vs-sqlite.html Also by the documentation the speed should be only "a bit slower" The SQLite based caches are a bit slower than the disk based caches, and may have write-locking issues at seed time if a high number of threads all try to insert new tiles concurrently. You wrote that SQLite cache is very slow for you which makes me think that there is something sub-optimal in your installation. First thing to check is if the SQLite database is on a fast disk. If SQLite cache is slow then MBTiles cache should be slow as well but I would make a test to be sure. -Jukka Rahkonen- Lähettäjä: Sebastiano Laini [mailto:sebastiano.la...@buchanancomputing.co.uk] Lähetetty: 17. lokakuuta 2018 15:00 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Kopio: Rahkonen Jukka (MML) mailto:jukka.rahko...@maanmittauslaitos.fi>> Aihe: RE: mapcache fastcgi Is there a way to improve the speed and match the speed of the sqLite3 cache with the disk one? We have 80 websites connected to MapCache and when I tried the sqLite3 I was testing it just on one website and on a complete different server with nothing in it. Sebastiano Laini Web Developer Buchanan Computing From: Sebastiano Laini Sent: 17 October 2018 12:43 To: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Cc: 'Rahkonen Jukka (MML)' mailto:jukka.rahko...@maanmittauslaitos.fi>> Subject: RE: mapcache fastcgi Still it redirect you to that link, and btw it seems that mapcache can be used also as a fastCGI. https://mapserver.org/mapcache/install.html#cgi-fastcgi-specific-instructions Though is true that I'm reading now in the CMaker Builder section MapCache can run as a FastCGI executable. Note that the overhead of FastCGI is non-negligible with respect to the throughput you may obtain with a native Apache module. The FastCGI build is less tested, and may lag behind the Apache module version on some minor details. YMMV. Sebastiano Laini Web Developer Buchanan Computing From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 17 October 2018 12:39 To: Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Subject: VS: mapcache fastcgi Hi, That documentation is about Mapserver, Mapcache is another software. It does belong to the same Mapserver but is not the same. MapCache is documented here: https://mapserver.org/mapcache/index.html#mapcache -Jukka Rahkonen- Lähettäjä: Sebastiano Laini [mailto:sebastiano.la...@buchanancomputing.co.uk] Lähetetty: 17. lokakuuta 2018 14:33 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Kopio: Rahkonen Jukka (MML) mailto:jukka.rahko...@maanmittauslaitos.fi>> Aihe: RE: mapcache fastcgi Well, for what I can read in the documentation of mapserver: https://mapserver.org/optimization/fastcgi.html FastCGI is a protocol for keeping cgi-bin style web applications running as a daemon to take advantage of preserving memory caches, and amortizing other high startup costs (like heavy database connections) over many requests. It seems that using mapcache as fastCGI over apache modules it that should improve speed also. Sebastiano Laini Web Developer Buchanan Computing From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 17 October 2018 12:30 To: Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Subject: Re: mapcache fastcgi Hi, Could you explain your plan with some more details? What I have understood is that you have experienced that SQLite cache is slower than disk cache in your environment. What I do not understand is what you are going to reach by running Mapcache as fastcgi instead of running
Re: [mapserver-users] mapcache fastcgi
Is there a way to improve the speed and match the speed of the sqLite3 cache with the disk one? We have 80 websites connected to MapCache and when I tried the sqLite3 I was testing it just on one website and on a complete different server with nothing in it. Sebastiano Laini Web Developer Buchanan Computing From: Sebastiano Laini Sent: 17 October 2018 12:43 To: 'mapserver-users@lists.osgeo.org' Cc: 'Rahkonen Jukka (MML)' Subject: RE: mapcache fastcgi Still it redirect you to that link, and btw it seems that mapcache can be used also as a fastCGI. https://mapserver.org/mapcache/install.html#cgi-fastcgi-specific-instructions Though is true that I'm reading now in the CMaker Builder section MapCache can run as a FastCGI executable. Note that the overhead of FastCGI is non-negligible with respect to the throughput you may obtain with a native Apache module. The FastCGI build is less tested, and may lag behind the Apache module version on some minor details. YMMV. Sebastiano Laini Web Developer Buchanan Computing From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 17 October 2018 12:39 To: Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Subject: VS: mapcache fastcgi Hi, That documentation is about Mapserver, Mapcache is another software. It does belong to the same Mapserver but is not the same. MapCache is documented here: https://mapserver.org/mapcache/index.html#mapcache -Jukka Rahkonen- Lähettäjä: Sebastiano Laini [mailto:sebastiano.la...@buchanancomputing.co.uk] Lähetetty: 17. lokakuuta 2018 14:33 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Kopio: Rahkonen Jukka (MML) mailto:jukka.rahko...@maanmittauslaitos.fi>> Aihe: RE: mapcache fastcgi Well, for what I can read in the documentation of mapserver: https://mapserver.org/optimization/fastcgi.html FastCGI is a protocol for keeping cgi-bin style web applications running as a daemon to take advantage of preserving memory caches, and amortizing other high startup costs (like heavy database connections) over many requests. It seems that using mapcache as fastCGI over apache modules it that should improve speed also. Sebastiano Laini Web Developer Buchanan Computing From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 17 October 2018 12:30 To: Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Subject: Re: mapcache fastcgi Hi, Could you explain your plan with some more details? What I have understood is that you have experienced that SQLite cache is slower than disk cache in your environment. What I do not understand is what you are going to reach by running Mapcache as fastcgi instead of running it as Apache module which should be the faster option, and if this has some connection with the SQLite cache vs. disk cache comparison. Do you guess that fastcgi woud make SQLite cache faster? -Jukka Rahkonen- Lähettäjä: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] Puolesta Sebastiano Laini Lähetetty: 17. lokakuuta 2018 12:16 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Aihe: [mapserver-users] mapcache fastcgi Hi all, In the end I tried MapCache with sqlite3 cache but it's very slow compared with the disk cache, so I want to setup MapCache with fastCGI on CentOS 7. I don't understand what I need to install as the fastCGI link in the documentation is broken and I don't know if is up to date. I already have apache fgcid installed from default in the httpd folder and I use PHP in fastCGI in all the domains, so I'm sure that fastCGI it works, though, I still need the fastCGI library? How can I install it in CentOS 7? In my apache conf I've already added the lines to load the mapcache module and the mapcache.xml file LoadModule mapcache_modulemodules/mod_mapcache.so Require all granted MapCacheAlias /mapcache "/path/to/directory/mapcache.xml" I need to leave the previous configuration and add these new lines in the conf files? FcgidInitialEnv "MAPCACHE_CONFIG_FILE" "/path/to/mapcache/mapcache.xml IPCCommTimeout 120 MaxProcessCount 10 FcgidInitialEnv "MAPCACHE_CONFIG_FILE" "/path/to/mapcache/mapcache.xml" Order Allow,Deny Allow from all SetHandler fcgid-script ScriptAlias /map.fcgi "/path/to/mapcache/src/mapcache" And then instead of http://myserver/mapcache I need to use http://myserver/fcgi-bin to access mapcache fcgi? Regards, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache fastcgi
Still it redirect you to that link, and btw it seems that mapcache can be used also as a fastCGI. https://mapserver.org/mapcache/install.html#cgi-fastcgi-specific-instructions Though is true that I'm reading now in the CMaker Builder section MapCache can run as a FastCGI executable. Note that the overhead of FastCGI is non-negligible with respect to the throughput you may obtain with a native Apache module. The FastCGI build is less tested, and may lag behind the Apache module version on some minor details. YMMV. Sebastiano Laini Web Developer Buchanan Computing From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 17 October 2018 12:39 To: Sebastiano Laini ; 'mapserver-users@lists.osgeo.org' Subject: VS: mapcache fastcgi Hi, That documentation is about Mapserver, Mapcache is another software. It does belong to the same Mapserver but is not the same. MapCache is documented here: https://mapserver.org/mapcache/index.html#mapcache -Jukka Rahkonen- Lähettäjä: Sebastiano Laini [mailto:sebastiano.la...@buchanancomputing.co.uk] Lähetetty: 17. lokakuuta 2018 14:33 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Kopio: Rahkonen Jukka (MML) mailto:jukka.rahko...@maanmittauslaitos.fi>> Aihe: RE: mapcache fastcgi Well, for what I can read in the documentation of mapserver: https://mapserver.org/optimization/fastcgi.html FastCGI is a protocol for keeping cgi-bin style web applications running as a daemon to take advantage of preserving memory caches, and amortizing other high startup costs (like heavy database connections) over many requests. It seems that using mapcache as fastCGI over apache modules it that should improve speed also. Sebastiano Laini Web Developer Buchanan Computing From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 17 October 2018 12:30 To: Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Subject: Re: mapcache fastcgi Hi, Could you explain your plan with some more details? What I have understood is that you have experienced that SQLite cache is slower than disk cache in your environment. What I do not understand is what you are going to reach by running Mapcache as fastcgi instead of running it as Apache module which should be the faster option, and if this has some connection with the SQLite cache vs. disk cache comparison. Do you guess that fastcgi woud make SQLite cache faster? -Jukka Rahkonen- Lähettäjä: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] Puolesta Sebastiano Laini Lähetetty: 17. lokakuuta 2018 12:16 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Aihe: [mapserver-users] mapcache fastcgi Hi all, In the end I tried MapCache with sqlite3 cache but it's very slow compared with the disk cache, so I want to setup MapCache with fastCGI on CentOS 7. I don't understand what I need to install as the fastCGI link in the documentation is broken and I don't know if is up to date. I already have apache fgcid installed from default in the httpd folder and I use PHP in fastCGI in all the domains, so I'm sure that fastCGI it works, though, I still need the fastCGI library? How can I install it in CentOS 7? In my apache conf I've already added the lines to load the mapcache module and the mapcache.xml file LoadModule mapcache_modulemodules/mod_mapcache.so Require all granted MapCacheAlias /mapcache "/path/to/directory/mapcache.xml" I need to leave the previous configuration and add these new lines in the conf files? FcgidInitialEnv "MAPCACHE_CONFIG_FILE" "/path/to/mapcache/mapcache.xml IPCCommTimeout 120 MaxProcessCount 10 FcgidInitialEnv "MAPCACHE_CONFIG_FILE" "/path/to/mapcache/mapcache.xml" Order Allow,Deny Allow from all SetHandler fcgid-script ScriptAlias /map.fcgi "/path/to/mapcache/src/mapcache" And then instead of http://myserver/mapcache I need to use http://myserver/fcgi-bin to access mapcache fcgi? Regards, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] mapcache fastcgi
Well, for what I can read in the documentation of mapserver: https://mapserver.org/optimization/fastcgi.html FastCGI is a protocol for keeping cgi-bin style web applications running as a daemon to take advantage of preserving memory caches, and amortizing other high startup costs (like heavy database connections) over many requests. It seems that using mapcache as fastCGI over apache modules it that should improve speed also. Sebastiano Laini Web Developer Buchanan Computing From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi] Sent: 17 October 2018 12:30 To: Sebastiano Laini ; 'mapserver-users@lists.osgeo.org' Subject: Re: mapcache fastcgi Hi, Could you explain your plan with some more details? What I have understood is that you have experienced that SQLite cache is slower than disk cache in your environment. What I do not understand is what you are going to reach by running Mapcache as fastcgi instead of running it as Apache module which should be the faster option, and if this has some connection with the SQLite cache vs. disk cache comparison. Do you guess that fastcgi woud make SQLite cache faster? -Jukka Rahkonen- Lähettäjä: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] Puolesta Sebastiano Laini Lähetetty: 17. lokakuuta 2018 12:16 Vastaanottaja: 'mapserver-users@lists.osgeo.org' mailto:mapserver-users@lists.osgeo.org>> Aihe: [mapserver-users] mapcache fastcgi Hi all, In the end I tried MapCache with sqlite3 cache but it's very slow compared with the disk cache, so I want to setup MapCache with fastCGI on CentOS 7. I don't understand what I need to install as the fastCGI link in the documentation is broken and I don't know if is up to date. I already have apache fgcid installed from default in the httpd folder and I use PHP in fastCGI in all the domains, so I'm sure that fastCGI it works, though, I still need the fastCGI library? How can I install it in CentOS 7? In my apache conf I've already added the lines to load the mapcache module and the mapcache.xml file LoadModule mapcache_modulemodules/mod_mapcache.so Require all granted MapCacheAlias /mapcache "/path/to/directory/mapcache.xml" I need to leave the previous configuration and add these new lines in the conf files? FcgidInitialEnv "MAPCACHE_CONFIG_FILE" "/path/to/mapcache/mapcache.xml IPCCommTimeout 120 MaxProcessCount 10 FcgidInitialEnv "MAPCACHE_CONFIG_FILE" "/path/to/mapcache/mapcache.xml" Order Allow,Deny Allow from all SetHandler fcgid-script ScriptAlias /map.fcgi "/path/to/mapcache/src/mapcache" And then instead of http://myserver/mapcache I need to use http://myserver/fcgi-bin to access mapcache fcgi? Regards, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] mapcache fastcgi
Hi all, In the end I tried MapCache with sqlite3 cache but it's very slow compared with the disk cache, so I want to setup MapCache with fastCGI on CentOS 7. I don't understand what I need to install as the fastCGI link in the documentation is broken and I don't know if is up to date. I already have apache fgcid installed from default in the httpd folder and I use PHP in fastCGI in all the domains, so I'm sure that fastCGI it works, though, I still need the fastCGI library? How can I install it in CentOS 7? In my apache conf I've already added the lines to load the mapcache module and the mapcache.xml file LoadModule mapcache_modulemodules/mod_mapcache.so Require all granted MapCacheAlias /mapcache "/path/to/directory/mapcache.xml" I need to leave the previous configuration and add these new lines in the conf files? FcgidInitialEnv "MAPCACHE_CONFIG_FILE" "/path/to/mapcache/mapcache.xml IPCCommTimeout 120 MaxProcessCount 10 FcgidInitialEnv "MAPCACHE_CONFIG_FILE" "/path/to/mapcache/mapcache.xml" Order Allow,Deny Allow from all SetHandler fcgid-script ScriptAlias /map.fcgi "/path/to/mapcache/src/mapcache" And then instead of http://myserver/mapcache I need to use http://myserver/fcgi-bin to access mapcache fcgi? Regards, Sebastiano Laini Web Developer Buchanan Computing ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
Re: [mapserver-users] Mapcache instruction
Thanks guys,
I will try to do it after I will recompile again Mapcache to use SQLite as I
compite it -DWITH_SQLITE=OFF \
I suppose that I can also think about to split the db as we are speaking about
more than 10milion files.
How exactly do you call the database and how this code
/{grid}/{tileset}/{z}/{x}-{y} works?
Sebastiano
From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] On
Behalf Of lars.schylb...@blixtmail.se
Sent: 20 September 2018 13:13
To: mapserver-users
Subject: Re: [mapserver-users] Mapcache instruction
I also ran into the problem of running out of i-nodes last year.
Here is the configuration I have with a sqlite cache:
/data/mapcache/{grid}/{tileset}/{z}/{x}-{y}.sqlite3
1
1
1573741823
I guess the pragma thing is described in the documentation.
xcount and ycount describes how much You want to store in each sqlite db file
before it creates a new db file.
mapcache is creating the db files automatically. So seeding a large area You
end up with many sqlite db files that all work together.
This has worked really good for our use case.
Lars Schylberg
September 20, 2018 1:24 PM, "Yves Jacolin"
mailto:yves.jaco...@camptocamp.com?to=%22yves%20jacolin%22%20%3cyves.jaco...@camptocamp.com%3e>>
wrote:
Here a short tuto from my personal notes:
1. create a cache entry in your mapcache file
/var/www/osm_google.db
2. create a empty sqlite file
- run sqlite3
- run this commande inside sqlite3: ".save filename.db" (without ")
3. use this cache in your settings
You don't need MySQL or PostgreSQL, only sqlite3 and its dependencies. Path
used in my example for cache block are for Unix like system, you should adapt
it for Windows : c:/....
Y.
Le jeu. 20 sept. 2018 à 13:07, Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>>
a écrit :
Hi all,
We have a MapCache setup with “disk_cache” but we are running out of Inodes
when I try to seed the whole UK with projection 27700.
I was thinking to propose to my supervisor to use the “SQLite Caches” but I
don’t know how it works. Can someone guide me through the process of setup
MapCache with this type of cache? And if I need some dependency, we use
PostgreSQL in our server so I assume I need to install also MySQL.
I’ve already read the documentation but still I have doubt about the db part,
how to create the db and locate it. I never use SQLite, just PostgreSQL.
Regards;
Sebastiano.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/mapserver-users
--
Training and support manager
Camptocamp
Bâtiment le Dauphin - 1er étage
18 rue du Lac Saint André
73382 Le Bourget-du Lac
Tel (France) : +33 4 58 48 20 43
Tel (Switzerland) : +41 21 619 10 43
Mob. : +33 6 18 75 42 21
email : yves.jaco...@camptocamp.com<mailto:yves.jaco...@camptocamp.com>
http://www.camptocamp.com
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
[mapserver-users] Mapcache instruction
Hi all, We have a MapCache setup with "disk_cache" but we are running out of Inodes when I try to seed the whole UK with projection 27700. I was thinking to propose to my supervisor to use the "SQLite Caches" but I don't know how it works. Can someone guide me through the process of setup MapCache with this type of cache? And if I need some dependency, we use PostgreSQL in our server so I assume I need to install also MySQL. I've already read the documentation but still I have doubt about the db part, how to create the db and locate it. I never use SQLite, just PostgreSQL. Regards; Sebastiano. ___ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users
