[MapServer-users] Mapcache - postgresql dimensions causing HUGE number of open db connections

[Tim Kempisty via MapServer-users]

Mapcache folks,

This is a technical question about the Mapcache connection pool and how 
postgreql dimensions are implemented.

I have a big mapcache WMS with several hundred layers defined.  Most of them 
have a postgresql dimension that reports back a list of available timestamps 
for these layers.  When we are under load, I end up with a couple thousand 
database connections all coming from Mapcache.  They're only lightweight 
dimension queries... it's just a LOT of connections hanging open.  I'm looking 
for the upper bound on the number of these connections.

Looking at the Mapcache source code...  I see a hard coded max_list_size  = 10 
in connection_pool.c.   I assume that means any one mapcache process/thread is 
allowed to keep up to ten network connections open at one time.

dimension_pg.c  seems to get its connections from this pool. Other connections 
use this pool though, right?   Like the source URLs?

So if I have Mapcache running as an Apache module, and if Apache can launch up 
to 256 threads...  I could be on the hook for up to 2560 connections sitting 
open (until the threads die off / time out.)  Is my understanding correct?

It also looks like dimension_pg uses prepared statements on the db server.  
That would probably break a connection pooler like PgBouncer.  So my options 
for managing the number of postgresql connections are: 

1.  Setting an obscene max_connections limit in postgresql.conf, allowing my 
mapcache role to consume a lot of them.
2.  Reducing the number of Apache threads in mpm_event   (At the risk of 
impacting general Apache performance)
3.  Using the fastcgi version of Mapcache, and limiting the number of mapcache 
fcgi processes
4.  Reducing max_list_size in connection_pool.c, which might impact performance.

Am I on the right track here?

-Tim

___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [MapServer-users] mapcache: Setting styles for a tileset and source

[Trygve Aspenes via MapServer-users]

hm, wait

The request from my app is based on the getcapability from mapcache, so 
no styles. But when mapcache is sending the request to mapserver the 
configured STYLES in the getmap section in the mapcache config is added 
correctly.


I don't know how I missed that, but I think I get what I expect now.

Trygve Aspenes

Den 2023-09-29 17:29, skrev Trygve Aspenes via MapServer-users:

Hi

In mapserver I have several styles.

In mapcache I see no styles info in the getcapabilities document, so my 
app can't know anything about it.
Anyway, you would need one tileset for each styles to cache the data. 
So right, that makes sense.


Then I try to make one tileset per style. I add to my source config, 
getmap like this:




http://mapserver:8080?



image/png
tileset
style_name




But no luck. STYLES is sent empty to my mapserver wms server. I haven't 
got time to debug the code now, but my first look, STYLES should be 
handled here:

https://github.com/MapServer/mapcache/blob/main/lib/source_wms.c#L154-L167

the for loop would parse all elements. But I would need a closer look 
to see if this is what actually happends.


Anyone else know how to handle styles in mapcache?

Trygve Aspenes
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[MapServer-users] mapcache: Setting styles for a tileset and source

[Trygve Aspenes via MapServer-users]

Hi

In mapserver I have several styles.

In mapcache I see no styles info in the getcapabilities document, so my 
app can't know anything about it.
Anyway, you would need one tileset for each styles to cache the data. So 
right, that makes sense.


Then I try to make one tileset per style. I add to my source config, 
getmap like this:




http://mapserver:8080?



image/png
tileset
style_name




But no luck. STYLES is sent empty to my mapserver wms server. I haven't 
got time to debug the code now, but my first look, STYLES should be 
handled here:

https://github.com/MapServer/mapcache/blob/main/lib/source_wms.c#L154-L167

the for loop would parse all elements. But I would need a closer look to 
see if this is what actually happends.


Anyone else know how to handle styles in mapcache?

Trygve Aspenes
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[MapServer-users] mapcache: time dimension default value and psql list_query

[Trygve Aspenes via MapServer-users]

Hi mapserver users list

I have a mapserver backend with mapcache to handle the tile queries and 
psql as a db.


It's working but I have some details I'm trying to improve.

1: The time dimension has a default value. The client I server data to 
require this. I have 1 week of data available adding more data every 5 
minute ( and deleting old data). Therefor the hard coded default value 
quickly become out of range. ie:


time="true" unit="ISO8601">


Is there a way to make the default value be the newest value from my 
psql?
(an emergency solution will be to make some kind of script that 
continuously update this value somehow, but I hoped to avoid that.)


2: I think I understand that  list all timestamps values 
from the psql. This will be visible in the mapcache getcapabilities as a 
list of timestamps, 2016 stamps with data every 5 minutes for one week. 
Like
units="ISO8601">2023-03-23T00:00:00Z,  
2023-03-30T13:30:00Z


Is there a way to make a start/end/step kind of list? eg 
2023-03-23T00:00:00Z/2023-03-30T13:30:00Z/PT5M


Thanks

Trygve Aspenes
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[MapServer-users] Mapcache authentication in source http requests

[Johannes Paul]

Hello,
I can't find anywhere in the documentation if and how mapcache handles
authentication in data sources http requests.
I found this old post
https://lists.osgeo.org/pipermail/mapserver-users/2013-April/074509.html
dealing with basic authentication headers.
Is there any built-in mechanism for dealing with NTLM authentication ?
Thanks,
Johannes
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [MapServer-users] Mapcache second level dimensions

[Johannes Paul]

Coming back on a semi-working 2nd level dimension use case, if I use the
below conf:
```

/nfs_tiles/test/{dim:product}.sqlite3


PNG
worldwind
test_cache

  
/nfs_tiles/test/mosa.db
select product from mosa where
sensor=:dim
select product from mosa
  


```
Then the *dim_product *is passed to the cache '*test_cache*' with the
sub-dimension result using parameters '
*=spot-img2_product=spot-img2*', only if the result returns
only 1 sud-dimension value ... if it returns X sub-dimensions values I
still get an error : `*dimension (product) for tileset (mosa_test) returned
invalid number (X) of subdimensions (1 expected)*`

It seems the mechanism is not working the way it should be, isn't it ?
Johannes

On Thu, 16 Feb 2023 at 09:54, Johannes Paul 
wrote:

> Hello,
>
> I'm trying to implement second level SQLite dimension as explained in the
> doc
> https://mapserver.org/mapcache/dimensions.html#second-level-dimensions
>
> Following that exemple on mapcache v1.12, I'm expecting that the below
> conf, with WMS request on 'mosa_test' tileset with 'dim_sensor=phr', shall
> use the “product” sub-dimension values to query the data source
> 'mosa_source' on tileset 'test_primary' with 'dim_product' for each product
> returned by the 2nd level dimension request (cache 'test_cache' is
> pre-existing) ...
>
> ```
> 
> /nfs_tiles/test/{dim:product}.sqlite3
> 
> 
> test_cache
> worldwind
> 
>
>  .*
>
> 
> PNG
> 
> 
> /nfs_tiles/test/mosa_{dim:sensor}.sqlite3
> 
> 
> PNG
> worldwind
> mosa_cache
> mosa_source
> 
>   
> /nfs_tiles/test/mosa.db
> select product from mosa where
> sensor=:dim
> select product from mosa
>   
> 
> 
> 
> 
>http://server/mapcache/
> 
> 
> 
> PNG
> test_primary
> true
> 
> 
> 
> ```
>
> However all I get is a mapcache error in the httpd log, as follows :
> `dimension (sensor) for tileset (mosa_test) returned invalid number (X) of
> subdimensions (1 expected)`
> If my 'sensor' dimension returns X products (which is the use case), I get
> the error above, and if my 'sensor' dimension returns 1 product (let say
> spot-img2), I get an error on the mapcache generated WMS request to the
> source 'mosa_source' on tileset 'test_primary' with parameters
> '=spot-img2_sensor=spot-img2' ... which can obviously not work !
> Why is mapcache not using 'dim_product' in the WMS request ?
>
> Have I missed something ?
> Thanks,
> Johannes
>
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[MapServer-users] Mapcache second level dimensions

[Johannes Paul]

Hello,

I'm trying to implement second level SQLite dimension as explained in the
doc
https://mapserver.org/mapcache/dimensions.html#second-level-dimensions

Following that exemple on mapcache v1.12, I'm expecting that the below
conf, with WMS request on 'mosa_test' tileset with 'dim_sensor=phr', shall
use the “product” sub-dimension values to query the data source
'mosa_source' on tileset 'test_primary' with 'dim_product' for each product
returned by the 2nd level dimension request (cache 'test_cache' is
pre-existing) ...

```

/nfs_tiles/test/{dim:product}.sqlite3


test_cache
worldwind

   
 .*
   

PNG


/nfs_tiles/test/mosa_{dim:sensor}.sqlite3


PNG
worldwind
mosa_cache
mosa_source

  
/nfs_tiles/test/mosa.db
select product from mosa where
sensor=:dim
select product from mosa
  




   http://server/mapcache/



PNG
test_primary
true



```

However all I get is a mapcache error in the httpd log, as follows :
`dimension (sensor) for tileset (mosa_test) returned invalid number (X) of
subdimensions (1 expected)`
If my 'sensor' dimension returns X products (which is the use case), I get
the error above, and if my 'sensor' dimension returns 1 product (let say
spot-img2), I get an error on the mapcache generated WMS request to the
source 'mosa_source' on tileset 'test_primary' with parameters
'=spot-img2_sensor=spot-img2' ... which can obviously not work !
Why is mapcache not using 'dim_product' in the WMS request ?

Have I missed something ?
Thanks,
Johannes
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[MapServer-users] MapCache 1.14.0 available (security release)

[Jeff McKenna]

MapCache 1.14.0 is now released, that includes handling a string 
formatting injection vulnerability.


Changelog: 
https://mapserver.org/development/changelog/mapcache/changelog-1-14.html

Download: https://download.osgeo.org/mapserver/mapcache-1.14.0.tar.gz

thanks,

--
MapServer / MapCache team








___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[MapServer-users] mapcache install -- libapache

[Greg Robillard]

Has anyone had luck installing libapache2-mod-mapcache? I installed
Mapserver 7.4 using apt on an Ubuntu 22.04 server. Mapserver works
beautifully. I attempted to do the same with apt to install the Apache
Mapcache module libapache2-mod-mapcache, but the installation fails. The
install goes through, but Apache fails to start. The message from apachctl
configtest reads: "Cannot load /usr/lib/apache2/modules/mod_mapcache.so
into server: libpng12.so.0: cannot open shared object file: No such file or
directory." I can verify that the .so file is in the correct place in the
system. When I run $ldd /usr/lib/apache2/modules/mod_mapcache.so, the
response is empty, which suggests to me that the install of
/usr/lib/apache2/modules/mod_mapcache.so failed.

I cannot find any documentation around the use of  libapache2-mod-mapcache,
so I am wondering if my best bet is to compile from source.
Thanks,

Greg
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [MapServer-users] Mapcache s3 - server replied: Not Found

[Rahkonen Jukka]

Hi,

It seems that you have tried to make a new WMS connection but you do not run 
any WMS server in s3. Instead, you have there tiles. Try this XYZ approach 
https://silverspringenergy.com/adding-xyz-tiles-to-a-qgis-project/.

-Jukka Rahkonen-

Lähettäjä: MapServer-users  Puolesta 
Marcin Niemyjski via MapServer-users
Lähetetty: perjantai 9. joulukuuta 2022 16.02
Vastaanottaja: Marcin Niemyjski via MapServer-users 

Aihe: [MapServer-users] Mapcache s3 - server replied: Not Found

Hello,

my mapcache xml to s3 bucket is working now but while trying to connect it to 
qgis using url created in xml 
(https://s3.waw3-1.cloudferro.com/mapcache/{tileset}/{grid}/{z}/{x}/{y}/{ext}<https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs3.waw3-1.cloudferro.com%2Fmapcache%2F%257Btileset%257D%2F%257Bgrid%257D%2F%257Bz%257D%2F%257Bx%257D%2F%257By%257D%2F%257Bext%257D=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7Cc64095fe878e47ab3c5f08dad9edebe2%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638061913142336382%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=SGzPtQlRQ1YHH9lp6Jb1h%2BLZBleq0mhTv8mt094Sxro%3D=0>)
 qgis returns:

Failed to download capabilities:
Download of capabilities failed: Error transferring 
https://s3.waw3-1.cloudferro.com/mapcache/%7Btileset%7D/%7Bgrid%7D/%7Bz%7D/%7Bx%7D/%7By%7D/%7Bext%7D?SERVICE=WMS=GetCapabilities
 - server replied: Not Found

How can I start using my tiles?
Best,
Marcin
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[MapServer-users] Mapcache s3 - server replied: Not Found

[Marcin Niemyjski via MapServer-users]

Hello,

my mapcache xml to s3 bucket is working now but while trying to connect it to 
qgis using url created in xml 
(https://s3.waw3-1.cloudferro.com/mapcache/{tileset}/{grid}/{z}/{x}/{y}/{ext}) 
qgis returns:

Failed to download capabilities:
Download of capabilities failed: Error transferring 
https://s3.waw3-1.cloudferro.com/mapcache/%7Btileset%7D/%7Bgrid%7D/%7Bz%7D/%7Bx%7D/%7By%7D/%7Bext%7D?SERVICE=WMS=GetCapabilities
 - server replied: Not Found

How can I start using my tiles?
Best,
Marcin
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[MapServer-users] Mapcache seeder - failed to parse file /var/www/aed_world.xml. Is it valid XML? on valid xml

[Marcin Niemyjski via MapServer-users]

Hello,

I have a problem, namely:

I created .xml file, which is used to cache tiles into s3 bucket, here it is:
https://pastebin.com/RwPAESUG

while starting seeder the error is recived: failed to parse file 
/var/www/aed_world.xml. Is it valid XML?

I've checked content of XML in online validator - no error was detected.

Maybe somebody knows how to overcome this situation?
Thank you,
Marcin
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache serving raw type data in WMS

[Johannes Paul]

Hello,
using SRTMGL1 elevation data, I would like to use Mapserver to output
Geotiff 16bits rasters, that would be tiled via Mapcache in raw type format
in db files, and served in WM(T)S.
However, from what I can see in the Mapcache documentation (
https://github.com/MapServer/mapcache/pull/166), raw type has been
developed for vector tiles, and WMS support for such format type might have
been disabled.
Could you please confirm what can be achieved and what cannot ?
Thanks,
Johannes
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache 1.12.0 released

[jbo-...@mailo.com]

The MapServer team is pleased to announce the major release of MapCache 
1.12.0


You can download the source code at 
.


The list of evolutions is found on the Changelog page at 



Many thanks to everyone how contributed to this version by finding bugs, 
fixing them, providing new features and improving the documentation.


--
The MapServer team


___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache disk cache gives unexpected paths to tiles

[Trygve Aspenes]

Ah, thanks Thomas!

I did not catch that from the documentation.
I also needed to add layout like this:



/mapcache/{tileset}/{grid}/{dim:time}/{z}/{x}/{y}.{ext}



to make it work ( both from testing and looking at the documentation)

If you use the default this will be like tilecache as far as I 
understand.


Do you have any idea what will be the fastest? To use default like:

/mapcache


or the solution above with template? ( As a lot of 000 directories will 
be created with the base one)


Trygve

Den 2022-02-09 12:24, skrev thomas bonfort:

You should remove the /mapcache from the >cache> block,
as it takes precedence over your 

--
thomas

On Wed, Feb 9, 2022 at 12:10 PM Trygve Aspenes
 wrote:


Hi

I have a mapcache running using disk as cache method with config
like
this:


/mapcache



/mapcache/{tileset}#{grid}#{dim}/{z}/{x}/{y}.{ext}



But here is what I get on disk:

/mapcache//EPSG:3857/2022-02-05T00:12:00Z/05/000/000/038/000/000/043.png

What I don't understand is below the zoom level (here 05) I get 000
followed by 000. Then the y followed by another double 000 before I
get
the x with my extension.

Where does all these 000 sub directories comes from and what
functions
do they have?

I don't think this affects the performance, but this must cause
creating
thousands and thousands of unnecessary sub directories?

Thanks.

Trygve
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache disk cache gives unexpected paths to tiles

[thomas bonfort]

You should remove the /mapcache from the >cache> block, as it
takes precedence over your 

--
thomas

On Wed, Feb 9, 2022 at 12:10 PM Trygve Aspenes 
wrote:

> Hi
>
> I have a mapcache running using disk as cache method with config like
> this:
>
>  
>  /mapcache
>
> /mapcache/{tileset}#{grid}#{dim}/{z}/{x}/{y}.{ext}
>  
>
> But here is what I get on disk:
>
> /mapcache/ name>/EPSG:3857/2022-02-05T00:12:00Z/05/000/000/038/000/000/043.png
>
> What I don't understand is below the zoom level (here 05) I get 000
> followed by 000. Then the y followed by another double 000 before I get
> the x with my extension.
>
> Where does all these 000 sub directories comes from and what functions
> do they have?
>
> I don't think this affects the performance, but this must cause creating
> thousands and thousands of unnecessary sub directories?
>
> Thanks.
>
> Trygve
> ___
> MapServer-users mailing list
> MapServer-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache disk cache gives unexpected paths to tiles

[Trygve Aspenes]

Hi

I have a mapcache running using disk as cache method with config like 
this:



/mapcache

/mapcache/{tileset}#{grid}#{dim}/{z}/{x}/{y}.{ext}



But here is what I get on disk:

/mapcache/name>/EPSG:3857/2022-02-05T00:12:00Z/05/000/000/038/000/000/043.png


What I don't understand is below the zoom level (here 05) I get 000 
followed by 000. Then the y followed by another double 000 before I get 
the x with my extension.


Where does all these 000 sub directories comes from and what functions 
do they have?


I don't think this affects the performance, but this must cause creating 
thousands and thousands of unnecessary sub directories?


Thanks.

Trygve
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache 1.12.0rc1

[jbo-...@mailo.com]

The MapCache / MapServer team is pleased to announce the first release 
candidate of MapCache 1.12.0.


Change log from 1.10 is found at 
https://mapserver.org/development/changelog/mapcache/changelog-1-12.html#changes-from-1-10-0-to-1-12-0-rc1


An archive of source code is found at 
https://download.osgeo.org/mapserver/mapcache-1.12.0-rc1.tar.gz


Please help us test and let us know of any findings 
(https://github.com/MapServer/mapcache/issues/new).


--
The MapCache / MapServer team




___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache time dimension default value; and 'tearing down pooled connection' message

[jbo-...@mailo.com]

Hello,

Despite being one of "the mapcache devs", I have no answer to the second 
question. This involves MapCache connection pool which uses APR pools 
and which I am not familiar with:


https://github.com/MapServer/mapcache/blob/8b4d277df4f7951edd84e4fb8903dc544c3315a9/lib/connection_pool.c#L153

As for the first question, "d1" is not a magical value: this is just a 
placeholder in the documentation for an actual default time value or 
time interval.


https://mapserver.org/mapcache/dimensions.html?highlight=d1#configuring-a-time-dimension-using-an-explicit-back-end

Jerome



Le 18/01/2022 à 02:52, David Hoese a écrit :

Trygve,

Hello from outside the Pytroll community ;)

I can't answer your second question as I don't have much experience 
with getCapabilities, but I think I have an answer for the first 
question. My understanding for the `default` parameter was that that 
was the returned time string if no times were returned by the query 
(empty result). I have always expected this to be a normal time string 
like 2022-01-01T12:00:00Z (to match your expected format. However, the 
fact that you always get the oldest image when the default is "d1" is 
surprising to me so I hope the mapcache devs can correct me.


As for the source code for this, I found that the default from the XML 
goes into a dimension object as "default_value". It then gets set as 
the default "requested_value" like here:


https://github.com/MapServer/mapcache/blob/447c0c6848fd101c11e6d54e451037e571117ef1/lib/tileset.c#L585 



Going down the rabbit hole led me to here:

https://github.com/MapServer/mapcache/blob/447c0c6848fd101c11e6d54e451037e571117ef1/lib/dimension_pg.c#L265-L271 



As far as I can tell "get_values_for_entry_query" is just the 
"validate_query" query from your XML config. So somehow your validate 
query with "dp1" input may be returning the oldest time in your 
database. Just a guess. I'm interested in this answer, but definitely 
not the best person to give it to you.


Dave

On 1/17/22 11:59, Trygve Aspenes wrote:

Hi there!

Slowly I'm getting mapserver/mapcache up running and looks now how to 
improve things.


First: When I load a mapcache tileset with time dimension from a pg 
db the oldest image is displayed in my client. Is it possible to 
somehow get the newest image as default?

My tileset dimension look something like this:

 time="true" unit="ISO8601">

 
 host=mapserver-db user=$PGSQ_USERNAME 
password=$PGSQ_PASSWORD dbname=mapserver port=5432

 
 SELECT to_char(time, 
'-MM-DD"T"HH24:MI:SS"Z"') FROM products where product_name='product>' order by Id

 
 SELECT 
to_char(time,'-MM-DD"T"HH24:MI:SS"Z"') FROM products

 WHERE time = to_timestamp(:start_timestamp)
 AND time = to_timestamp(:end_timestamp)
 AND product_name=''
 ORDER BY time DESC
 
 

I saw the `d1` default value in the documentation, but don't 
understand of that is a dummy or an actual selector. I have grepped 
around the source code, but could not see anything obvious.


Second: My client updates/fetch the getCapabilities of my mapcache 
server every minute. Each time this is fetched the mapcache log is 
filled with messages like:


mod_mapcache.c(164): [client ] tearing down pooled connection 
(dim__time) to make room, referer: 


I dont understand why these connections is teared down every time 
getCapabilites is requested.


I have looked in the documentation, but could not see how to control 
this. But in the mapcache.xml.sample the connection pool config is 
described. But adding this to my mapcache config does not remove 
these tear downs.


 
 2000
12000
 

Any suggestions why this connection tear down is happening and if 
there is something to do about it. It slows down the getCapabilities 
request a bit, but if I add to many tileset I guess this will even 
further slow down.


Thanks

Trygve Aspenes
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users




___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache time dimension default value; and 'tearing down pooled connection' message

[David Hoese]

Trygve,

Hello from outside the Pytroll community ;)

I can't answer your second question as I don't have much experience with 
getCapabilities, but I think I have an answer for the first question. My 
understanding for the `default` parameter was that that was the returned 
time string if no times were returned by the query (empty result). I 
have always expected this to be a normal time string like 
2022-01-01T12:00:00Z (to match your expected format. However, the fact 
that you always get the oldest image when the default is "d1" is 
surprising to me so I hope the mapcache devs can correct me.


As for the source code for this, I found that the default from the XML 
goes into a dimension object as "default_value". It then gets set as the 
default "requested_value" like here:


https://github.com/MapServer/mapcache/blob/447c0c6848fd101c11e6d54e451037e571117ef1/lib/tileset.c#L585

Going down the rabbit hole led me to here:

https://github.com/MapServer/mapcache/blob/447c0c6848fd101c11e6d54e451037e571117ef1/lib/dimension_pg.c#L265-L271

As far as I can tell "get_values_for_entry_query" is just the 
"validate_query" query from your XML config. So somehow your validate 
query with "dp1" input may be returning the oldest time in your 
database. Just a guess. I'm interested in this answer, but definitely 
not the best person to give it to you.


Dave

On 1/17/22 11:59, Trygve Aspenes wrote:

Hi there!

Slowly I'm getting mapserver/mapcache up running and looks now how to 
improve things.


First: When I load a mapcache tileset with time dimension from a pg db 
the oldest image is displayed in my client. Is it possible to somehow 
get the newest image as default?

My tileset dimension look something like this:

     time="true" unit="ISO8601">

     
     host=mapserver-db user=$PGSQ_USERNAME 
password=$PGSQ_PASSWORD dbname=mapserver port=5432

     
     SELECT to_char(time, 
'-MM-DD"T"HH24:MI:SS"Z"') FROM products where product_name='product>' order by Id

     
     SELECT to_char(time,'-MM-DD"T"HH24:MI:SS"Z"') 
FROM products

     WHERE time = to_timestamp(:start_timestamp)
     AND time = to_timestamp(:end_timestamp)
     AND product_name=''
     ORDER BY time DESC
     
     

I saw the `d1` default value in the documentation, but don't understand 
of that is a dummy or an actual selector. I have grepped around the 
source code, but could not see anything obvious.


Second: My client updates/fetch the getCapabilities of my mapcache 
server every minute. Each time this is fetched the mapcache log is 
filled with messages like:


mod_mapcache.c(164): [client ] tearing down pooled connection (dim_product>_time) to make room, referer: 


I dont understand why these connections is teared down every time 
getCapabilites is requested.


I have looked in the documentation, but could not see how to control 
this. But in the mapcache.xml.sample the connection pool config is 
described. But adding this to my mapcache config does not remove these 
tear downs.


     
     2000
     12000
     

Any suggestions why this connection tear down is happening and if there 
is something to do about it. It slows down the getCapabilities request a 
bit, but if I add to many tileset I guess this will even further slow down.


Thanks

Trygve Aspenes
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache time dimension default value; and 'tearing down pooled connection' message

[Trygve Aspenes]

Hi there!

Slowly I'm getting mapserver/mapcache up running and looks now how to 
improve things.


First: When I load a mapcache tileset with time dimension from a pg db 
the oldest image is displayed in my client. Is it possible to somehow 
get the newest image as default?

My tileset dimension look something like this:

time="true" unit="ISO8601">


host=mapserver-db user=$PGSQ_USERNAME 
password=$PGSQ_PASSWORD dbname=mapserver port=5432


SELECT to_char(time, 
'-MM-DD"T"HH24:MI:SS"Z"') FROM products where product_name='product>' order by Id


SELECT to_char(time,'-MM-DD"T"HH24:MI:SS"Z"') 
FROM products

WHERE time = to_timestamp(:start_timestamp)
AND time = to_timestamp(:end_timestamp)
AND product_name=''
ORDER BY time DESC



I saw the `d1` default value in the documentation, but don't understand 
of that is a dummy or an actual selector. I have grepped around the 
source code, but could not see anything obvious.


Second: My client updates/fetch the getCapabilities of my mapcache 
server every minute. Each time this is fetched the mapcache log is 
filled with messages like:


mod_mapcache.c(164): [client ] tearing down pooled connection (dim_product>_time) to make room, referer: 


I dont understand why these connections is teared down every time 
getCapabilites is requested.


I have looked in the documentation, but could not see how to control 
this. But in the mapcache.xml.sample the connection pool config is 
described. But adding this to my mapcache config does not remove these 
tear downs.



2000
12000


Any suggestions why this connection tear down is happening and if there 
is something to do about it. It slows down the getCapabilities request a 
bit, but if I add to many tileset I guess this will even further slow 
down.


Thanks

Trygve Aspenes
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache behind nginx with ssl

[Trygve Aspenes]

OK So I will answer my self then after a major hint from a mapcache 
user.


In the mapcache.xml.sample file there is a section 
https://github.com/MapServer/mapcache/blob/11f7713c5efb730a1699dfd35bbf7300b38c8e06/mapcache.xml.sample#L7-L14 
Where you can do exactly what I want.


This solves my problem


Den 2021-09-27 22:40, skrev Trygve Aspenes:

Hi there

I have put up a mapserver behind nginx with ssl and a curl like this

curl -s
"https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map=WMS=GetCapabilities;
| grep OnlineResource

Gives me like:

 http://www.w3.org/1999/xlink;
xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
  http://www.w3.org/1999/xlink;
xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
  http://www.w3.org/1999/xlink;
xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
  http://www.w3.org/1999/xlink;
xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
  http://www.w3.org/1999/xlink;
xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>

Yeah you get the idea. Everything perfect with https in the href link. 
Thanks!


So I put up a mapcache which I intend to put up i front of mapserver.
When I do a similar curl like:

curl -s
"https://mapcache/mapcache?Overview=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map=WMS=GetCapabilities;
| xmllint --format - | grep OnlineResource

I get:

http://www.w3.org/1999/xlink;
xlink:href="http://mapcache:8090/mapcache?"/>
  http://www.w3.org/1999/xlink;
xlink:href="http://mapcache:8090/mapcache?"/>
  http://www.w3.org/1999/xlink;
xlink:href="http://mapcache:8090/mapcache?"/>
  http://www.w3.org/1999/xlink;
xlink:href="http://mapcache:8090/mapcache?"/>

I here starts the problems. If I do this in eg QGIS, QGIS are told to
go to the http address with port 8090(mapcache is setup to run on port
8090 behind the scenes) which is not reachable. It should return https
(with no specified port, ie. using standard https port) if I
understands correctly.

And I just cant figure put why.

The configuration for the two are almost identical and I have double
and tripple check my config.

Any idea what I should check next?

If it could be of any help I could open access to my mapcache server 
running.


Thanks

Trygve

PS: when I test mapcache and mapserver setup locally without ssl and
nginx in front it works fine.
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache behind nginx with ssl

[Bjørn Ove Grøtan]

I had the same issue with HAProxy in front and Mapserver on Apache in the back.
1. http->http (returns http-urls)
2. https->http (returns http-urls)
3. https->https (returns https-urls)

Even if I tell the backend that the client request protocol was https on port 
443
(forwardfor,forwardport,forwardprotocol). It would be nice if Mapserver could 
utilize
information from forwardport/forwardprotocol, so we could do ssl-offloading 
instead
of ssl-passthrough - if we wanted to.

My solution was to serve a dummy-certificate on the backend, and configure alt. 
1 and 2.

-Bjørn Ove

Trygve Aspenes:
> Hi there
> 
> I have put up a mapserver behind nginx with ssl and a curl like this
> 
> curl -s 
> "https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map=WMS=GetCapabilities;
> | grep OnlineResource
> 
> Gives me like:
> 
>  http://www.w3.org/1999/xlink; 
> xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
>   http://www.w3.org/1999/xlink; 
> xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
>   http://www.w3.org/1999/xlink; 
> xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
>   http://www.w3.org/1999/xlink; 
> xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
>   http://www.w3.org/1999/xlink; 
> xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
> 
> Yeah you get the idea. Everything perfect with https in the href link.
> Thanks!
> 
> So I put up a mapcache which I intend to put up i front of mapserver. When I
> do a similar curl like:
> 
> curl -s 
> "https://mapcache/mapcache?Overview=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map=WMS=GetCapabilities;
> | xmllint --format - | grep OnlineResource
> 
> I get:
> 
> http://www.w3.org/1999/xlink;
> xlink:href="http://mapcache:8090/mapcache?"/>
>   http://www.w3.org/1999/xlink;
> xlink:href="http://mapcache:8090/mapcache?"/>
>   http://www.w3.org/1999/xlink;
> xlink:href="http://mapcache:8090/mapcache?"/>
>   http://www.w3.org/1999/xlink;
> xlink:href="http://mapcache:8090/mapcache?"/>
> 
> I here starts the problems. If I do this in eg QGIS, QGIS are told to go to
> the http address with port 8090(mapcache is setup to run on port 8090 behind
> the scenes) which is not reachable. It should return https (with no
> specified port, ie. using standard https port) if I understands correctly.
> 
> And I just cant figure put why.
> 
> The configuration for the two are almost identical and I have double and
> tripple check my config.
> 
> Any idea what I should check next?
> 
> If it could be of any help I could open access to my mapcache server
> running.
> 
> Thanks
> 
> Trygve
> 
> PS: when I test mapcache and mapserver setup locally without ssl and nginx
> in front it works fine.
> ___
> MapServer-users mailing list
> MapServer-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] mapcache behind nginx with ssl

[Trygve Aspenes]

Hi there

I have put up a mapserver behind nginx with ssl and a curl like this

curl -s 
"https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map=WMS=GetCapabilities; 
| grep OnlineResource


Gives me like:

 http://www.w3.org/1999/xlink; 
xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
  xmlns:xlink="http://www.w3.org/1999/xlink; 
xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
  xmlns:xlink="http://www.w3.org/1999/xlink; 
xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
  xmlns:xlink="http://www.w3.org/1999/xlink; 
xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>
  xmlns:xlink="http://www.w3.org/1999/xlink; 
xlink:href="https://mapserver/cgi-bin/mapserv?map=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map"/>


Yeah you get the idea. Everything perfect with https in the href link. 
Thanks!


So I put up a mapcache which I intend to put up i front of mapserver. 
When I do a similar curl like:


curl -s 
"https://mapcache/mapcache?Overview=/data/mapserver/map-files/mapserver-8505ad3e-f9e3-4de3-a080-8253443ac954-20210908T120432Z.map=WMS=GetCapabilities; 
| xmllint --format - | grep OnlineResource


I get:

http://www.w3.org/1999/xlink; 
xlink:href="http://mapcache:8090/mapcache?"/>
  http://www.w3.org/1999/xlink; 
xlink:href="http://mapcache:8090/mapcache?"/>
  http://www.w3.org/1999/xlink; 
xlink:href="http://mapcache:8090/mapcache?"/>
  http://www.w3.org/1999/xlink; 
xlink:href="http://mapcache:8090/mapcache?"/>


I here starts the problems. If I do this in eg QGIS, QGIS are told to go 
to the http address with port 8090(mapcache is setup to run on port 8090 
behind the scenes) which is not reachable. It should return https (with 
no specified port, ie. using standard https port) if I understands 
correctly.


And I just cant figure put why.

The configuration for the two are almost identical and I have double and 
tripple check my config.


Any idea what I should check next?

If it could be of any help I could open access to my mapcache server 
running.


Thanks

Trygve

PS: when I test mapcache and mapserver setup locally without ssl and 
nginx in front it works fine.

___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache: Memory leak in WMTSCapabilities request?

[Schmid Andreas]

Hi

I'm running MapCache 1.10.0 in a Docker container. The image is based on 
debian:bullseye; Apache and MapCache is installed into the image with apt. 
Everything works fine; however when I set the health checks to regularly 
request the WMTS Capabilities at mapcache/wmts/1.0.0/WMTSCapabilities.xml, the 
memory usage of the container increases continuously. In a test setup with 12 
of these requests per minute and no other requests, the increase is around 25MB 
per day.
When I set the health checks to simply request a certain tile, there is no such 
increase of memory usage.

Does this mean there is a memory leak in the WMTSCapabilities functionality? 
Something similar has been described in 
http://osgeo-org.1560.x6.nabble.com/Memory-leak-in-mapcache-td5401587.html two 
years ago, but no real solution has been found there.


Best regards,
Andy
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache WMTS tile artifacts

[Yoann CHARIEAU]

Hi everyone, 

I have configured a mapcache server with a few sources. 
Everything seems to work fine except for some wmts sources : i get random 
purple/blue artifacts on tiles (screenshots : https://imgur.com/a/hXgWxao) 
So far i tried playing around gdal_wmts xml config file, with different caches 
(disk, sqlite, memcache), formats (png and jpg, with rgb and ycbcr colorspace) 
and grids, and also tried updating gdal. 
Strangest thing is that artifacts disappear on tile expiration or by manually 
deleting tiles. 
Right now i’m thinking about network issues. 
Has anyone ever encountered these kind of artifacts ? 

Thanks ! 

Yoann 
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache WMS layer group - WMTS Themes

[BRIAN STIFFENN LUNA BOLIVAR]

 Hello everyone, good day.I'm trying hard to set the WMS layer group as in
the mapfile, and WMTS theme section too, but I couldn't find anything in
the mapcache documentation. Do you know the correct way to set this?

Thank you.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache building question

[jbo-ads]

Hi Иван,

Actually, for compilation, CMake needs to locate 'httpd.h' header file 
and 'apxs' or 'apxs2' executables. You may need to install specific 
packages, e.g. on Ubuntu: 'apt-get install apache2-dev'.


Jérome.


Le 31/08/2020 à 08:44, Иван Иванов a écrit :

Hi everyone!
I have a trouble while trying to cmake mapcache. I've resolved all 
dependencies problems, except one about APACHE, here is the output of 
cmake command:


```
- Could NOT find APACHE (missing:  APACHE_INCLUDE_DIR)
CMake Error at CMakeLists.txt:45 (message):
  APACHE library/component could not be found and is a mandatory 
dependency


    HINT:
    - add the APACHE install directory to the CMAKE_PREFIX_PATH 
variable 
(-DCMAKE_PREFIX_PATH="/path/to/APACHE-install-dir;/path/to/other/dirs"

Call Stack (most recent call first):
```

Next i've added suggested parameter: ```cmake 
-DCMAKE_PREFIX_PATH="/etc/apache2" ..```

But nothing have changed - the same error remains.

The contents of ```/etc/apache2``` are:

```
-rw-r--r-- 1 root root  7263 июн 20  2018 apache2.conf
drwxr-xr-x 2 root root  4096 авг 18 10:27 conf-available
drwxr-xr-x 2 root root  4096 авг 18 10:28 conf-enabled
-rw-r--r-- 1 root root  1782 июн 20  2018 envvars
-rw-r--r-- 1 root root 31063 июн 20  2018 magic
drwxr-xr-x 2 root root 12288 авг 18 11:52 mods-available
drwxr-xr-x 2 root root  4096 авг 18 11:52 mods-enabled
-rw-r--r-- 1 root root   344 авг 18 11:58 ports.conf
drwxr-xr-x 2 root root  4096 авг 18 13:54 sites-available
drwxr-xr-x 2 root root  4096 авг 18 13:54 sites-enabled
```

What should i do to fix this problem?

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users



___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache building question

[Иван Иванов]

Hi everyone!
I have a trouble while trying to cmake mapcache. I've resolved all
dependencies problems, except one about APACHE, here is the output of cmake
command:

```
- Could NOT find APACHE (missing:  APACHE_INCLUDE_DIR)
CMake Error at CMakeLists.txt:45 (message):
  APACHE library/component could not be found and is a mandatory dependency

HINT:
- add the APACHE install directory to the CMAKE_PREFIX_PATH variable
(-DCMAKE_PREFIX_PATH="/path/to/APACHE-install-dir;/path/to/other/dirs"
Call Stack (most recent call first):
```

Next i've added suggested parameter: ```cmake
-DCMAKE_PREFIX_PATH="/etc/apache2" ..```
But nothing have changed - the same error remains.

The contents of ```/etc/apache2``` are:

```
-rw-r--r-- 1 root root  7263 июн 20  2018 apache2.conf
drwxr-xr-x 2 root root  4096 авг 18 10:27 conf-available
drwxr-xr-x 2 root root  4096 авг 18 10:28 conf-enabled
-rw-r--r-- 1 root root  1782 июн 20  2018 envvars
-rw-r--r-- 1 root root 31063 июн 20  2018 magic
drwxr-xr-x 2 root root 12288 авг 18 11:52 mods-available
drwxr-xr-x 2 root root  4096 авг 18 11:52 mods-enabled
-rw-r--r-- 1 root root   344 авг 18 11:58 ports.conf
drwxr-xr-x 2 root root  4096 авг 18 13:54 sites-available
drwxr-xr-x 2 root root  4096 авг 18 13:54 sites-enabled
```

What should i do to fix this problem?
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache dimensions with MapServer run time substitution

[Travis Kirstine]

Is it possible to utilize mapcache regex dimensions with mapserver run time
substitution (wms) to create a more dynamic and generic tile service

Here is my basic configuration for mapserver and mapcache, I can't seem to
get it working with QGIS when connecting to the WMTS

For example I use run time substitution with my PG DATA settings

  LAYER
NAME "sub_layer"
...
VALIDATION
  'layer_table' ^.*$'
  'layer_class' '^.*$'
END
CONNECTIONTYPE postgis
CONNECTION "port= user=user password=pw dbname=db host=mydb"
DATA "the_geom from (SELECT gid, ms_color, the_geom FROM %layer_table%
WHERE class='%layer_class%') AS foo USING UNIQUE gid using SRID=4326"
 
  END

I set my MapCache config up with the same runtime variables as dimensions

   
  
 
image/png
true
sub_layer
 
  
  
 http://testserver.com/cgi-bin/wms?
  
   
   
  source_wms
  composite
  GoogleMapsCompatible
  PNG
  5 5
  10
  3600
  604800
  

  ^.*$


  ^.*$

  
   
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] MapCache Larger tiles, reduce number of files

[Jean-Philippe Chenel]

Dear Steve,

Thanks for your help and here is the solution.

The grid in mapcache.xml was good (same resolution as the default with 256x256, 
the only difference is the size of the tile)


-20037508.3427892480 -20037508.3427892480 20037508.3427892480 
20037508.3427892480
EPSG:3857
m
512 512
156543.0339280410 78271.51696402048 39135.75848201023 
19567.87924100512 9783.939620502561 4891.969810251280 2445.984905125640 
1222.992452562820 611.4962262814100 305.7481131407048 152.8740565703525 
76.43702828517624 38.21851414258813 19.10925707129406 9.554628535647032 
4.777314267823516 2.388657133911758 1.194328566955879 
0.5971642834779395


The problem was in the OL configuration, when you said that the tilesize is 
something else, I suppose that the resolution also changes without to do 
anything. The solution was to make another tilegrid configuration like that and 
to bind it to the source.

var projExtent = ol.proj.get('EPSG:3857').getExtent();
var startResolution = ol.extent.getWidth(projExtent) / 256;
var resolutions = new Array(22);
var tileSize = 512;
var urlTemplate = '/GoogleMaps512/{z}/{x}/{y}.png';

for (var i = 0, ii = resolutions.length; i < ii; ++i) {
  resolutions[i] = startResolution / Math.pow(2, i);
}

var tileGrid = new ol.tilegrid.TileGrid({
  extent: [-20037508.3427892480, -20037508.3427892480, 20037508.3427892480, 
20037508.3427892480],
  resolutions: resolutions,
  tileSize: [512, 512]
});

var source1 = new ol.source.XYZ({
url: urlTemplate,
tileSize: tileSize,
tileGrid: tileGrid
});

With best regards,


De : Stephen Woodbridge 
Envoyé : 9 mars 2020 16:46
À : Jean-Philippe Chenel ; mapserver-users@lists.osgeo.org 

Objet : Re: [mapserver-users] MapCache Larger tiles, reduce number of files

Are you sure you have the set the tile size correctly in OpenLayers?
Because if OL is requesting tiles that you don't have it sounds like
that is the reason.
And if you do load tile and there is a miss-match, then I would expect
the resolution of be off.

Here are a few links that might help:

https://openlayers.org/en/latest/examples/wms-custom-tilegrid-512x256.html
https://openlayers.org/en/latest/apidoc/module-ol_tilegrid_TileGrid-TileGrid.html
https://stackoverflow.com/questions/57072232/how-to-change-tilesize-in-xyz-tilelayer-from-default256-256-to-something-e

On 3/9/2020 3:21 PM, Jean-Philippe Chenel wrote:
> Steve,
>
> It is what I supposed, but when I've generated the tiles 512x512 using
> the default resolution GoogleMapCompatible, with the cache template
> {z}/{x}/{y}.{ext} and loaded it in OpenLayers using ol.source.XYZ,
> I've noticied that the x tile being tried to load was not in the good
> range.
>
> z/x/-y.png
>
> Tiles generated:
> .../8/37 and 38/64 and 65.png
>
> Tiles loaded by OpenLayers (with error 404):
> .../8/73 to 77/130 and 131.png
>
> Thanks,
> 
> *De :* mapserver-users  de la
> part de Stephen Woodbridge 
> *Envoyé :* 9 mars 2020 14:56
> *À :* mapserver-users@lists.osgeo.org 
> *Objet :* Re: [mapserver-users] MapCache Larger tiles, reduce number
> of files
> Jean-Philippe,
>
> I don't think it works like that. Resolution is determined by zoom
> factor so you generate  metatile at that resolution then you chop that
> metatile in tiles of whatever size you specify. So resolution is
> controled by zoom not tile size.
>
> -Steve W
>
> On 3/9/2020 2:45 PM, Jean-Philippe Chenel wrote:
> > Hi,
> >
> > With MapCache, I want to generate the tiles that will be copied to a
> > mobile device, on a viewer created with OpenLayers.
> >
> > When generating a large extent, with multiple levels of zoom, the
> > files count is really big, so I want to reduce the number of files by
> > increasing the tile size. I've made a test with tiles 512x512, but
> > I've realized that the resolution must be changed when changing tiles
> > size. So, the tiles are bigger, but changing the resolution make
> > generating the same number of files.
> >
> > I'm open for workaround and other tricks.
> >
> > mapcache.xml
> >
> > 
> > 
> > GoogleMaps512
> > 
> > -20037508.3427892480 -20037508.3427892480 20037508.3427892480
> > 20037508.3427892480
> > EPSG:3857
> > EPSG:900913
> > m
> > 512 512
> > 156543.0339280410 78271.51696402048 39135.75848201023
> > 19567.87924100512 9783.939620502561 4891.969810251280
> > 2445.984905125640 1222.992452562820 611.4962262814100
> > 305.7481131407048 152.8740565703525 76.43702828517624
> > 38.21851414258813 19.10925707129406 9.554628535647032
> > 4.777314267823516 2.388657133911758 1.194328566955879
> > 0.5971642834779395
> > 
> >
> > 
> >
> 

Re: [mapserver-users] MapCache Larger tiles, reduce number of files

[Stephen Woodbridge]

Are you sure you have the set the tile size correctly in OpenLayers?
Because if OL is requesting tiles that you don't have it sounds like 
that is the reason.
And if you do load tile and there is a miss-match, then I would expect 
the resolution of be off.


Here are a few links that might help:

https://openlayers.org/en/latest/examples/wms-custom-tilegrid-512x256.html
https://openlayers.org/en/latest/apidoc/module-ol_tilegrid_TileGrid-TileGrid.html
https://stackoverflow.com/questions/57072232/how-to-change-tilesize-in-xyz-tilelayer-from-default256-256-to-something-e

On 3/9/2020 3:21 PM, Jean-Philippe Chenel wrote:

Steve,

It is what I supposed, but when I've generated the tiles 512x512 using 
the default resolution GoogleMapCompatible, with the cache template 
{z}/{x}/{y}.{ext} and loaded it in OpenLayers using ol.source.XYZ, 
I've noticied that the x tile being tried to load was not in the good 
range.


z/x/-y.png

Tiles generated:
.../8/37 and 38/64 and 65.png

Tiles loaded by OpenLayers (with error 404):
.../8/73 to 77/130 and 131.png

Thanks,

*De :* mapserver-users  de la 
part de Stephen Woodbridge 

*Envoyé :* 9 mars 2020 14:56
*À :* mapserver-users@lists.osgeo.org 
*Objet :* Re: [mapserver-users] MapCache Larger tiles, reduce number 
of files

Jean-Philippe,

I don't think it works like that. Resolution is determined by zoom
factor so you generate  metatile at that resolution then you chop that
metatile in tiles of whatever size you specify. So resolution is
controled by zoom not tile size.

-Steve W

On 3/9/2020 2:45 PM, Jean-Philippe Chenel wrote:
> Hi,
>
> With MapCache, I want to generate the tiles that will be copied to a
> mobile device, on a viewer created with OpenLayers.
>
> When generating a large extent, with multiple levels of zoom, the
> files count is really big, so I want to reduce the number of files by
> increasing the tile size. I've made a test with tiles 512x512, but
> I've realized that the resolution must be changed when changing tiles
> size. So, the tiles are bigger, but changing the resolution make
> generating the same number of files.
>
> I'm open for workaround and other tricks.
>
> mapcache.xml
>
> 
> 
> GoogleMaps512
> 
> -20037508.3427892480 -20037508.3427892480 20037508.3427892480
> 20037508.3427892480
> EPSG:3857
> EPSG:900913
> m
> 512 512
> 156543.0339280410 78271.51696402048 39135.75848201023
> 19567.87924100512 9783.939620502561 4891.969810251280
> 2445.984905125640 1222.992452562820 611.4962262814100
> 305.7481131407048 152.8740565703525 76.43702828517624
> 38.21851414258813 19.10925707129406 9.554628535647032
> 4.777314267823516 2.388657133911758 1.194328566955879
> 0.5971642834779395
> 
>
> 
> 
/data/mapcache/tiles/{tileset}/{grid}/{z}/{x}/{y}.{ext}

> 
>
> 
> vmap0
> tmpl_xyz
> GoogleMaps512
> PNG
> 
>
> With best regards,
>
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] MapCache Larger tiles, reduce number of files

[Jean-Philippe Chenel]

Steve,

It is what I supposed, but when I've generated the tiles 512x512 using the 
default resolution GoogleMapCompatible, with the cache template 
{z}/{x}/{y}.{ext} and loaded it in OpenLayers using ol.source.XYZ, I've 
noticied that the x tile being tried to load was not in the good range.

z/x/-y.png

Tiles generated:
.../8/37 and 38/64 and 65.png

Tiles loaded by OpenLayers (with error 404):
.../8/73 to 77/130 and 131.png

Thanks,

De : mapserver-users  de la part de 
Stephen Woodbridge 
Envoyé : 9 mars 2020 14:56
À : mapserver-users@lists.osgeo.org 
Objet : Re: [mapserver-users] MapCache Larger tiles, reduce number of files

Jean-Philippe,

I don't think it works like that. Resolution is determined by zoom
factor so you generate  metatile at that resolution then you chop that
metatile in tiles of whatever size you specify. So resolution is
controled by zoom not tile size.

-Steve W

On 3/9/2020 2:45 PM, Jean-Philippe Chenel wrote:
> Hi,
>
> With MapCache, I want to generate the tiles that will be copied to a
> mobile device, on a viewer created with OpenLayers.
>
> When generating a large extent, with multiple levels of zoom, the
> files count is really big, so I want to reduce the number of files by
> increasing the tile size. I've made a test with tiles 512x512, but
> I've realized that the resolution must be changed when changing tiles
> size. So, the tiles are bigger, but changing the resolution make
> generating the same number of files.
>
> I'm open for workaround and other tricks.
>
> mapcache.xml
>
> 
> 
> GoogleMaps512
> 
> -20037508.3427892480 -20037508.3427892480 20037508.3427892480
> 20037508.3427892480
> EPSG:3857
> EPSG:900913
> m
> 512 512
> 156543.0339280410 78271.51696402048 39135.75848201023
> 19567.87924100512 9783.939620502561 4891.969810251280
> 2445.984905125640 1222.992452562820 611.4962262814100
> 305.7481131407048 152.8740565703525 76.43702828517624
> 38.21851414258813 19.10925707129406 9.554628535647032
> 4.777314267823516 2.388657133911758 1.194328566955879
> 0.5971642834779395
> 
>
> 
> /data/mapcache/tiles/{tileset}/{grid}/{z}/{x}/{y}.{ext}
> 
>
> 
> vmap0
> tmpl_xyz
> GoogleMaps512
> PNG
> 
>
> With best regards,
>
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] MapCache Larger tiles, reduce number of files

[Stephen Woodbridge]

Jean-Philippe,

I don't think it works like that. Resolution is determined by zoom 
factor so you generate  metatile at that resolution then you chop that 
metatile in tiles of whatever size you specify. So resolution is 
controled by zoom not tile size.


-Steve W

On 3/9/2020 2:45 PM, Jean-Philippe Chenel wrote:

Hi,

With MapCache, I want to generate the tiles that will be copied to a 
mobile device, on a viewer created with OpenLayers.


When generating a large extent, with multiple levels of zoom, the 
files count is really big, so I want to reduce the number of files by 
increasing the tile size. I've made a test with tiles 512x512, but 
I've realized that the resolution must be changed when changing tiles 
size. So, the tiles are bigger, but changing the resolution make 
generating the same number of files.


I'm open for workaround and other tricks.

mapcache.xml



GoogleMaps512

-20037508.3427892480 -20037508.3427892480 20037508.3427892480 
20037508.3427892480

EPSG:3857
EPSG:900913
m
512 512
156543.0339280410 78271.51696402048 39135.75848201023 
19567.87924100512 9783.939620502561 4891.969810251280 
2445.984905125640 1222.992452562820 611.4962262814100 
305.7481131407048 152.8740565703525 76.43702828517624 
38.21851414258813 19.10925707129406 9.554628535647032 
4.777314267823516 2.388657133911758 1.194328566955879 
0.5971642834779395




/data/mapcache/tiles/{tileset}/{grid}/{z}/{x}/{y}.{ext}



vmap0
tmpl_xyz
GoogleMaps512
PNG


With best regards,

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache Larger tiles, reduce number of files

[Jean-Philippe Chenel]

Hi,

With MapCache, I want to generate the tiles that will be copied to a mobile 
device, on a viewer created with OpenLayers.

When generating a large extent, with multiple levels of zoom, the files count 
is really big, so I want to reduce the number of files by increasing the tile 
size. I've made a test with tiles 512x512, but I've realized that the 
resolution must be changed when changing tiles size. So, the tiles are bigger, 
but changing the resolution make generating the same number of files.

I'm open for workaround and other tricks.

mapcache.xml



GoogleMaps512

-20037508.3427892480 -20037508.3427892480 20037508.3427892480 
20037508.3427892480
EPSG:3857
EPSG:900913
m
512 512
156543.0339280410 78271.51696402048 39135.75848201023 
19567.87924100512 9783.939620502561 4891.969810251280 2445.984905125640 
1222.992452562820 611.4962262814100 305.7481131407048 152.8740565703525 
76.43702828517624 38.21851414258813 19.10925707129406 9.554628535647032 
4.777314267823516 2.388657133911758 1.194328566955879 
0.5971642834779395



  /data/mapcache/tiles/{tileset}/{grid}/{z}/{x}/{y}.{ext}



vmap0
tmpl_xyz
GoogleMaps512
PNG


With best regards,
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache : Incompatible libpng version in application and library

[Johannes Paul]

I had indeed gdal running with libpng-1.2.56, while mapcache was compiled
with libpng-1.5.13.
Thanks a lot for your help.
Johannes

Le ven. 28 févr. 2020 à 18:48, Even Rouault  a
écrit :

> On vendredi 28 février 2020 11:38:38 CET Steve Lime wrote:
> > I'd think you must multiple libpng installed. Did you search your system
> > independent of the package manager?
>
> libpng-1.2.56 rings a bell to me. This is copy of libpng that is embedded
> in
> GDAL source tree. So you'd better make sure that your GDAL is built
> against
> system libpng, instead of the internal copy.
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache : Incompatible libpng version in application and library

[Even Rouault]

On vendredi 28 février 2020 11:38:38 CET Steve Lime wrote:
> I'd think you must multiple libpng installed. Did you search your system
> independent of the package manager?

libpng-1.2.56 rings a bell to me. This is copy of libpng that is embedded in 
GDAL source tree. So you'd better make sure that your GDAL is built against 
system libpng, instead of the internal copy.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache : Incompatible libpng version in application and library

[Steve Lime]

I'd think you must multiple libpng installed. Did you search your system
independent of the package manager?

On Wed, Feb 26, 2020 at 4:12 AM Johannes Paul 
wrote:

> Hello,
> I've compiled Mapcache from source, and the installation worked just fine,
> without any error, however when I run mapcache_seed, I get the following
> error :
>
> libpng warning: Application was compiled with png.h from libpng-1.5.13
> libpng warning: Application  is  running with png.c from libpng-1.2.56
> libpng error: Incompatible libpng version in application and library
>
> Mapcache has been compiled correctly using libpng 1.5.13, however I don't
> understand why mapcache_seed is trying to run with libpng 1.2.56 which is
> not even installed !
>
> Any idea ?
> Thanks,
> Johannes
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache : Incompatible libpng version in application and library

[Johannes Paul]

Hello,
I've compiled Mapcache from source, and the installation worked just fine,
without any error, however when I run mapcache_seed, I get the following
error :

libpng warning: Application was compiled with png.h from libpng-1.5.13
libpng warning: Application  is  running with png.c from libpng-1.2.56
libpng error: Incompatible libpng version in application and library

Mapcache has been compiled correctly using libpng 1.5.13, however I don't
understand why mapcache_seed is trying to run with libpng 1.2.56 which is
not even installed !

Any idea ?
Thanks,
Johannes
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache 1.10.0 released

[jbo-ads]

Hi,

The project team is glad to announce the release of MapCache 1.10.0.

This version includes a long list of bug fixes and feature enhancements 
since 1.8.0.


Changelog: 
https://mapserver.org/development/changelog/mapcache/changelog-1-10.html

Download: http://download.osgeo.org/mapserver/mapcache-1.10.0.tar.gz

Thanks,
MapServer / MapCache team


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache 1.10.0-rc2 ready for testing

[jbo-ads]

My bad, links were wrong in the previous message.

Hi all,

The second release candidate of MapCache 1.10.0 is out. It fixes some 
memory leaks.


Changelog: 
https://mapserver.org/development/changelog/mapcache/changelog-1-10.html#changelog-mapcache-1-10-0-rc2


Download: http://download.osgeo.org/mapserver/mapcache-1.10.0-rc2.tar.gz

Thank,
MapServer / MapCache team


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache 1.10.0-rc2 ready for testing

[jbo-ads]

Hi all,

The second release candidate of MapCache 1.10.0 is out. It fixes some 
memory leaks.


Changelog: 
https://mapserver.org/development/changelog/mapcache/changelog-1-10.html#changelog-mapcache-1-10-0-rc2


Download: http://download.osgeo.org/mapserver/mapcache-1.10.0-rc1.tar.gz

Thank,
MapServer / MapCache team


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache 1.10.0-rc1 ready for testing

[jbo-ads]

Today the project team has packaged the first release candidate of 
MapCache 1.10.0, which includes a long list of changes.


Changelog: 
https://mapserver.org/development/changelog/mapcache/changelog-1-10.html#changelog-mapcache-1-10-0-rc1


Download: http://download.osgeo.org/mapserver/mapcache-1.10.0-rc1.tar.gz

Thanks,

MapServer / MapCache team

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache Wiki Update: "MapCache Performance Analysis A Case Study"

[jbo-ads]

Hi all,

I've added an article to the young MapCache Wiki. This is called 
"MapCache Performance Analysis: A Case Study".




Here is the context of that study:

"A satellite imagery provider wants a browser to display its product 
catalog. Each product is an image of a small area on Earth (about 100km 
wide) and there are hundreds of products.


The solution is built around MapCache serving selected products through 
its WMS/WMTS map services. Products are preprocessed to take the form of 
single SQLite caches or Geopackages directly accessible by MapCache. 
Product selection takes place using dimension feature of those services. 
This is implemented with second level dimensions 
 
in MapCache, so that new products can be added without interrupting 
MapCache service."


Regards,

Jérome Boué.


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] MapCache clusters

[Sebastiano Laini]

Is not just the MapCache instances but also MapServer instances included so in 
theory you have X MapServers accepting request depending on the number of 
instances.
Making tests today I discover that if we use just Vector and Raster layer I can 
reach 100 tiles/s in a VM with 8core and 8GB RAM even though the ram is under 
1GB of use.
A 4core reach the same speed, and I was using –p 5 and 10 respectively.

Yesterday I came across this article 
https://www.maptiler.com/blog/2018/04/maps-with-docker.html and I was wondering 
if was something possible also with MapServer/MapCache.

Again if we use just Vector and Raster I think I can seed in 15 days instead of 
more than 60 days but is still a lot because I have 4 cache to seed twice a 
year.

Sebastiano Laini
Web Developer
Buchanan Computing

From: Travis Kirstine [mailto:traviskirst...@gmail.com]
Sent: 19 November 2019 16:11
To: Sebastiano Laini 
Cc: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] MapCache clusters

I don't think multiple instances of MapCache will help for seeding, in general 
most of the seeding time depends on the zoom level, with each higher zoom level 
taking 4 x the time as the proceeding zoom level (e.g. zoom 20 take 4 x the 
time to seed as zoom 19)  .  This can vary depending on your source service as 
there may be certain map scales / zoom levels that take longer per tile to 
render due to expression, labels, # of features etc... If you were to seed two 
levels for each mapcache instance you still would be waiting for the higher 
zoom levels to complete.   Using multiple load balanced mapserver nodes across 
multiple VM may improve things a bit however if all the VMs are hosted on the 
same physical server this may be slower overall.  For seeding I have heard you 
can use 2x the number of cores so you could set up a single mapcache server 
using -n 16 pointing at the load balanced source.

Regards



On Mon, 18 Nov 2019 at 10:47, Sebastiano Laini 
mailto:sebastiano.la...@buchanancomputing.co.uk>>
 wrote:
Hi All,

I was wondering if is possible to dockerise my server and then use multiple 
cluster to seed the cache? I know I can use multiple processes with the –n and 
–p flag though I’m wondering there is a limit on how much you can really 
increase those number.

E.g. I dockerise the server and I spawn a 8core VM with 16GB ram, I can use 
sure –p 8 or maybe 10 but still I think that will require like 60 days to seed 
the whole UK with our data, can I spawn another machine and speed up the 
process? For example another 8core VM.

We have 4 caches with different style that are updated twice a year so if it 
takes me 60 days for each by the time I finish it’s already time to start again.

I was wondering maybe I can run multiple VM and seed one or maybe 2 levels on 
each VM or if I spawn a new VM the original process will take advantage of the 
new VM?
In a VM I’m going to have MapServer and MapCache and I will use a subdomain 
under a load balancer just to seed while the domain will be use just for the 
seeded cache and in the same way will be under a load balancer.

Regards,

Sebastiano Laini
Web Developer
Buchanan Computing
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] MapCache clusters

[Travis Kirstine]

I don't think multiple instances of MapCache will help for seeding, in
general most of the seeding time depends on the zoom level, with each
higher zoom level taking 4 x the time as the proceeding zoom level (e.g.
zoom 20 take 4 x the time to seed as zoom 19)  .  This can vary depending
on your source service as there may be certain map scales / zoom levels
that take longer per tile to render due to expression, labels, # of
features etc... If you were to seed two levels for each mapcache instance
you still would be waiting for the higher zoom levels to complete.   Using
multiple load balanced mapserver nodes across multiple VM may improve
things a bit however if all the VMs are hosted on the same physical server
this may be slower overall.  For seeding I have heard you can use 2x the
number of cores so you could set up a single mapcache server using -n 16
pointing at the load balanced source.

Regards



On Mon, 18 Nov 2019 at 10:47, Sebastiano Laini <
sebastiano.la...@buchanancomputing.co.uk> wrote:

> Hi All,
>
>
>
> I was wondering if is possible to dockerise my server and then use
> multiple cluster to seed the cache? I know I can use multiple processes
> with the –n and –p flag though I’m wondering there is a limit on how much
> you can really increase those number.
>
>
>
> E.g. I dockerise the server and I spawn a 8core VM with 16GB ram, I can
> use sure –p 8 or maybe 10 but still I think that will require like 60 days
> to seed the whole UK with our data, can I spawn another machine and speed
> up the process? For example another 8core VM.
>
>
>
> We have 4 caches with different style that are updated twice a year so if
> it takes me 60 days for each by the time I finish it’s already time to
> start again.
>
>
>
> I was wondering maybe I can run multiple VM and seed one or maybe 2 levels
> on each VM or if I spawn a new VM the original process will take advantage
> of the new VM?
>
> In a VM I’m going to have MapServer and MapCache and I will use a
> subdomain under a load balancer just to seed while the domain will be use
> just for the seeded cache and in the same way will be under a load balancer.
>
>
>
> Regards,
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache clusters

[Sebastiano Laini]

Hi All,

I was wondering if is possible to dockerise my server and then use multiple 
cluster to seed the cache? I know I can use multiple processes with the -n and 
-p flag though I'm wondering there is a limit on how much you can really 
increase those number.

E.g. I dockerise the server and I spawn a 8core VM with 16GB ram, I can use 
sure -p 8 or maybe 10 but still I think that will require like 60 days to seed 
the whole UK with our data, can I spawn another machine and speed up the 
process? For example another 8core VM.

We have 4 caches with different style that are updated twice a year so if it 
takes me 60 days for each by the time I finish it's already time to start again.

I was wondering maybe I can run multiple VM and seed one or maybe 2 levels on 
each VM or if I spawn a new VM the original process will take advantage of the 
new VM?
In a VM I'm going to have MapServer and MapCache and I will use a subdomain 
under a load balancer just to seed while the domain will be use just for the 
seeded cache and in the same way will be under a load balancer.

Regards,

Sebastiano Laini
Web Developer
Buchanan Computing
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed speed optimization

[Stephen Woodbridge]

Thank Lars.

You can now fetch it from https://github.com/woodbri/imaptools.com

Enjoy,
  -Steve W

On 11/11/2019 3:07 PM, lars.schylb...@blixtmail.se wrote:

Hi

I have had a similar situation as You with large national data sets in 
shape format. One of the biggest improvements for me has been to split 
large shape files into smaller and

then use vector tile indexes in Mapserver.
The tool shp2tile has been working well for me. Stephen Woodbridge has 
written that tool. It can be found at 
http://imaptools.com/download-software.html

I think that You had to compile yourself on linux, but that was easy.

I have an example where I have automated this in a script that runs 
through all shape file in a directory and splits shape files larger 
than a certain size and then creates the vector tile index files.

The code snippet that is part of larger import script can be found here:
https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359

The script snippet also shows the normal optimizations we do.

Hope this could give You some ideas how optimize and make Mapserver 
and Mapcache run faster. I also ran into the problem with to many inodes.

That got solved by using sqlite as backend in Mapcache.

Best regards / Lars Schylberg



11 november 2019 kl. 15:42, "Travis Kirstine" 
> 
skrev:


I may not be unrealistic for it take mapserver to 4-6 seconds to
generate 4096px image depending on the amount data being rendered
and the complexity of your styles / expressions / labels. If your
mapserver is able to generate a 1024x1024 in under a second that
is pretty good. Like Jukka and other suggest I would look at
translating the files to shp files with spatial indexes or postgis
tables with spatial and attribute indexes if necessary.
Preprocessing the data (simplification / prefiltering / point
thinning cluster) and using SCALETOKENs to reference different
source layers at different scales can lead to big performance
boosts at the mapserver end.
For us the biggest limiting factor has been the write blocking to
sqlite cache when seeding tiles, if you increase the number of
threads / process in mapcache seed those process will just end up
waiting to write to the cache. You can generally figure out what
the max number of processes the cache can handle by running a seed
on a test area and look at the number of tiles seeded per second,
as some point -n will have no effect which your process are just
waiting to write the cache. I would look at using this method to
test sqlite vs disk or other backends. The GeoTiff cache may be
promising as it appears to be non-blocking but experimental...
We use a riak cache with a riak leveldb backend which works well
for us but is a bit of a pain to manage as it difficult to delete
objects so it's not great for caches that need to be refreshed.
Finally you may want to test MapProxy as an alternative to
MapCache (we use both). MapProxy supports a non-blocking compact
cache that could solve your inode problem
Regards
On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>> wrote:

We don’t create the data, we rely on OS (ordnance survey) to
supply us the maps files and then we publish them in our
service but I assume that probably we will need to create some
flow to improve it or download it in other format that is faster.

Sebastiano Laini

Web Developer

Buchanan Computing

*From:*Fawcett, David (MNIT) [mailto:david.fawc...@state.mn.us
]
*Sent:* 08 November 2019 16:41
*To:* Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'Rahkonen
Jukka (MML)' mailto:jukka.rahko...@maanmittauslaitos.fi>>;
'mapserver-users@lists.osgeo.org
'
mailto:mapserver-users@lists.osgeo.org>>
*Subject:* RE: mapcache seed speed optimization

For data formats that are slower to read, would it add fit
your workflow to convert it to different data format before
creating the tiles?

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/mapserver-users 





___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed speed optimization

[Sebastiano Laini]

I manage to seed at 300 tiles/s in the less complex layers, now I need to speak 
with the technical director and see how we are going to optimize our data.

I was trying to test MapProxy but I couldn’t install it in our Centos 7 
machine, I may give it a try again.
For your experience which one is the fastest one? Not just seeding but in 
delivering the requests also.

What tiles size are you using normally? Are you still using HTTP1 or HTTP2?

Regards,

Sebastiano Laini
Web Developer
Buchanan Computing

From: Travis Kirstine [mailto:traviskirst...@gmail.com]
Sent: 11 November 2019 14:42
To: Sebastiano Laini 
Cc: Fawcett, David (MNIT) ; Rahkonen Jukka (MML) 
; mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] mapcache seed speed optimization

I may not be unrealistic for it take mapserver to 4-6 seconds to generate 
4096px image depending on the amount data being rendered and the complexity of 
your styles / expressions / labels.  If your mapserver is able to generate a 
1024x1024 in under a second that is pretty good.  Like Jukka and other suggest 
I would look at translating the files to shp files with spatial indexes or 
postgis tables with spatial and attribute indexes if necessary.  Preprocessing 
the data (simplification / prefiltering / point thinning cluster) and using 
SCALETOKENs to reference different source layers at different scales can lead 
to big performance boosts at the mapserver end.

For us the biggest limiting factor has been the write blocking to sqlite cache 
when seeding tiles, if you increase the number of threads / process in mapcache 
seed those process will just end up waiting to write to the cache.  You can 
generally figure out what the max number of processes the cache can handle by 
running a seed on a test area and look at the number of tiles seeded per 
second, as some point -n will have no effect which your process are just 
waiting to write the cache.  I would look at using this method to test sqlite 
vs disk or other backends.  The GeoTiff cache may be promising as it appears to 
be non-blocking but experimental...

We use a riak cache with a riak leveldb backend which works well for us but is 
a bit of a pain to manage as it difficult to delete objects so it's not great 
for caches that need to be refreshed.

Finally you may want to test MapProxy as an alternative to MapCache (we use 
both).  MapProxy supports a non-blocking compact cache that could solve your 
inode problem

Regards









On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini 
mailto:sebastiano.la...@buchanancomputing.co.uk>>
 wrote:
We don’t create the data, we rely on OS (ordnance survey) to supply us the maps 
files and then we publish them in our service but I assume that probably we 
will need to create some flow to improve it or download it in other format that 
is faster.

Sebastiano Laini
Web Developer
Buchanan Computing

From: Fawcett, David (MNIT) 
[mailto:david.fawc...@state.mn.us<mailto:david.fawc...@state.mn.us>]
Sent: 08 November 2019 16:41
To: Sebastiano Laini 
mailto:sebastiano.la...@buchanancomputing.co.uk>>;
 'Rahkonen Jukka (MML)' 
mailto:jukka.rahko...@maanmittauslaitos.fi>>;
 'mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org>' 
mailto:mapserver-users@lists.osgeo.org>>
Subject: RE: mapcache seed speed optimization

For data formats that are slower to read, would it add fit your workflow to 
convert it to different data format before creating the tiles?
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed speed optimization

[Stephen Woodbridge]

FYI, imaptools.com is not online any more (except the we are moving pages).

Lars if you email me the files, I'll push them to my github repository 
so others can find them.


Thanks,
  -Steve W

On 11/11/2019 3:07 PM, lars.schylb...@blixtmail.se wrote:

Hi

I have had a similar situation as You with large national data sets in 
shape format. One of the biggest improvements for me has been to split 
large shape files into smaller and

then use vector tile indexes in Mapserver.
The tool shp2tile has been working well for me. Stephen Woodbridge has 
written that tool. It can be found at 
http://imaptools.com/download-software.html

I think that You had to compile yourself on linux, but that was easy.

I have an example where I have automated this in a script that runs 
through all shape file in a directory and splits shape files larger 
than a certain size and then creates the vector tile index files.

The code snippet that is part of larger import script can be found here:
https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359

The script snippet also shows the normal optimizations we do.

Hope this could give You some ideas how optimize and make Mapserver 
and Mapcache run faster. I also ran into the problem with to many inodes.

That got solved by using sqlite as backend in Mapcache.

Best regards / Lars Schylberg



11 november 2019 kl. 15:42, "Travis Kirstine" 
> 
skrev:


I may not be unrealistic for it take mapserver to 4-6 seconds to
generate 4096px image depending on the amount data being rendered
and the complexity of your styles / expressions / labels. If your
mapserver is able to generate a 1024x1024 in under a second that
is pretty good. Like Jukka and other suggest I would look at
translating the files to shp files with spatial indexes or postgis
tables with spatial and attribute indexes if necessary.
Preprocessing the data (simplification / prefiltering / point
thinning cluster) and using SCALETOKENs to reference different
source layers at different scales can lead to big performance
boosts at the mapserver end.
For us the biggest limiting factor has been the write blocking to
sqlite cache when seeding tiles, if you increase the number of
threads / process in mapcache seed those process will just end up
waiting to write to the cache. You can generally figure out what
the max number of processes the cache can handle by running a seed
on a test area and look at the number of tiles seeded per second,
as some point -n will have no effect which your process are just
waiting to write the cache. I would look at using this method to
test sqlite vs disk or other backends. The GeoTiff cache may be
promising as it appears to be non-blocking but experimental...
We use a riak cache with a riak leveldb backend which works well
for us but is a bit of a pain to manage as it difficult to delete
objects so it's not great for caches that need to be refreshed.
Finally you may want to test MapProxy as an alternative to
MapCache (we use both). MapProxy supports a non-blocking compact
cache that could solve your inode problem
Regards
On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>> wrote:

We don’t create the data, we rely on OS (ordnance survey) to
supply us the maps files and then we publish them in our
service but I assume that probably we will need to create some
flow to improve it or download it in other format that is faster.

Sebastiano Laini

Web Developer

Buchanan Computing

*From:*Fawcett, David (MNIT) [mailto:david.fawc...@state.mn.us
]
*Sent:* 08 November 2019 16:41
*To:* Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'Rahkonen
Jukka (MML)' mailto:jukka.rahko...@maanmittauslaitos.fi>>;
'mapserver-users@lists.osgeo.org
'
mailto:mapserver-users@lists.osgeo.org>>
*Subject:* RE: mapcache seed speed optimization

For data formats that are slower to read, would it add fit
your workflow to convert it to different data format before
creating the tiles?

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/mapserver-users 





___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org

Re: [mapserver-users] mapcache seed speed optimization

[Jeff McKenna]

For Windows users I have been distributing shp2tile with MS4W 
(https://ms4w.com/), and in fact the last MS4W contains a rebuilt 
shp2tile (and an updated Shapelib), which as Lars said is still very useful.


-jeff



On 2019-11-11 4:07 PM, lars.schylb...@blixtmail.se wrote:

Hi

I have had a similar situation as You with large national data sets in 
shape format. One of the biggest improvements for me has been to split 
large shape files into smaller and

then use vector tile indexes in Mapserver.
The tool shp2tile has been working well for me. Stephen Woodbridge has 
written that tool. It can be found at 
http://imaptools.com/download-software.html 


I think that You had to compile yourself on linux, but that was easy.

I have an example where I have automated this in a script that runs 
through all shape file in a directory and splits shape files larger than 
a certain size and then creates the vector tile index files.

The code snippet that is part of larger import script can be found here:
https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359

The script snippet also shows the normal optimizations we do.

Hope this could give You some ideas how optimize and make Mapserver and 
Mapcache run faster. I also ran into the problem with to many inodes.

That got solved by using sqlite as backend in Mapcache.

Best regards / Lars Schylberg



11 november 2019 kl. 15:42, "Travis Kirstine" > 
skrev:


I may not be unrealistic for it take mapserver to 4-6 seconds to
generate 4096px image depending on the amount data being rendered
and the complexity of your styles / expressions / labels. If your
mapserver is able to generate a 1024x1024 in under a second that is
pretty good. Like Jukka and other suggest I would look at
translating the files to shp files with spatial indexes or postgis
tables with spatial and attribute indexes if necessary.
Preprocessing the data (simplification / prefiltering / point
thinning cluster) and using SCALETOKENs to reference different
source layers at different scales can lead to big performance boosts
at the mapserver end.
For us the biggest limiting factor has been the write blocking to
sqlite cache when seeding tiles, if you increase the number of
threads / process in mapcache seed those process will just end up
waiting to write to the cache. You can generally figure out what the
max number of processes the cache can handle by running a seed on a
test area and look at the number of tiles seeded per second, as some
point -n will have no effect which your process are just waiting to
write the cache. I would look at using this method to test sqlite vs
disk or other backends. The GeoTiff cache may be promising as it
appears to be non-blocking but experimental...
We use a riak cache with a riak leveldb backend which works well for
us but is a bit of a pain to manage as it difficult to delete
objects so it's not great for caches that need to be refreshed.
Finally you may want to test MapProxy as an alternative to MapCache
(we use both). MapProxy supports a non-blocking compact cache that
could solve your inode problem
Regards
On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini
mailto:sebastiano.la...@buchanancomputing.co.uk>> wrote:

We don’t create the data, we rely on OS (ordnance survey) to
supply us the maps files and then we publish them in our service
but I assume that probably we will need to create some flow to
improve it or download it in other format that is faster.

Sebastiano Laini

Web Developer

Buchanan Computing

*From:*Fawcett, David (MNIT) [mailto:david.fawc...@state.mn.us
]
*Sent:* 08 November 2019 16:41
*To:* Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk>>; 'Rahkonen
Jukka (MML)' mailto:jukka.rahko...@maanmittauslaitos.fi>>;
'mapserver-users@lists.osgeo.org
'
mailto:mapserver-users@lists.osgeo.org>>
*Subject:* RE: mapcache seed speed optimization

For data formats that are slower to read, would it add fit your
workflow to convert it to different data format before creating
the tiles?

   

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed speed optimization

[lars . schylberg]

Hi

I have had a similar situation as You with large national data sets in shape 
format. One of the biggest improvements for me has been to split large shape 
files into smaller and
then use vector tile indexes in Mapserver.
The tool shp2tile has been working well for me. Stephen Woodbridge has written 
that tool. It can be found at http://imaptools.com/download-software.html  
(http://imaptools.com/download-software.html)
I think that You had to compile yourself on linux, but that was easy.

I have an example where I have automated this in a script that runs through all 
shape file in a directory and splits shape files larger than a certain size and 
then creates the vector tile index files.
The code snippet that is part of larger import script can be found here:
https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359 
(https://gist.github.com/LarsSchy/644b37ab2b1e58b48808bfb2b57d0359)

The script snippet also shows the normal optimizations we do.

Hope this could give You some ideas how optimize and make Mapserver and 
Mapcache run faster. I also ran into the problem with to many inodes. 
That got solved by using sqlite as backend in Mapcache.

Best regards / Lars Schylberg
11 november 2019 kl. 15:42, "Travis Kirstine" mailto:traviskirst...@gmail.com?to=%22Travis%20Kirstine%22%20)>
 skrev:
I may not be unrealistic for it take mapserver to 4-6 seconds to generate 
4096px image depending on the amount data being rendered and the complexity of 
your styles / expressions / labels. If your mapserver is able to generate a 
1024x1024 in under a second that is pretty good. Like Jukka and other suggest I 
would look at translating the files to shp files with spatial indexes or 
postgis tables with spatial and attribute indexes if necessary. Preprocessing 
the data (simplification / prefiltering / point thinning cluster) and using 
SCALETOKENs to reference different source layers at different scales can lead 
to big performance boosts at the mapserver end. 
For us the biggest limiting factor has been the write blocking to sqlite cache 
when seeding tiles, if you increase the number of threads / process in mapcache 
seed those process will just end up waiting to write to the cache. You can 
generally figure out what the max number of processes the cache can handle by 
running a seed on a test area and look at the number of tiles seeded per 
second, as some point -n will have no effect which your process are just 
waiting to write the cache. I would look at using this method to test sqlite vs 
disk or other backends. The GeoTiff cache may be promising as it appears to be 
non-blocking but experimental... 
We use a riak cache with a riak leveldb backend which works well for us but is 
a bit of a pain to manage as it difficult to delete objects so it's not great 
for caches that need to be refreshed. 
Finally you may want to test MapProxy as an alternative to MapCache (we use 
both). MapProxy supports a non-blocking compact cache that could solve your 
inode problem 
Regards 
 On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini 
mailto:sebastiano.la...@buchanancomputing.co.uk)> wrote:  
We don’t create the data, we rely on OS (ordnance survey) to supply us 
the maps files and then we publish them in our service but I assume that 
probably we will need to create some flow to improve it or download it in other 
format that is faster. 

Sebastiano Laini 

Web Developer 

Buchanan Computing 
From: Fawcett, David (MNIT) [mailto:david.fawc...@state.mn.us 
(mailto:david.fawc...@state.mn.us)]
Sent: 08 November 2019 16:41
To: Sebastiano Laini mailto:sebastiano.la...@buchanancomputing.co.uk)>; 'Rahkonen Jukka (MML)' 
mailto:jukka.rahko...@maanmittauslaitos.fi)>; 'mapserver-users@lists.osgeo.org 
(mailto:mapserver-users@lists.osgeo.org)' mailto:mapserver-users@lists.osgeo.org)>
Subject: RE: mapcache seed speed optimization 

For data formats that are slower to read, would it add fit your 
workflow to convert it to different data format before creating the tiles?  
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org (mailto:mapserver-users@lists.osgeo.org)
https://lists.osgeo.org/mailman/listinfo/mapserver-users 
(https://lists.osgeo.org/mailman/listinfo/mapserver-users)
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed speed optimization

[Travis Kirstine]

I may not be unrealistic for it take mapserver to 4-6 seconds to generate
4096px image depending on the amount data being rendered and the complexity
of your styles / expressions / labels.  If your mapserver is able to
generate a 1024x1024 in under a second that is pretty good.  Like Jukka and
other suggest I would look at translating the files to shp files with
spatial indexes or postgis tables with spatial and attribute indexes if
necessary.  Preprocessing the data (simplification / prefiltering / point
thinning cluster) and using SCALETOKENs to reference different source
layers at different scales can lead to big performance boosts at the
mapserver end.

For us the biggest limiting factor has been the write blocking to sqlite
cache when seeding tiles, if you increase the number of threads / process
in mapcache seed those process will just end up waiting to write to the
cache.  You can generally figure out what the max number of processes the
cache can handle by running a seed on a test area and look at the number of
tiles seeded per second, as some point -n will have no effect which your
process are just waiting to write the cache.  I would look at using this
method to test sqlite vs disk or other backends.  The GeoTiff cache may be
promising as it appears to be non-blocking but experimental...

We use a riak cache with a riak leveldb backend which works well for us but
is a bit of a pain to manage as it difficult to delete objects so it's not
great for caches that need to be refreshed.

Finally you may want to test MapProxy as an alternative to MapCache (we use
both).  MapProxy supports a non-blocking compact cache that could solve
your inode problem

Regards









On Fri, 8 Nov 2019 at 11:48, Sebastiano Laini <
sebastiano.la...@buchanancomputing.co.uk> wrote:

> We don’t create the data, we rely on OS (ordnance survey) to supply us the
> maps files and then we publish them in our service but I assume that
> probably we will need to create some flow to improve it or download it in
> other format that is faster.
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
>
>
>
> *From:* Fawcett, David (MNIT) [mailto:david.fawc...@state.mn.us]
> *Sent:* 08 November 2019 16:41
> *To:* Sebastiano Laini ;
> 'Rahkonen Jukka (MML)' ; '
> mapserver-users@lists.osgeo.org' 
> *Subject:* RE: mapcache seed speed optimization
>
>
>
> For data formats that are slower to read, would it add fit your workflow
> to convert it to different data format before creating the tiles?
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed speed optimization

[Sebastiano Laini]

We don't create the data, we rely on OS (ordnance survey) to supply us the maps 
files and then we publish them in our service but I assume that probably we 
will need to create some flow to improve it or download it in other format that 
is faster.

Sebastiano Laini
Web Developer
Buchanan Computing

From: Fawcett, David (MNIT) [mailto:david.fawc...@state.mn.us]
Sent: 08 November 2019 16:41
To: Sebastiano Laini ; 'Rahkonen 
Jukka (MML)' ; 
'mapserver-users@lists.osgeo.org' 
Subject: RE: mapcache seed speed optimization

For data formats that are slower to read, would it add fit your workflow to 
convert it to different data format before creating the tiles?
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed speed optimization

[Sebastiano Laini]

 to seed from multiple threads.

When is comes to ticket https://github.com/mapserver/mapcache/issues/169 the 
latest comment is from a Mpaserver developer and their last words "I wouldn't 
mind helping to fix if there were interested parties..." feel rather polite to 
me.

-Jukka Rahkonen-




Lähettäjä: mapserver-users 
mailto:mapserver-users-boun...@lists.osgeo.org>>
 Puolesta Sebastiano Laini
Lähetetty: perjantai 8. marraskuuta 2019 14.35
Vastaanottaja: 'mapserver-users@lists.osgeo.org' 
mailto:mapserver-users@lists.osgeo.org>>
Aihe: [mapserver-users] mapcache seed speed optimization

Hi all,

I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache 
due to the input source of our maps, so I'm trying to improve the speed of the 
mapserver/mapcache stack and what I cannot improve is the seed speed.

The -n and -p parameters seems to be useless.

-n | -nthreads: number of parallel threads that should be used to request tiles 
from the WMS source. The default is 1, but can be set higher if the WMS server 
can withstand parallel requests. (As a rule of thumb, the value chosen here 
should never be much higher than the number of CPUs on the WMS server.)

-p | -nprocesses: number of parallel processes that should be used to request 
tiles from the WMS source.
I've tried to use -n 2 and -p 6 but for some reason the CPU load is max 
100% in 1 core, no way to make it use more power and speed up the seed process.
Max memory used is 470MB between Centos 7 (with all the services running) and 
the seed process.

I've setup apache to use HTT/2 and it make quite a difference when the cache is 
already seeded but during the seed process doesn't seems to be affected.

My MapServer build is:

./configure \
--with-ogr=/usr/local/bin/gdal-config \
--with-gdal=/usr/local/bin/gdal-config \
--with-wfsclient \
--with-wmsclient \
--with-wfs \
--enable-debug \
--with-curl-config=/usr/bin/curl-config \
--with-proj \
--with-jpeg \
--with-freetype \
--with-postgis=/usr/pgsql-9.3/bin/pg_config \
--with-geos=/usr/local/bin/geos-config

I know there is mapserver 7 and yet I'm using 6 for the test.

While the MapCache is:

cmake   -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \
-DWITH_SQLITE=1 \
-DWITH_BERKELEY_DB=0 \
-DWITH_TIFF=0 \
-DWITH_GEOTIFF=0 \
-DWITH_FCGI=0 \
-DWITH_PCRE=0 \
-DWITH_PIXMAN=1 \
-DWITH_OGR=1 \
-DWITH_GEOS=1 \
 ../

MapCache is the latest version and this is my mapcache.xml file, we use sqlite 
as a cache method

  
  
/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3
  1
  1
  1573741823
  

  
bcmaps
cache_sqlite
bcgrid
PNG
16 16
0
2628000
  

  
   
   BC custom grid
   
   EPSG:27700
   256 256
   0 0 70 125
   70 28 14 7 2.8 1.4 0.7
  

  
   best
  

  
assemble
bilinear
mypng
4096
  

I've also tried to use -DWITH_MAPSERVER but seems that the integration is not 
working and they doen't care? https://github.com/mapserver/mapcache/issues/169 
this issue was open 2 years ago and still open, not even a reply from the 
developers.

How can I improve the seed process?
Should I increase the maxsize and metatile to requests more tiles and cache 
faster?

Regards,

Sebastiano Laini
Web Developer
Buchanan Computing

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed speed optimization

[Rahkonen Jukka (MML)]

Hi,

You can see examples about how to debug and improve the rendering speed with 
Mapserver in the presentation of Lars Schylberg 
https://media.ccc.de/v/bucharest-271-mapserver-optimize-for-performance.

Your maps look pretty and I hope you will find some bottleneck layers so you 
can make them faster.

-Jukka-

Lähettäjä: Sebastiano Laini 
Lähetetty: perjantai 8. marraskuuta 2019 16.30
Vastaanottaja: 'mapserver-users@lists.osgeo.org' 

Kopio: Rahkonen Jukka (MML) ; 'Yves 
Jacolin' 
Aihe: RE: mapcache seed speed optimization

My MapServer is on the same server as MapCache and my network is pretty fast.

I was just trying to experiment with the -n and -p value and checking the CPU 
load and I didn't seen any difference, but now that I read the responses I'm 
trying -n 6 and the CPU and the RAM is more in use now and seems to seed faster 
than before. I reach that numbers because the last time I was seeding with -n 
10 was incredibly slow so I started to increase the numbers.

Now in HTOP I can actually see 6 mapcache processes and not just 2

Sadly to seed the whole UK in disk cache we run out of inodes, I would like to 
use SSD.

About our data we use a mix of everything, here the final result 
https://www.buchananmapping.co.uk/

We use raster, vector and tab files; the reason I can't move out from mapserver 
are mainly the mapinfo tables (tab), possibly those tab files are the slowest 
layers
Even though some of the vector layers are very huge, some of them the data is 
about 21GB.

I'll try to debug layer by layer through MapServer and strip down to the 
minimum or replace some layers.

Lastly, the last time I've tried to seed the whole UK it took me more than a 
month and I simply stop it because was unacceptable to seed 7 levels of 
resolutions 70, 28, 14, 7, 2.8, 1.4, 0.7 that correspond to these scales 2500, 
5000, 1, 25000, 5, 10, 25, 50

Sebastiano Laini
Web Developer

From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi]
Sent: 08 November 2019 14:02
To: Sebastiano Laini 
mailto:sebastiano.la...@buchanancomputing.co.uk>>;
 'mapserver-users@lists.osgeo.org' 
mailto:mapserver-users@lists.osgeo.org>>
Subject: Re: mapcache seed speed optimization

Hi,

When you seed tiles from WMS the slowest component is usually the WMS server. 
And what the WMS server is spending its time is usually rendering. And what 
makes rendering slow is usually that you render too much, and sometimes that 
WMS server gets the vector data too slow because the backend data source is 
slow, very often because it has poor indexes which do not for suit the queries 
that WMS server is
sending.

If you have a slow WMS you can't get more speed out of it by bombing it with 
very many parallel threads. I would have a try if using 2-4, perhaps 8 threads 
makes seeding faster but if there is no difference I would concentrate on the 
WMS configuration. BTW. why did you
select to use 2 threads?

I am not sure why you see just 1 core running on 100% load. I guess that you 
are now running WMS and MapCache on the same computer. Mapserver program is 
using only 1 core but http server should start more processes for parallel 
requests. That CPU load is at 100% is generally a good thing because it tells 
that CPU is working, instead of waiting data from a slow disk. Another thing is 
that CPU may be doing something heavy and stupid.

Myself I would start by gathering  some information about how Mapserver WMS is 
working be setting "DEBUG 2" on each layer or for the whole mapfile. By looking 
at the rendering times of each layer the slowest layers can be found and then 
it is time to think about how to make those layers faster.

You wrote that you use SQLite cache so it would be good to know that SQLite 
database does not support parallel writes https://sqlite.org/lockingv3.html. 
Seeding with a few parallel threads may increase the speed if the WMS is the 
slowest part in your chain but when you have saturated the SQLite writing speed 
adding threads will do no good. If you have optimized your WMS so well that 
SQLite sets the limits then you should test other cache alternatives. I guess 
that file cache would be faster to seed from multiple threads.

When is comes to ticket https://github.com/mapserver/mapcache/issues/169 the 
latest comment is from a Mpaserver developer and their last words "I wouldn't 
mind helping to fix if there were interested parties..." feel rather polite to 
me.

-Jukka Rahkonen-




Lähettäjä: mapserver-users 
mailto:mapserver-users-boun...@lists.osgeo.org>>
 Puolesta Sebastiano Laini
Lähetetty: perjantai 8. marraskuuta 2019 14.35
Vastaanottaja: 'mapserver-users@lists.osgeo.org' 
mailto:mapserver-users@lists.osgeo.org>>
Aihe: [mapserver-users] mapcache seed speed optimization

Hi all,

I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache 
due to the input source of our maps, so I'm trying to improv

Re: [mapserver-users] mapcache seed speed optimization

[Sebastiano Laini]

My MapServer is on the same server as MapCache and my network is pretty fast.

I was just trying to experiment with the -n and -p value and checking the CPU 
load and I didn't seen any difference, but now that I read the responses I'm 
trying -n 6 and the CPU and the RAM is more in use now and seems to seed faster 
than before. I reach that numbers because the last time I was seeding with -n 
10 was incredibly slow so I started to increase the numbers.

Now in HTOP I can actually see 6 mapcache processes and not just 2

Sadly to seed the whole UK in disk cache we run out of inodes, I would like to 
use SSD.

About our data we use a mix of everything, here the final result 
https://www.buchananmapping.co.uk/

We use raster, vector and tab files; the reason I can't move out from mapserver 
are mainly the mapinfo tables (tab), possibly those tab files are the slowest 
layers
Even though some of the vector layers are very huge, some of them the data is 
about 21GB.

I'll try to debug layer by layer through MapServer and strip down to the 
minimum or replace some layers.

Lastly, the last time I've tried to seed the whole UK it took me more than a 
month and I simply stop it because was unacceptable to seed 7 levels of 
resolutions 70, 28, 14, 7, 2.8, 1.4, 0.7 that correspond to these scales 2500, 
5000, 1, 25000, 5, 10, 25, 50

Sebastiano Laini
Web Developer

From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi]
Sent: 08 November 2019 14:02
To: Sebastiano Laini ; 
'mapserver-users@lists.osgeo.org' 
Subject: Re: mapcache seed speed optimization

Hi,

When you seed tiles from WMS the slowest component is usually the WMS server. 
And what the WMS server is spending its time is usually rendering. And what 
makes rendering slow is usually that you render too much, and sometimes that 
WMS server gets the vector data too slow because the backend data source is 
slow, very often because it has poor indexes which do not for suit the queries 
that WMS server is
sending.

If you have a slow WMS you can't get more speed out of it by bombing it with 
very many parallel threads. I would have a try if using 2-4, perhaps 8 threads 
makes seeding faster but if there is no difference I would concentrate on the 
WMS configuration. BTW. why did you
select to use 2 threads?

I am not sure why you see just 1 core running on 100% load. I guess that you 
are now running WMS and MapCache on the same computer. Mapserver program is 
using only 1 core but http server should start more processes for parallel 
requests. That CPU load is at 100% is generally a good thing because it tells 
that CPU is working, instead of waiting data from a slow disk. Another thing is 
that CPU may be doing something heavy and stupid.

Myself I would start by gathering  some information about how Mapserver WMS is 
working be setting "DEBUG 2" on each layer or for the whole mapfile. By looking 
at the rendering times of each layer the slowest layers can be found and then 
it is time to think about how to make those layers faster.

You wrote that you use SQLite cache so it would be good to know that SQLite 
database does not support parallel writes https://sqlite.org/lockingv3.html. 
Seeding with a few parallel threads may increase the speed if the WMS is the 
slowest part in your chain but when you have saturated the SQLite writing speed 
adding threads will do no good. If you have optimized your WMS so well that 
SQLite sets the limits then you should test other cache alternatives. I guess 
that file cache would be faster to seed from multiple threads.

When is comes to ticket https://github.com/mapserver/mapcache/issues/169 the 
latest comment is from a Mpaserver developer and their last words "I wouldn't 
mind helping to fix if there were interested parties..." feel rather polite to 
me.

-Jukka Rahkonen-




Lähettäjä: mapserver-users 
mailto:mapserver-users-boun...@lists.osgeo.org>>
 Puolesta Sebastiano Laini
Lähetetty: perjantai 8. marraskuuta 2019 14.35
Vastaanottaja: 'mapserver-users@lists.osgeo.org' 
mailto:mapserver-users@lists.osgeo.org>>
Aihe: [mapserver-users] mapcache seed speed optimization

Hi all,

I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache 
due to the input source of our maps, so I'm trying to improve the speed of the 
mapserver/mapcache stack and what I cannot improve is the seed speed.

The -n and -p parameters seems to be useless.

-n | -nthreads: number of parallel threads that should be used to request tiles 
from the WMS source. The default is 1, but can be set higher if the WMS server 
can withstand parallel requests. (As a rule of thumb, the value chosen here 
should never be much higher than the number of CPUs on the WMS server.)

-p | -nprocesses: number of parallel processes that should be used to request 
tiles from the WMS source.
I've tried to use -n 2 and -p 6 but for some reason the CPU load is max 
100% 

Re: [mapserver-users] mapcache seed speed optimization

[Rahkonen Jukka (MML)]

Hi,

When you seed tiles from WMS the slowest component is usually the WMS server. 
And what the WMS server is spending its time is usually rendering. And what 
makes rendering slow is usually that you render too much, and sometimes that 
WMS server gets the vector data too slow because the backend data source is 
slow, very often because it has poor indexes which do not for suit the queries 
that WMS server is
sending.

If you have a slow WMS you can't get more speed out of it by bombing it with 
very many parallel threads. I would have a try if using 2-4, perhaps 8 threads 
makes seeding faster but if there is no difference I would concentrate on the 
WMS configuration. BTW. why did you
select to use 2 threads?

I am not sure why you see just 1 core running on 100% load. I guess that you 
are now running WMS and MapCache on the same computer. Mapserver program is 
using only 1 core but http server should start more processes for parallel 
requests. That CPU load is at 100% is generally a good thing because it tells 
that CPU is working, instead of waiting data from a slow disk. Another thing is 
that CPU may be doing something heavy and stupid.

Myself I would start by gathering  some information about how Mapserver WMS is 
working be setting "DEBUG 2" on each layer or for the whole mapfile. By looking 
at the rendering times of each layer the slowest layers can be found and then 
it is time to think about how to make those layers faster.

You wrote that you use SQLite cache so it would be good to know that SQLite 
database does not support parallel writes https://sqlite.org/lockingv3.html. 
Seeding with a few parallel threads may increase the speed if the WMS is the 
slowest part in your chain but when you have saturated the SQLite writing speed 
adding threads will do no good. If you have optimized your WMS so well that 
SQLite sets the limits then you should test other cache alternatives. I guess 
that file cache would be faster to seed from multiple threads.

When is comes to ticket https://github.com/mapserver/mapcache/issues/169 the 
latest comment is from a Mpaserver developer and their last words "I wouldn't 
mind helping to fix if there were interested parties..." feel rather polite to 
me.

-Jukka Rahkonen-




Lähettäjä: mapserver-users  Puolesta 
Sebastiano Laini
Lähetetty: perjantai 8. marraskuuta 2019 14.35
Vastaanottaja: 'mapserver-users@lists.osgeo.org' 

Aihe: [mapserver-users] mapcache seed speed optimization

Hi all,

I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache 
due to the input source of our maps, so I'm trying to improve the speed of the 
mapserver/mapcache stack and what I cannot improve is the seed speed.

The -n and -p parameters seems to be useless.

-n | -nthreads: number of parallel threads that should be used to request tiles 
from the WMS source. The default is 1, but can be set higher if the WMS server 
can withstand parallel requests. (As a rule of thumb, the value chosen here 
should never be much higher than the number of CPUs on the WMS server.)

-p | -nprocesses: number of parallel processes that should be used to request 
tiles from the WMS source.
I've tried to use -n 2 and -p 6 but for some reason the CPU load is max 
100% in 1 core, no way to make it use more power and speed up the seed process.
Max memory used is 470MB between Centos 7 (with all the services running) and 
the seed process.

I've setup apache to use HTT/2 and it make quite a difference when the cache is 
already seeded but during the seed process doesn't seems to be affected.

My MapServer build is:

./configure \
--with-ogr=/usr/local/bin/gdal-config \
--with-gdal=/usr/local/bin/gdal-config \
--with-wfsclient \
--with-wmsclient \
--with-wfs \
--enable-debug \
--with-curl-config=/usr/bin/curl-config \
--with-proj \
--with-jpeg \
--with-freetype \
--with-postgis=/usr/pgsql-9.3/bin/pg_config \
--with-geos=/usr/local/bin/geos-config

I know there is mapserver 7 and yet I'm using 6 for the test.

While the MapCache is:

cmake   -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \
-DWITH_SQLITE=1 \
-DWITH_BERKELEY_DB=0 \
-DWITH_TIFF=0 \
-DWITH_GEOTIFF=0 \
-DWITH_FCGI=0 \
-DWITH_PCRE=0 \
-DWITH_PIXMAN=1 \
-DWITH_OGR=1 \
-DWITH_GEOS=1 \
 ../

MapCache is the latest version and this is my mapcache.xml file, we use sqlite 
as a cache method

  
  
/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3
  1
  1
  1573741823
  

  
bcmaps
cache_sqlite
bcgrid
PNG
16 16
0
2628000
  

  
   
   BC custom grid
   
   EPSG:27700
   256 256
   0 0 70 125
   70 28 14 7 2.8 1.4 0.7
  

  
   best
  

  
assemble
bilinear
mypng
4096
  

I've also tried to use -DWITH_MAPSERVER but seems that the integration is not

Re: [mapserver-users] mapcache seed speed optimization

[Yves Jacolin]

Sebastiano,

Some though in my mind:
* –n 2 : do you really have 2'000 CPU on your WMS server? if not, it
could be worst than better, flooding your WMS server.
* what is the bottleneck?  If WMS server is slow you can't make MapCache
faster, you should optimise WMS Server. Multi-domain can help, data
improvement, etc. also.
* Does your WMS is on the same server? same network? Does you network fast
enough?
* metatile can help definitively!
* "we use sqlite as a cache method" => I don't know if sqlite is really
fast or not, but this could be another bottleneck :)

What do you mean exactly by improve seed speed? :) This is matter of
personal point of view some time.

Y.

Le ven. 8 nov. 2019 à 13:43, Sebastiano Laini <
sebastiano.la...@buchanancomputing.co.uk> a écrit :

> Hi all,
>
>
>
> I’m evaluating new stacks but seems that I’m stuck with mapserver and
> mapcache due to the input source of our maps, so I’m trying to improve the
> speed of the mapserver/mapcache stack and what I cannot improve is the seed
> speed.
>
>
>
> The –n and –p parameters seems to be useless.
>
> *-n | –nthreads*: number of parallel threads that should be used to
> request tiles from the WMS source. The default is 1, but can be set higher
> if the WMS server can withstand parallel requests. (As a rule of thumb, the
> value chosen here should never be much higher than the number of CPUs on
> the WMS server.)
>
> *-p | –nprocesses*: number of parallel processes that should be used to
> request tiles from the WMS source.
>
> I’ve tried to use –n 2 and –p 6 but for some reason the CPU load
> is max 100% in 1 core, no way to make it use more power and speed up the
> seed process.
>
> Max memory used is 470MB between Centos 7 (with all the services running)
> and the seed process.
>
>
>
> I’ve setup apache to use HTT/2 and it make quite a difference when the
> cache is already seeded but during the seed process doesn’t seems to be
> affected.
>
>
>
> My MapServer build is:
>
>
>
> ./configure \
>
> --with-ogr=/usr/local/bin/gdal-config \
>
> --with-gdal=/usr/local/bin/gdal-config \
>
> --with-wfsclient \
>
> --with-wmsclient \
>
> --with-wfs \
>
> --enable-debug \
>
> --with-curl-config=/usr/bin/curl-config \
>
> --with-proj \
>
> --with-jpeg \
>
> --with-freetype \
>
> --with-postgis=/usr/pgsql-9.3/bin/pg_config \
>
> --with-geos=/usr/local/bin/geos-config
>
>
>
> I know there is mapserver 7 and yet I’m using 6 for the test.
>
>
>
> While the MapCache is:
>
>
>
> cmake   -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \
>
> -DWITH_SQLITE=1 \
>
> -DWITH_BERKELEY_DB=0 \
>
> -DWITH_TIFF=0 \
>
> -DWITH_GEOTIFF=0 \
>
> -DWITH_FCGI=0 \
>
> -DWITH_PCRE=0 \
>
> -DWITH_PIXMAN=1 \
>
> -DWITH_OGR=1 \
>
> -DWITH_GEOS=1 \
>
>  ../
>
>
>
> MapCache is the latest version and this is my mapcache.xml file, we use
> sqlite as a cache method
>
>
>
>   
>
>
> /home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3
>
>   1
>
>   1
>
>   1573741823
>
>   
>
>
>
>   
>
> bcmaps
>
> cache_sqlite
>
> bcgrid
>
> PNG
>
> 16 16
>
> 0
>
> 2628000
>
>   
>
>
>
>   
>
>
>
>BC custom grid
>
>
>
>EPSG:27700
>
>256 256
>
>0 0 70 125
>
>70 28 14 7 2.8 1.4 0.7
>
>   
>
>
>
>   
>
>best
>
>   
>
>
>
>   
>
> assemble
>
> bilinear
>
> mypng
>
> 4096
>
>   
>
>
>
> I’ve also tried to use –DWITH_MAPSERVER but seems that the integration is
> not working and they doen’t care?
> https://github.com/mapserver/mapcache/issues/169 this issue was open 2
> years ago and still open, not even a reply from the developers.
>
>
>
> How can I improve the seed process?
>
> Should I increase the maxsize and metatile to requests more tiles and
> cache faster?
>
>
>
> Regards,
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
>
>
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users



-- 
Yves Jacolin
Training and support manager - Team Manager
Camptocamp

Tel (France) : +33 4 58 48 20 43
Tel (Switzerland) : +41 21 619 10 43
Mob. : +33 6 18 75 42 21

email : yves.jaco...@camptocamp.com
http://www.camptocamp.com
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] mapcache seed speed optimization

[Sebastiano Laini]

Hi all,

I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache 
due to the input source of our maps, so I'm trying to improve the speed of the 
mapserver/mapcache stack and what I cannot improve is the seed speed.

The -n and -p parameters seems to be useless.

-n | -nthreads: number of parallel threads that should be used to request tiles 
from the WMS source. The default is 1, but can be set higher if the WMS server 
can withstand parallel requests. (As a rule of thumb, the value chosen here 
should never be much higher than the number of CPUs on the WMS server.)

-p | -nprocesses: number of parallel processes that should be used to request 
tiles from the WMS source.
I've tried to use -n 2 and -p 6 but for some reason the CPU load is max 
100% in 1 core, no way to make it use more power and speed up the seed process.
Max memory used is 470MB between Centos 7 (with all the services running) and 
the seed process.

I've setup apache to use HTT/2 and it make quite a difference when the cache is 
already seeded but during the seed process doesn't seems to be affected.

My MapServer build is:

./configure \
--with-ogr=/usr/local/bin/gdal-config \
--with-gdal=/usr/local/bin/gdal-config \
--with-wfsclient \
--with-wmsclient \
--with-wfs \
--enable-debug \
--with-curl-config=/usr/bin/curl-config \
--with-proj \
--with-jpeg \
--with-freetype \
--with-postgis=/usr/pgsql-9.3/bin/pg_config \
--with-geos=/usr/local/bin/geos-config

I know there is mapserver 7 and yet I'm using 6 for the test.

While the MapCache is:

cmake   -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \
-DWITH_SQLITE=1 \
-DWITH_BERKELEY_DB=0 \
-DWITH_TIFF=0 \
-DWITH_GEOTIFF=0 \
-DWITH_FCGI=0 \
-DWITH_PCRE=0 \
-DWITH_PIXMAN=1 \
-DWITH_OGR=1 \
-DWITH_GEOS=1 \
 ../

MapCache is the latest version and this is my mapcache.xml file, we use sqlite 
as a cache method

  
  
/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3
  1
  1
  1573741823
  

  
bcmaps
cache_sqlite
bcgrid
PNG
16 16
0
2628000
  

  
   
   BC custom grid
   
   EPSG:27700
   256 256
   0 0 70 125
   70 28 14 7 2.8 1.4 0.7
  

  
   best
  

  
assemble
bilinear
mypng
4096
  

I've also tried to use -DWITH_MAPSERVER but seems that the integration is not 
working and they doen't care? https://github.com/mapserver/mapcache/issues/169 
this issue was open 2 years ago and still open, not even a reply from the 
developers.

How can I improve the seed process?
Should I increase the maxsize and metatile to requests more tiles and cache 
faster?

Regards,

Sebastiano Laini
Web Developer
Buchanan Computing

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache - Is it possible to have two tilesets share a single cache at particular zoom levels?

[Eichner, Andreas - SID]

Hmm, seems the Composite-Cache would be the better way to go: 
https://mapserver.org/mapcache/caches.html#composite-caches

In the past symbolic links (aka soft links) worked for me on Linux. But with 
that tiles might be seeded through multiple caches giving error "unknown error 
(another thread/process failed to create the tile I was waiting for)" in the 
logs.
So better try a composite cache.

> -Original Message-
> From: mapserver-users [mailto:mapserver-users-
> boun...@lists.osgeo.org] On Behalf Of Mark Volz
> Sent: Thursday, October 24, 2019 10:35 PM
> To: mapserver-users@lists.osgeo.org
> Subject: Re: [mapserver-users] Mapcache - Is it possible to have
> two tilesets share a single cache at particular zoom levels?
> 
> Andreas,
> 
> Good idea.  It looks like Windows can create symlinks.  I suppose
> I could try symlinking the cache for levels 5-11.  Do you know if
> I would need to create hard or soft links to do this?   Also, I
> would like to make sure our backup software does not try to backup
> the actual data twice...
> Reference:  https://www.howtogeek.com/howto/16226/complete-guide-
> to-symbolic-links-symlinks-on-windows-or-linux/
> 
> Thanks!
> 
> Sincerely,
> Mark Volz, GISP
> Lyon County GIS Coordinator
> 504 Fairgrounds Rd
> Marshall, MN 56258
> Ph:  (507) 532-8218
> http://lyonco.org/
> http://geomoose.lyonco.org/
> 
> 
> >What about symlinking the level directories?
> 
> 
> > -Original Message-
> > From: mapserver-users [mailto:mapserver-users-
> > boun...@lists.osgeo.org] On Behalf Of Mark Volz
> > Sent: Wednesday, October 23, 2019 4:12 PM
> > To: mapserver-users@lists.osgeo.org
> > Subject: [mapserver-users] Mapcache - Is it possible to have two
> > tilesets share a single cache at particular zoom levels?
> >
> > Hello,
> >
> >
> >
> > I have two similar tilesets configured in Mapcache.  One of the
> > tilesets displays a political map at small scales, then changes
> to an
> > air photo map as the user zooms into the map.  The second
> tileset
> > displays air photos at every zoom level.  It would be useful if
> these
> > tilesets could share a single cache for zoom levels 5-11.  This
> way we
> > could save a significant amount of space on the server.
> >
> >
> >
> > Example #1:
> >
> > Political and Air Photo Basemap:
> >
> > * Zoom level 0-4:  Use the cache from the "political map
> > cache"
> >
> > * Zoom level 5-11: Use the cache from the "air photo
> > cache"
> >
> > Air Photo Basemap
> >
> > * Zoom level 0-11:  Use the cache from the "air photo
> > cache"
> >
> > Result:
> >
> > * Any tiles at level 5-11, which will take up the
> majority
> > of the space on the sever will not need to be duplicated between
> the
> > two tilesets.
> >
> >
> >
> > Please let me know if it is possible to share caches between
> different
> > tilesets.
> >
> >
> >
> > Thanks!
> >
> >
> >
> > Sincerely,
> >
> > Mark Volz, GISP
> >
> > Lyon County GIS Coordinator
> >
> >
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache - Is it possible to have two tilesets share a single cache at particular zoom levels?

[Steve Lime]

You'd still need to configure the political map tileset not to generate
tiles in the symlinked folders but to serve tiles from those folders - I
don't think that's possible. IMHO it's better to configure your political
map tiles to only be available for a specific range of zoom levels and
advertise that explicitly.

On Thu, Oct 24, 2019 at 3:34 PM Mark Volz  wrote:

> Andreas,
>
> Good idea.  It looks like Windows can create symlinks.  I suppose I could
> try symlinking the cache for levels 5-11.  Do you know if I would need to
> create hard or soft links to do this?   Also, I would like to make sure our
> backup software does not try to backup the actual data twice...
> Reference:
> https://www.howtogeek.com/howto/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/
>
> Thanks!
>
> Sincerely,
> Mark Volz, GISP
> Lyon County GIS Coordinator
> 504 Fairgrounds Rd
> Marshall, MN 56258
> Ph:  (507) 532-8218
> http://lyonco.org/
> http://geomoose.lyonco.org/
>
>
> >What about symlinking the level directories?
>
>
> > -Original Message-
> > From: mapserver-users [mailto:mapserver-users-
> > boun...@lists.osgeo.org] On Behalf Of Mark Volz
> > Sent: Wednesday, October 23, 2019 4:12 PM
> > To: mapserver-users@lists.osgeo.org
> > Subject: [mapserver-users] Mapcache - Is it possible to have two
> > tilesets share a single cache at particular zoom levels?
> >
> > Hello,
> >
> >
> >
> > I have two similar tilesets configured in Mapcache.  One of the
> > tilesets displays a political map at small scales, then changes to an
> > air photo map as the user zooms into the map.  The second tileset
> > displays air photos at every zoom level.  It would be useful if these
> > tilesets could share a single cache for zoom levels 5-11.  This way we
> > could save a significant amount of space on the server.
> >
> >
> >
> > Example #1:
> >
> > Political and Air Photo Basemap:
> >
> > * Zoom level 0-4:  Use the cache from the "political map
> > cache"
> >
> > * Zoom level 5-11: Use the cache from the "air photo
> > cache"
> >
> > Air Photo Basemap
> >
> > * Zoom level 0-11:  Use the cache from the "air photo
> > cache"
> >
> > Result:
> >
> > * Any tiles at level 5-11, which will take up the majority
> > of the space on the sever will not need to be duplicated between the
> > two tilesets.
> >
> >
> >
> > Please let me know if it is possible to share caches between different
> > tilesets.
> >
> >
> >
> > Thanks!
> >
> >
> >
> > Sincerely,
> >
> > Mark Volz, GISP
> >
> > Lyon County GIS Coordinator
> >
> >
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache - Is it possible to have two tilesets share a single cache at particular zoom levels?

[Mark Volz]

Andreas,

Good idea.  It looks like Windows can create symlinks.  I suppose I could try 
symlinking the cache for levels 5-11.  Do you know if I would need to create 
hard or soft links to do this?   Also, I would like to make sure our backup 
software does not try to backup the actual data twice...
Reference:  
https://www.howtogeek.com/howto/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/

Thanks!

Sincerely,
Mark Volz, GISP
Lyon County GIS Coordinator
504 Fairgrounds Rd
Marshall, MN 56258
Ph:  (507) 532-8218
http://lyonco.org/
http://geomoose.lyonco.org/


>What about symlinking the level directories?


> -Original Message-
> From: mapserver-users [mailto:mapserver-users- 
> boun...@lists.osgeo.org] On Behalf Of Mark Volz
> Sent: Wednesday, October 23, 2019 4:12 PM
> To: mapserver-users@lists.osgeo.org
> Subject: [mapserver-users] Mapcache - Is it possible to have two 
> tilesets share a single cache at particular zoom levels?
>
> Hello,
>
>
>
> I have two similar tilesets configured in Mapcache.  One of the 
> tilesets displays a political map at small scales, then changes to an 
> air photo map as the user zooms into the map.  The second tileset 
> displays air photos at every zoom level.  It would be useful if these 
> tilesets could share a single cache for zoom levels 5-11.  This way we 
> could save a significant amount of space on the server.
>
>
>
> Example #1:
>
> Political and Air Photo Basemap:
>
> * Zoom level 0-4:  Use the cache from the "political map
> cache"
>
> * Zoom level 5-11: Use the cache from the "air photo
> cache"
>
> Air Photo Basemap
>
> * Zoom level 0-11:  Use the cache from the "air photo
> cache"
>
> Result:
>
> * Any tiles at level 5-11, which will take up the majority
> of the space on the sever will not need to be duplicated between the 
> two tilesets.
>
>
>
> Please let me know if it is possible to share caches between different 
> tilesets.
>
>
>
> Thanks!
>
>
>
> Sincerely,
>
> Mark Volz, GISP
>
> Lyon County GIS Coordinator
>
>
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache - Is it possible to have two tilesets share a single cache at particular zoom levels?

[Eichner, Andreas - SID]

What about symlinking the level directories?

> -Original Message-
> From: mapserver-users [mailto:mapserver-users-
> boun...@lists.osgeo.org] On Behalf Of Mark Volz
> Sent: Wednesday, October 23, 2019 4:12 PM
> To: mapserver-users@lists.osgeo.org
> Subject: [mapserver-users] Mapcache - Is it possible to have two
> tilesets share a single cache at particular zoom levels?
> 
> Hello,
> 
> 
> 
> I have two similar tilesets configured in Mapcache.  One of the
> tilesets displays a political map at small scales, then changes to
> an air photo map as the user zooms into the map.  The second
> tileset displays air photos at every zoom level.  It would be
> useful if these tilesets could share a single cache for zoom
> levels 5-11.  This way we could save a significant amount of space
> on the server.
> 
> 
> 
> Example #1:
> 
> Political and Air Photo Basemap:
> 
> * Zoom level 0-4:  Use the cache from the "political map
> cache"
> 
> * Zoom level 5-11: Use the cache from the "air photo
> cache"
> 
> Air Photo Basemap
> 
> * Zoom level 0-11:  Use the cache from the "air photo
> cache"
> 
> Result:
> 
> * Any tiles at level 5-11, which will take up the majority
> of the space on the sever will not need to be duplicated between
> the two tilesets.
> 
> 
> 
> Please let me know if it is possible to share caches between
> different tilesets.
> 
> 
> 
> Thanks!
> 
> 
> 
> Sincerely,
> 
> Mark Volz, GISP
> 
> Lyon County GIS Coordinator
> 
> 

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache - Is it possible to have two tilesets share a single cache at particular zoom levels?

[Mark Volz]

Steve,

Thank you for your reply.  I had also thought about configuring the caches at 
the client level as you suggest.  Unfortunately, I do not have control over all 
of the potential clients as some outside users might connect to my services via 
Leaflet, OpenLayers, ArcMap, QGIS, etc.  It sounds like my best option is to 
swap caches as you suggest (the political cache will only have 4 zoom levels), 
and hope outside users either repeat the same process or only use the air photo 
cache.

Thanks!
Sincerely,
Mark Volz

From: Steve Lime 
Sent: Wednesday, October 23, 2019 10:31 AM
To: Mark Volz 
Cc: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] Mapcache - Is it possible to have two tilesets 
share a single cache at particular zoom levels?

SWHHS/LYON/LINCOLN COUNTY SECURITY NOTICE:
This email originated from an external sender. Exercise caution before clicking 
on any links or attachments and consider whether you know the sender. For more 
information please contact IT support.

That seems more like a client configuration issue to me. That is, you'd have 
the two caches (air photos and political) but then configure the client to swap 
at different zoom levels. For example, with Leaflet you could simply set 
maxZoom/minZoom map attributes. Here's an example that uses a map (1 tileset) 
at small scales and then transitions to aerial photos w/overlays (2 tilesets) 
as the user zooms in:

   http://maps1.dnr.state.mn.us/test/tiles/switch.html

Presumably you can tell MapCache to only generate tiles over certain zoom 
ranges for each tileset - thus reducing disk space. Perhaps I'm miss 
interpreting your question though. --Steve

On Wed, Oct 23, 2019 at 9:29 AM Mark Volz 
mailto:markv...@co.lyon.mn.us>> wrote:
Hello,

I have two similar tilesets configured in Mapcache.  One of the tilesets 
displays a political map at small scales, then changes to an air photo map as 
the user zooms into the map.  The second tileset displays air photos at every 
zoom level.  It would be useful if these tilesets could share a single cache 
for zoom levels 5-11.  This way we could save a significant amount of space on 
the server.

Example #1:
Political and Air Photo Basemap:

• Zoom level 0-4:  Use the cache from the “political map cache”

• Zoom level 5-11: Use the cache from the “air photo cache”
Air Photo Basemap

• Zoom level 0-11:  Use the cache from the “air photo cache”
Result:

• Any tiles at level 5-11, which will take up the majority of the space 
on the sever will not need to be duplicated between the two tilesets.

Please let me know if it is possible to share caches between different tilesets.

Thanks!

Sincerely,
Mark Volz, GISP
Lyon County GIS Coordinator

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/mapserver-users<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2flists.osgeo.org%2fmailman%2flistinfo%2fmapserver-users=E,1,lECnGbIX0YoSzreux3ITULYLABHISgEy3M_AVOrLyIpIHAKjbzPFhtCfz0QD_UctHcbbpR6bY3qojgl7junn8uKhJwSvBg_KRa4We3r-VY8FeePCam7CGw,,=1>
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache - Is it possible to have two tilesets share a single cache at particular zoom levels?

[Steve Lime]

That seems more like a client configuration issue to me. That is, you'd
have the two caches (air photos and political) but then configure the
client to swap at different zoom levels. For example, with Leaflet you
could simply set maxZoom/minZoom map attributes. Here's an example that
uses a map (1 tileset) at small scales and then transitions to aerial
photos w/overlays (2 tilesets) as the user zooms in:

   http://maps1.dnr.state.mn.us/test/tiles/switch.html

Presumably you can tell MapCache to only generate tiles over certain zoom
ranges for each tileset - thus reducing disk space. Perhaps I'm miss
interpreting your question though. --Steve

On Wed, Oct 23, 2019 at 9:29 AM Mark Volz  wrote:

> Hello,
>
>
>
> I have two similar tilesets configured in Mapcache.  One of the tilesets
> displays a political map at small scales, then changes to an air photo map
> as the user zooms into the map.  The second tileset displays air photos at
> every zoom level.  It would be useful if these tilesets could share a
> single cache for zoom levels 5-11.  This way we could save a significant
> amount of space on the server.
>
>
>
> Example #1:
>
> Political and Air Photo Basemap:
>
> · Zoom level 0-4:  Use the cache from the “political map cache”
>
> · Zoom level 5-11: Use the cache from the “air photo cache”
>
> Air Photo Basemap
>
> · Zoom level 0-11:  Use the cache from the “air photo cache”
>
> Result:
>
> · Any tiles at level 5-11, which will take up the majority of the
> space on the sever will not need to be duplicated between the two tilesets.
>
>
>
> Please let me know if it is possible to share caches between different
> tilesets.
>
>
>
> Thanks!
>
>
>
> Sincerely,
>
> *Mark Volz, GISP*
>
> *Lyon County GIS Coordinator*
>
>
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache - Is it possible to have two tilesets share a single cache at particular zoom levels?

[Mark Volz]

Hello,

I have two similar tilesets configured in Mapcache.  One of the tilesets 
displays a political map at small scales, then changes to an air photo map as 
the user zooms into the map.  The second tileset displays air photos at every 
zoom level.  It would be useful if these tilesets could share a single cache 
for zoom levels 5-11.  This way we could save a significant amount of space on 
the server.

Example #1:
Political and Air Photo Basemap:

* Zoom level 0-4:  Use the cache from the "political map cache"

* Zoom level 5-11: Use the cache from the "air photo cache"
Air Photo Basemap

* Zoom level 0-11:  Use the cache from the "air photo cache"
Result:

* Any tiles at level 5-11, which will take up the majority of the space 
on the sever will not need to be duplicated between the two tilesets.

Please let me know if it is possible to share caches between different tilesets.

Thanks!

Sincerely,
Mark Volz, GISP
Lyon County GIS Coordinator

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache: restricted extent and ruleset to limit cache area seems ineffective

[Mark Volz]

Steve,
Mapcache is unexpectedly generating and returning a new air photo image outside 
of the visibility extent.  

Thank You!
Sincerely,
Mark Volz

--

Message: 1
Date: Fri, 6 Sep 2019 21:34:04 +
From: "Lime, Steve D (MNIT)" 
To: Mark Volz , Travis Kirstine

Cc: "mapserver-users@lists.osgeo.org"
    
Subject: Re: [mapserver-users] mapcache: restricted extent and ruleset
to limit cache area seems ineffective
Message-ID:



Content-Type: text/plain; charset="utf-8"

Is it returning a blank tile? If so that might be expected. I thought mapcache 
would return a blank tile for requests outside the grid extent.

From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] On 
Behalf Of Mark Volz
Sent: Friday, September 06, 2019 1:36 PM
To: Travis Kirstine 
Cc: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] mapcache: restricted extent and ruleset to limit 
cache area seems ineffective

This message may be from an external email source.
Do not select links or open attachments unless verified. Report all suspicious 
emails to Minnesota IT Services Security Operations Center.

Hello,

I am still having issues restricting the extent in my mapcache area using 
rulesets.  I created a rule for all of my zoom levels that has a limited 
visibility extent.  However, Mapcache is still returning imagery for the entire 
grid area even though I created a rule that should only display a limited 
extent.  I am still not sure why the visibility extent rule seems ineffective.

Thanks!
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache: restricted extent and ruleset to limit cache area seems ineffective

[TC Haddad]

I’ve had success setting a mask at the Mapserver level, and MapCache has
not problem only rendering imagery to the edge of whatever irregular shape
you need to use (e.g. watershed or county boundary). So maybe try that?

However It’s true, MapCache will still return blank tiles outside the mask
area.

Tanya

On Fri, Sep 6, 2019 at 1:36 PM Mark Volz  wrote:

> Hello,
>
>
>
> I am still having issues restricting the extent in my mapcache area using
> rulesets.  I created a rule for all of my zoom levels that has a limited
> visibility extent.  However, Mapcache is still returning imagery for the
> entire grid area even though I created a rule that should only display a
> limited extent.  I am still not sure why the visibility extent rule seems
> ineffective.
>
>
>
> Thanks!
>
>
> --
>
> You can set the
>
> empty_img
>
> To return a blank tile.  I believe the status is still 404 - I don't think 
> you can control this
>
>
>
> On Fri, 30 Aug 2019 at 15:44, Mark Volz  wrote:
>
> Hello,
>
>  I am in the process of limiting the cached area in mapcache.  I am having
> trouble with both the restricted_extent and ruleset parameters.  I first
> started by using restricted_extent such as “LyonCC”.  That parameter seemed to work.
> However mapcache returned 404 errors for any tiles outside of the
> restricted extent.  Next I tried to use rulesets instead.  The rulesets
> seem to have no effect, and new tiles were getting generated outside of the
> visibility extent.
>
>  Question 1:  When we use restricted extents on a grid should MapCache
> return a transparent tile, white tile, or a 404 error?  Can we control what
> it returns?
>
>  Question 2:  Do I have my ruleset configured appropriately?
>
>  
>
> LyonCC
>
> 
>
>  
>
>  levels?-->
>
> 
>
> 435567 98889 568247 259687
>
> 
>
> 
>
> 
>
>  Thank You!
>
>
>
> Sincerely,
>
> *Mark Volz, GISP*
>
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
> 
>
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache: restricted extent and ruleset to limit cache area seems ineffective

[Lime, Steve D (MNIT)]

Is it returning a blank tile? If so that might be expected. I thought mapcache 
would return a blank tile for requests outside the grid extent.

From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] On 
Behalf Of Mark Volz
Sent: Friday, September 06, 2019 1:36 PM
To: Travis Kirstine 
Cc: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] mapcache: restricted extent and ruleset to limit 
cache area seems ineffective

This message may be from an external email source.
Do not select links or open attachments unless verified. Report all suspicious 
emails to Minnesota IT Services Security Operations Center.

Hello,

I am still having issues restricting the extent in my mapcache area using 
rulesets.  I created a rule for all of my zoom levels that has a limited 
visibility extent.  However, Mapcache is still returning imagery for the entire 
grid area even though I created a rule that should only display a limited 
extent.  I am still not sure why the visibility extent rule seems ineffective.

Thanks!


You can set the

empty_img

To return a blank tile.  I believe the status is still 404 - I don't think you 
can control this

On Fri, 30 Aug 2019 at 15:44, Mark Volz 
mailto:markv...@co.lyon.mn.us>> wrote:
Hello,
 I am in the process of limiting the cached area in mapcache.  I am having 
trouble with both the restricted_extent and ruleset parameters.  I first 
started by using restricted_extent such as “LyonCC”.  That parameter seemed to work.  However 
mapcache returned 404 errors for any tiles outside of the restricted extent.  
Next I tried to use rulesets instead.  The rulesets seem to have no effect, and 
new tiles were getting generated outside of the visibility extent.
 Question 1:  When we use restricted extents on a grid should MapCache return a 
transparent tile, white tile, or a 404 error?  Can we control what it returns?
 Question 2:  Do I have my ruleset configured appropriately?
 
LyonCC

 


435567 98889 568247 259687



 Thank You!

Sincerely,
Mark Volz, GISP
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/mapserver-users<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-users=02%7C01%7Csteve.lime%40state.mn.us%7Cd77753cd04294d16be5708d732f91114%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C637033917625038578=%2FdtbdLEeeiE8S4PsFkY84yNJRe2TeJyQVA%2BdJGflxsg%3D=0>
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache: restricted extent and ruleset to limit cache area seems ineffective

[Mark Volz]

Hello,

I am still having issues restricting the extent in my mapcache area using 
rulesets.  I created a rule for all of my zoom levels that has a limited 
visibility extent.  However, Mapcache is still returning imagery for the entire 
grid area even though I created a rule that should only display a limited 
extent.  I am still not sure why the visibility extent rule seems ineffective.

Thanks!


You can set the

empty_img

To return a blank tile.  I believe the status is still 404 - I don't think you 
can control this

On Fri, 30 Aug 2019 at 15:44, Mark Volz 
mailto:markv...@co.lyon.mn.us>> wrote:
Hello,
 I am in the process of limiting the cached area in mapcache.  I am having 
trouble with both the restricted_extent and ruleset parameters.  I first 
started by using restricted_extent such as “LyonCC”.  That parameter seemed to work.  However 
mapcache returned 404 errors for any tiles outside of the restricted extent.  
Next I tried to use rulesets instead.  The rulesets seem to have no effect, and 
new tiles were getting generated outside of the visibility extent.
 Question 1:  When we use restricted extents on a grid should MapCache return a 
transparent tile, white tile, or a 404 error?  Can we control what it returns?
 Question 2:  Do I have my ruleset configured appropriately?
 
LyonCC

 


435567 98889 568247 259687



 Thank You!

Sincerely,
Mark Volz, GISP
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache: restricted extent and ruleset to limit cache area seems ineffective

[Travis Kirstine]

You can set the

empty_img

To return a blank tile.  I believe the status is still 404 - I don't
think you can control this


On Fri, 30 Aug 2019 at 15:44, Mark Volz  wrote:

> Hello,
>
>
>
> I am in the process of limiting the cached area in mapcache.  I am having
> trouble with both the restricted_extent and ruleset parameters.  I first
> started by using restricted_extent such as “LyonCC”.  That parameter seemed to work.
> However mapcache returned 404 errors for any tiles outside of the
> restricted extent.  Next I tried to use rulesets instead.  The rulesets
> seem to have no effect, and new tiles were getting generated outside of the
> visibility extent.
>
>
>
> Question 1:  When we use restricted extents on a grid should MapCache
> return a transparent tile, white tile, or a 404 error?  Can we control what
> it returns?
>
>
>
> Question 2:  Do I have my ruleset configured appropriately?
>
>
>
> 
>
> LyonCC
>
> 
>
>
>
> 
>
>  levels?-->
>
> 
>
> 435567 98889 568247 259687
>
> 
>
> 
>
> 
>
>
>
> Thank You!
>
>
>
> Sincerely,
>
> *Mark Volz, GISP*
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] mapcache: restricted extent and ruleset to limit cache area seems ineffective

[Mark Volz]

Hello,

I am in the process of limiting the cached area in mapcache.  I am having 
trouble with both the restricted_extent and ruleset parameters.  I first 
started by using restricted_extent such as "LyonCC".  That parameter seemed to work.  However 
mapcache returned 404 errors for any tiles outside of the restricted extent.  
Next I tried to use rulesets instead.  The rulesets seem to have no effect, and 
new tiles were getting generated outside of the visibility extent.

Question 1:  When we use restricted extents on a grid should MapCache return a 
transparent tile, white tile, or a 404 error?  Can we control what it returns?

Question 2:  Do I have my ruleset configured appropriately?


LyonCC





435567 98889 568247 259687




Thank You!

Sincerely,
Mark Volz, GISP
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache: Limiting a cached area by using a shapefile, Adjusting the Google Compatible 3857 grid, Considerations if multiple grids use the same projection?

[Mark Volz]

Hello,

Question 1:  Limit a cached area by using a shapefile.
Does map server give us the ability to limit the extent of a cache using a 
shapefile?  I know we can set up a rectangular grid for the extent.   However, 
I would like to make sure tiles are not generated beyond the area of interest.

Question 2:  Can we or should we adjust the Google Compatible 3857 grid?
I realize I should start supporting the 3857 grid for some applications.  
However, I also don't see the need to cache the entire world.  What is the best 
approach to build a cache for a small area that is in the same tile schema that 
Google, Bing, and ArcGIS use?
*Is there a tool available to calculate the appropriate rectangular grid for an 
area of interest?
*Similar to question 1, can we limit the cached area using a shapefile?

Question 3: Are there any considerations if multiple grids use the same 
projection?
Are there any considerations if multiple grids use the same projection.  For 
example, will there be a problem if we have a cache for County A, County B, and 
County C all using the 3857 grid?  After adding the WMS layer I noticed that 
the dialog for ArcMap-->WMS Layer-->Change Coordinate System will list every 
projection configured in MapCache even if that particular projection applies to 
a different county.
*Will Mapcache work with multiple grids using the same projection?
*Am I better off setting up several mapcache aliases such as 
https://jingthegisguy.wordpress.com/2015/05/06/mapcache-the-basic-settings/?

Thank You!
Sincerely,
Mark Volz, GISP

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache seeding from ogr source

[tellett]

Hi, I have several seeding jobs set up that are using an ogr source (PostGIS
table with line geometries). I'm wondering if mapcache reads the postgis
table into memory and run the seeding from there, or creates a new
conenction for every row in the table?

Thanks

Tom



--
Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache 1.8.0 Released

[Jérome Boué]

The project team is glad to annouce the release of MapCache 1.8.0.

This version includes a long list of changes since 1.6.1.


Changelog: 
https://mapserver.org/development/changelog/mapcache/changelog-1-8.html


Download: http://download.osgeo.org/mapserver/mapcache-1.8.0.tar.gz


Thanks,

MapServer / MapCache team


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache seeder question

[Matej Mailing]

Hello,
I have found out an interesting 'issue'.
If I execute mapcache with
mapcache_seed -c /mapcache.xml -t dof025 -g SLOGRID -n 1 -e
533124,86639,544462,103589 -z 1,5, it works (!), but if I change the
parameters order to

mapcache_seed -c /etc/mapserver/mapcache.xml -t dof025 -g SLOGRID -n 1
-z 1,5 -e 533124,86639,544462,103589, it doesn't.
Basically I have found out that -z has to be defined after the extent
to make it work.

This might be a bug, can someone confirm please?
Thanks,
Matej

V V tor., 16. jul. 2019 ob 11:24 je oseba Matej Mailing
 napisala:
>
> Yes, it works and the cache directory size is increasing.
>
> V V pon., 8. jul. 2019 ob 17:29 je oseba Lime, Steve D (MNIT)
>  napisala:
> >
> > And regular tile generation (no seeding) works ok?
> >
> > -Original Message-
> > From: Matej Mailing [mailto:mail...@tam.si]
> > Sent: Monday, July 08, 2019 12:26 AM
> > To: Lime, Steve D (MNIT) ; 
> > mapserver-users@lists.osgeo.org
> > Subject: Re: [mapserver-users] Mapcache seeder question
> >
> > This message may be from an external email source.
> > Do not select links or open attachments unless verified. Report all 
> > suspicious emails to Minnesota IT Services Security Operations Center.
> >
> >
> >
> > Hello Steve,
> >
> > thanks for answering. I have changed the extent definition in grid
> > declaration, but I still get the "0 tiles needed to be seeded,
> > exiting" message back from mapcache_seed.
> >
> > Any ideas will be appreciated.
> >
> > Thanks,
> > Matej
> >
> > V V pet., 5. jul. 2019 ob 23:00 je oseba Lime, Steve D (MNIT)
> >  napisala:
> > >
> > > I believe the  in your grid should be space delimited - 
> > > no commas. At least that's how I have things setup, for example...
> > >
> > > 
> > > 
> > >   Standard Minnesota UTM Grid
> > > 
> > > 105000 4785000 809000 5489000
> > > epsg:26915
> > > 256 256
> > > 5500 2750 1375 687.5 343.75 171.875 85.9375 42.96875 
> > > 21.484375 10.7421875 5.37109375 2.685546875 1.3427734375
> > >   
> > >
> > > --Steve
> > >
> > > -Original Message-
> > > From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] On 
> > > Behalf Of Matej Mailing
> > > Sent: Thursday, July 04, 2019 10:35 AM
> > > To: mapserver-users@lists.osgeo.org
> > > Subject: [mapserver-users] Mapcache seeder question
> > >
> > > This message may be from an external email source.
> > > Do not select links or open attachments unless verified. Report all 
> > > suspicious emails to Minnesota IT Services Security Operations Center.
> > >
> > >
> > >
> > > Hello,
> > > I am having issues with running Mapcache Seeder with extent options.
> > > My grid is defined like:
> > >   
> > > 256 256
> > > 373233,31868 625985,194134
> > > EPSG:3912
> > > m
> > > 18 9 45000 22500 11250 5000 2500 1250 600 300
> > > 200 100 50 25 12 6 3 1 0.5 0.25 0.12 0.06
> > >
> > >
> > > and when I try to run seeder with:
> > > mapcache_seed -c mapcache.xml -t dof025 -g SLOGRID -z 3,10 -n 6 -e
> > > 533124,86639,544462,103589
> > >
> > > I get an output 0 tiles needed to be seeded, exiting, which is
> > > certainly not true since plenty of those do not exist in the cache
> > > directory.
> > >
> > > What am I missing?
> > >
> > > Thanks,
> > > Matej
> > > ___
> > > mapserver-users mailing list
> > > mapserver-users@lists.osgeo.org
> > > https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-usersdata=02%7C01%7Csteve.lime%40state.mn.us%7C202cd663129445fe4f3908d70364db3a%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C636981604044024635sdata=H7Hfyb9kyJueMQ%2Bh9puMB4C1WyTzrI3I8RzeFqvLXVI%3Dreserved=0
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache 1.8.0-rc2 ready for testing

[Jérome Boué]

Today the project team has packaged the second release candidate of
MapCache 1.8.0, which includes a  bug fix in addition to a long list of 
changes from first release candidate.



Changelog: 



Download: 


Thanks,

- MapServer / MapCache team



___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache seeder question

[Matej Mailing]

Yes, it works and the cache directory size is increasing.

V V pon., 8. jul. 2019 ob 17:29 je oseba Lime, Steve D (MNIT)
 napisala:
>
> And regular tile generation (no seeding) works ok?
>
> -Original Message-
> From: Matej Mailing [mailto:mail...@tam.si]
> Sent: Monday, July 08, 2019 12:26 AM
> To: Lime, Steve D (MNIT) ; 
> mapserver-users@lists.osgeo.org
> Subject: Re: [mapserver-users] Mapcache seeder question
>
> This message may be from an external email source.
> Do not select links or open attachments unless verified. Report all 
> suspicious emails to Minnesota IT Services Security Operations Center.
>
>
>
> Hello Steve,
>
> thanks for answering. I have changed the extent definition in grid
> declaration, but I still get the "0 tiles needed to be seeded,
> exiting" message back from mapcache_seed.
>
> Any ideas will be appreciated.
>
> Thanks,
> Matej
>
> V V pet., 5. jul. 2019 ob 23:00 je oseba Lime, Steve D (MNIT)
>  napisala:
> >
> > I believe the  in your grid should be space delimited - no 
> > commas. At least that's how I have things setup, for example...
> >
> > 
> > 
> >   Standard Minnesota UTM Grid
> > 
> > 105000 4785000 809000 5489000
> > epsg:26915
> > 256 256
> > 5500 2750 1375 687.5 343.75 171.875 85.9375 42.96875 
> > 21.484375 10.7421875 5.37109375 2.685546875 1.3427734375
> >   
> >
> > --Steve
> >
> > -Original Message-
> > From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] On 
> > Behalf Of Matej Mailing
> > Sent: Thursday, July 04, 2019 10:35 AM
> > To: mapserver-users@lists.osgeo.org
> > Subject: [mapserver-users] Mapcache seeder question
> >
> > This message may be from an external email source.
> > Do not select links or open attachments unless verified. Report all 
> > suspicious emails to Minnesota IT Services Security Operations Center.
> >
> >
> >
> > Hello,
> > I am having issues with running Mapcache Seeder with extent options.
> > My grid is defined like:
> >   
> > 256 256
> > 373233,31868 625985,194134
> > EPSG:3912
> > m
> > 18 9 45000 22500 11250 5000 2500 1250 600 300
> > 200 100 50 25 12 6 3 1 0.5 0.25 0.12 0.06
> >
> >
> > and when I try to run seeder with:
> > mapcache_seed -c mapcache.xml -t dof025 -g SLOGRID -z 3,10 -n 6 -e
> > 533124,86639,544462,103589
> >
> > I get an output 0 tiles needed to be seeded, exiting, which is
> > certainly not true since plenty of those do not exist in the cache
> > directory.
> >
> > What am I missing?
> >
> > Thanks,
> > Matej
> > ___
> > mapserver-users mailing list
> > mapserver-users@lists.osgeo.org
> > https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-usersdata=02%7C01%7Csteve.lime%40state.mn.us%7C202cd663129445fe4f3908d70364db3a%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C636981604044024635sdata=H7Hfyb9kyJueMQ%2Bh9puMB4C1WyTzrI3I8RzeFqvLXVI%3Dreserved=0
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache configuration for openlayers XYZ source

[Eichner, Andreas - SID]

Hello Sacha,

according to https://mapserver.org/mapcache/sources.html#wmts-sources MapCache 
has no direct support for such source. You might try the support through GDAL 
mentioned there using a service description 
(https://gdal.org/drivers/raster/wmts.html#local-service-description-xml-file ).

HTH


> -Original Message-
> From: mapserver-users [mailto:mapserver-users-
> boun...@lists.osgeo.org] On Behalf Of Sacha Fouchard
> Sent: Thursday, July 11, 2019 10:21 AM
> To: mapserver-users@lists.osgeo.org
> Subject: [mapserver-users] mapcache configuration for openlayers
> XYZ source
> 
> Hi Jeff,
> I followed the link you gave and it works perfectly but for
> ol.source.tileWMS.
> I need to use ol.source.XYZ and get tiles with
> http://my.server/z/x/y.png scheme.
> In my example, i have a source: http://{1-
> 4}.basemaps.cartocdn.com/dark_nolabels/{z}/{x}/{y}.png
> <http://basemaps.cartocdn.com/dark_nolabels/%7Bz%7D/%7Bx%7D/%7By%7
> D.png> , declared in openlayers as  ol.source.XYZ
> <http://ol.source.xyz/>  <http://ol.source.XYZ
> <http://ol.source.xyz/> >.
> In line, no problem. Off line, I really don't know how to cache
> it. What could be a simple mapcache configuration? and what url
> from openlayers?
> regards
> Sacha
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache configuration for openlayers XYZ source

[Sacha Fouchard]

Hi Jeff,
I followed the link you gave and it works perfectly but for
ol.source.tileWMS.
I need to use ol.source.XYZ and get tiles with http://my.server/z/x/y.png
scheme.
In my example, i have a source:
http://{1-4}.basemaps.cartocdn.com/dark_nolabels/{z}/{x}/{y}.png, declared
in openlayers as  ol.source.XYZ .
In line, no problem. Off line, I really don't know how to cache it. What
could be a simple mapcache configuration? and what url from openlayers?
regards
Sacha



--
Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-User-f4226646.html
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] mapcache configuration for openlayers XYZ source

[Sacha Fouchard]

Hi Jeff,
I followed the link you gave and it works perfectly but for
ol.source.tileWMS.
I need to use ol.source.XYZ and get tiles with http://my.server/z/x/y.png
scheme.
In my example, i have a source: http://{1-4}.
basemaps.cartocdn.com/dark_nolabels/{z}/{x}/{y}.png
,
declared in openlayers as  ol.source.XYZ  <
http://ol.source.XYZ >.
In line, no problem. Off line, I really don't know how to cache it. What
could be a simple mapcache configuration? and what url from openlayers?
regards
Sacha
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache configuration for openlayers XYZ source

[Jeff McKenna]

I don't know that it applies directly to your current needs, but there 
is a wiki page outlining specific steps to enable OpenLayers + MapCache 
at https://ms4w.com/trac/wiki/MapCacheOpenLayers


-jeff




--
Jeff McKenna
MapServer Consulting and Training Services
https://gatewaygeomatics.com/



On 2019-07-10 10:12 AM, Sacha Fouchard wrote:

Dear mapserver users,
I am pretty new with mapcache configuration and try without success to 
manage an openlayers 4 ol.source.XYZ  cache.
Url I need to cache is: 
http://{1-4}.basemaps.cartocdn.com/dark_nolabels/{z}/{x}/{y}.png 


This is working in openlayers directly.
For a mapcache use, what would be the simplest mapcache.xml configuration?

For ol.source.TileWMS, i fallowed mapcache configuration example and it 
works prefectly.


Best regards for all of you.



___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] mapcache configuration for openlayers XYZ source

[Sacha Fouchard]

Dear mapserver users,
I am pretty new with mapcache configuration and try without success to
manage an openlayers 4  ol.source.XYZ cache.
Url I need to cache is: http://{1-4}.
basemaps.cartocdn.com/dark_nolabels/{z}/{x}/{y}.png
This is working in openlayers directly.
For a mapcache use, what would be the simplest mapcache.xml configuration?

For ol.source.TileWMS, i fallowed mapcache configuration example and it
works prefectly.

Best regards for all of you.
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] MapCache 1.8.0-rc1 ready for testing

[Jeff McKenna]

Today the project team has packaged the first release candidate of 
MapCache 1.8.0, which includes a long list of changes.


  changelog: 
https://mapserver.org/development/changelog/mapcache/changelog-1-8.html#changelog-mapcache-1-8-0-rc1

  download: https://mapserver.org/download.html#development-releases

thanks,


-MapServer / MapCache team






___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache seeder question

[Lime, Steve D (MNIT)]

And regular tile generation (no seeding) works ok? 

-Original Message-
From: Matej Mailing [mailto:mail...@tam.si] 
Sent: Monday, July 08, 2019 12:26 AM
To: Lime, Steve D (MNIT) ; 
mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] Mapcache seeder question

This message may be from an external email source.
Do not select links or open attachments unless verified. Report all suspicious 
emails to Minnesota IT Services Security Operations Center.



Hello Steve,

thanks for answering. I have changed the extent definition in grid
declaration, but I still get the "0 tiles needed to be seeded,
exiting" message back from mapcache_seed.

Any ideas will be appreciated.

Thanks,
Matej

V V pet., 5. jul. 2019 ob 23:00 je oseba Lime, Steve D (MNIT)
 napisala:
>
> I believe the  in your grid should be space delimited - no 
> commas. At least that's how I have things setup, for example...
>
> 
> 
>   Standard Minnesota UTM Grid
> 
> 105000 4785000 809000 5489000
> epsg:26915
> 256 256
> 5500 2750 1375 687.5 343.75 171.875 85.9375 42.96875 
> 21.484375 10.7421875 5.37109375 2.685546875 1.3427734375
>   
>
> --Steve
>
> -Original Message-
> From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] On 
> Behalf Of Matej Mailing
> Sent: Thursday, July 04, 2019 10:35 AM
> To: mapserver-users@lists.osgeo.org
> Subject: [mapserver-users] Mapcache seeder question
>
> This message may be from an external email source.
> Do not select links or open attachments unless verified. Report all 
> suspicious emails to Minnesota IT Services Security Operations Center.
>
>
>
> Hello,
> I am having issues with running Mapcache Seeder with extent options.
> My grid is defined like:
>   
> 256 256
> 373233,31868 625985,194134
> EPSG:3912
> m
> 18 9 45000 22500 11250 5000 2500 1250 600 300
> 200 100 50 25 12 6 3 1 0.5 0.25 0.12 0.06
>
>
> and when I try to run seeder with:
> mapcache_seed -c mapcache.xml -t dof025 -g SLOGRID -z 3,10 -n 6 -e
> 533124,86639,544462,103589
>
> I get an output 0 tiles needed to be seeded, exiting, which is
> certainly not true since plenty of those do not exist in the cache
> directory.
>
> What am I missing?
>
> Thanks,
> Matej
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-usersdata=02%7C01%7Csteve.lime%40state.mn.us%7C202cd663129445fe4f3908d70364db3a%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C636981604044024635sdata=H7Hfyb9kyJueMQ%2Bh9puMB4C1WyTzrI3I8RzeFqvLXVI%3Dreserved=0
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache seeder question

[Matej Mailing]

Hello Steve,

thanks for answering. I have changed the extent definition in grid
declaration, but I still get the "0 tiles needed to be seeded,
exiting" message back from mapcache_seed.

Any ideas will be appreciated.

Thanks,
Matej

V V pet., 5. jul. 2019 ob 23:00 je oseba Lime, Steve D (MNIT)
 napisala:
>
> I believe the  in your grid should be space delimited - no 
> commas. At least that's how I have things setup, for example...
>
> 
> 
>   Standard Minnesota UTM Grid
> 
> 105000 4785000 809000 5489000
> epsg:26915
> 256 256
> 5500 2750 1375 687.5 343.75 171.875 85.9375 42.96875 
> 21.484375 10.7421875 5.37109375 2.685546875 1.3427734375
>   
>
> --Steve
>
> -Original Message-
> From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] On 
> Behalf Of Matej Mailing
> Sent: Thursday, July 04, 2019 10:35 AM
> To: mapserver-users@lists.osgeo.org
> Subject: [mapserver-users] Mapcache seeder question
>
> This message may be from an external email source.
> Do not select links or open attachments unless verified. Report all 
> suspicious emails to Minnesota IT Services Security Operations Center.
>
>
>
> Hello,
> I am having issues with running Mapcache Seeder with extent options.
> My grid is defined like:
>   
> 256 256
> 373233,31868 625985,194134
> EPSG:3912
> m
> 18 9 45000 22500 11250 5000 2500 1250 600 300
> 200 100 50 25 12 6 3 1 0.5 0.25 0.12 0.06
>
>
> and when I try to run seeder with:
> mapcache_seed -c mapcache.xml -t dof025 -g SLOGRID -z 3,10 -n 6 -e
> 533124,86639,544462,103589
>
> I get an output 0 tiles needed to be seeded, exiting, which is
> certainly not true since plenty of those do not exist in the cache
> directory.
>
> What am I missing?
>
> Thanks,
> Matej
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-usersdata=02%7C01%7Csteve.lime%40state.mn.us%7Ca8f92e6de3474b3eb52608d7009534a0%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C636978513162884312sdata=FdCj%2F%2F704BA8lCcJ7cO0t7erzHPXs56%2BpgCXLIugok8%3Dreserved=0
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache seeder question

[Lime, Steve D (MNIT)]

I believe the  in your grid should be space delimited - no 
commas. At least that's how I have things setup, for example...



  Standard Minnesota UTM Grid

105000 4785000 809000 5489000
epsg:26915
256 256
5500 2750 1375 687.5 343.75 171.875 85.9375 42.96875 21.484375 
10.7421875 5.37109375 2.685546875 1.3427734375
  

--Steve

-Original Message-
From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] On 
Behalf Of Matej Mailing
Sent: Thursday, July 04, 2019 10:35 AM
To: mapserver-users@lists.osgeo.org
Subject: [mapserver-users] Mapcache seeder question

This message may be from an external email source.
Do not select links or open attachments unless verified. Report all suspicious 
emails to Minnesota IT Services Security Operations Center.



Hello,
I am having issues with running Mapcache Seeder with extent options.
My grid is defined like:
  
256 256
373233,31868 625985,194134
EPSG:3912
m
18 9 45000 22500 11250 5000 2500 1250 600 300
200 100 50 25 12 6 3 1 0.5 0.25 0.12 0.06
   

and when I try to run seeder with:
mapcache_seed -c mapcache.xml -t dof025 -g SLOGRID -z 3,10 -n 6 -e
533124,86639,544462,103589

I get an output 0 tiles needed to be seeded, exiting, which is
certainly not true since plenty of those do not exist in the cache
directory.

What am I missing?

Thanks,
Matej
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-usersdata=02%7C01%7Csteve.lime%40state.mn.us%7Ca8f92e6de3474b3eb52608d7009534a0%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C636978513162884312sdata=FdCj%2F%2F704BA8lCcJ7cO0t7erzHPXs56%2BpgCXLIugok8%3Dreserved=0
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache seeder question

[Matej Mailing]

Hello,
I am having issues with running Mapcache Seeder with extent options.
My grid is defined like:
  
256 256
373233,31868 625985,194134
EPSG:3912
m
18 9 45000 22500 11250 5000 2500 1250 600 300
200 100 50 25 12 6 3 1 0.5 0.25 0.12 0.06
   

and when I try to run seeder with:
mapcache_seed -c mapcache.xml -t dof025 -g SLOGRID -z 3,10 -n 6 -e
533124,86639,544462,103589

I get an output 0 tiles needed to be seeded, exiting, which is
certainly not true since plenty of those do not exist in the cache
directory.

What am I missing?

Thanks,
Matej
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache: Modifying GoogleMapsCompatible grid?

[Lime, Steve D (MNIT)]

I would create a new grid, coping from the default grid, and then add 
additional zoom levels...

From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] On 
Behalf Of Mark Volz
Sent: Monday, April 29, 2019 12:25 PM
To: mapserver-users@lists.osgeo.org
Subject: [mapserver-users] Mapcache: Modifying GoogleMapsCompatible grid?

Hello,

Within Mapcache, Is it considered "good form" to modify the 
GoogleMapsCompatible grid to add additional resolutions so we can zoom in 
closer to the map?  Will I just need to override the grid in mapcache.xml, or 
will I need to create a new grid by copying the parameters on 
https://mapserver.org/mapcache/config.html<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmapserver.org%2Fmapcache%2Fconfig.html=02%7C01%7Csteve.lime%40state.mn.us%7C120c5ea99312453f14ce08d6ccca27c2%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C636921565955905031=JuyBvLPkDGNIjvhxEVA5mpFIJdPwH0VpRA6T35VBopY%3D=0>?

Background:
WMS seems to work in ArcMap using any MapCache grid (UTM 15N, Lyon County, 
Google, WGS84).  However when using WMTS, ArcMap only displays the Google and 
WGS 84 grids in the correct location.  Therefore I plan on using the 
GoogleMapsCompatible grid until the issue with ArcMap is resolved, but I would 
like to zoom in closer on the map.

Thanks

Sincerely,
Mark Volz, GISP
Lyon County GIS Coordinator

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache: Modifying GoogleMapsCompatible grid?

[Mark Volz]

Hello,

Within Mapcache, Is it considered "good form" to modify the 
GoogleMapsCompatible grid to add additional resolutions so we can zoom in 
closer to the map?  Will I just need to override the grid in mapcache.xml, or 
will I need to create a new grid by copying the parameters on 
https://mapserver.org/mapcache/config.html?

Background:
WMS seems to work in ArcMap using any MapCache grid (UTM 15N, Lyon County, 
Google, WGS84).  However when using WMTS, ArcMap only displays the Google and 
WGS 84 grids in the correct location.  Therefore I plan on using the 
GoogleMapsCompatible grid until the issue with ArcMap is resolved, but I would 
like to zoom in closer on the map.

Thanks

Sincerely,
Mark Volz, GISP
Lyon County GIS Coordinator

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapcache WMTS - imagery is offset in ArcGIS when using WMTS services.

[Lime, Steve D (MNIT)]

The resolution doesn't matter. That just impacts scale computations internally 
so when layers/classes are on/off and how symbol/font scaling is done.

I just tried our Compass MapCache/WTMS service overlaid with a shapefile in UTM 
projected appropriately and they line up perfectly in ArcGIS 10.6. You can try 
it too with your data since you're in MN 
(http://maps1.dnr.state.mn.us/mapcache/wmts/1.0.0/WMTSCapabilities.xml). Does 
it behave any differently for you?

--Steve

From: mapserver-users [mailto:mapserver-users-boun...@lists.osgeo.org] On 
Behalf Of Mark Volz
Sent: Friday, April 12, 2019 12:08 PM
To: mapserver-users@lists.osgeo.org
Subject: [mapserver-users] Mapcache WMTS - imagery is offset in ArcGIS when 
using WMTS services.

Hello,

I noticed that WMTS services are extremely fast when using Mapcache to serve 
tiles to ArcGIS clients.  However, the imagery appears to be shifted down 
significantly from the actual location.   I did notice that Mapserver and 
Mapcache use a 72 DPI while ArcGIS is using a 90.714 DPI.  In addition the 
image is the correct width and height.  In addition, when I use WMS the imagery 
appears in the correct location, however it is not as fast as WMTS.

Please let me know if there is a way to get ArcGIS to place imagery in the 
correct location when using WMTS and Mapcache.

Thanks

Sincerely,
Mark Volz, GISP
Lyon County GIS Coordinator

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapcache WMTS - imagery is offset in ArcGIS when using WMTS services.

[Mark Volz]

Hello,

I noticed that WMTS services are extremely fast when using Mapcache to serve 
tiles to ArcGIS clients.  However, the imagery appears to be shifted down 
significantly from the actual location.   I did notice that Mapserver and 
Mapcache use a 72 DPI while ArcGIS is using a 90.714 DPI.  In addition the 
image is the correct width and height.  In addition, when I use WMS the imagery 
appears in the correct location, however it is not as fast as WMTS.

Please let me know if there is a way to get ArcGIS to place imagery in the 
correct location when using WMTS and Mapcache.

Thanks

Sincerely,
Mark Volz, GISP
Lyon County GIS Coordinator

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] mapcache for website tips

[Sebastiano Laini]

Hi all,

I'm curious to know what are your mapcache configurations for a use in a 
website.

I'm using tiles of 128x128 and the maximum weight of a single tile can be 80KB, 
our map is counting 33 layers and a total of 60GB of data (whole UK EPSG:27700, 
Ireland is just the border and a white background). On first load with 
OpenLayer 4 and preload infinity (Preload. Load low-resolution tiles up to 
preload levels. 0 means no preloading) I load around 350 tiles and the load 
times is around 8 second (if the zoom layer is cached)

I'm also seeding with metatile 4 4 that was the old standard used before I 
changed the tilesize, which are the best settings I should use for web (tile 
size and which should be the max file size) and seed caching? As you know in 
previous posts I'm having really big issue in seeding the whole projection and 
I'm trying to figure out how to improve the service. (bear in mind that we 
update the data 2 at year so I need to recache again every 6 months)

Sebastiano Laini
Web Developer
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed slow

[Sebastiano Laini]

We have 2 dedicated servers that we use to serve the full mapping system and 
when we update the map we switch the system to just one server and use the 
other one just to see the new map with a different URL.

Sebastiano Laini
Web Developer

From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi]
Sent: 22 February 2019 15:03
To: Sebastiano Laini ; 
'mapserver-users@lists.osgeo.org' 
Subject: Re: mapcache seed slow

Hi,

Stupid question, but do you know how fast your WMS service is? How many GetMap 
requests/sec it can serve with the metatile size that you are seeding with? 
Have you some evidence that WMS is running idle while MapCache is busy with 
cutting the tiles and filling the cache?

-Jukka Rahkonen-

Lähettäjä: mapserver-users 
mailto:mapserver-users-boun...@lists.osgeo.org>>
 Puolesta Sebastiano Laini
Lähetetty: perjantai 22. helmikuuta 2019 13.35
Vastaanottaja: 'mapserver-users@lists.osgeo.org' 
mailto:mapserver-users@lists.osgeo.org>>
Aihe: [mapserver-users] mapcache seed slow

Hi all,

I'm trying to seed the whole UK at all zoom lv with these resolutions

280 140 70 28 14 7 6.16 2.8 1.4 0.7

But after 13 days it arrives at the 5th level where before I kill the process 
it seed just 900MB on that level.

This is my script

su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c 
/usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 
400" 2>&1>  /var/www/vhosts/$SITE/scripts/cache_log.txt

I know that in the documentation say that -n should be less or max the number 
of cpu in the server but is really slow if I put 2, so I tried with 100 and 
after 13 days that was the result so now I'm trying 400

Is there something else I can use to speed up the seed? I'm using the SQlite 
cache method

Should I use the -p (| -nprocesses: number of parallel processes that should be 
used to request tiles from the WMS source) Instead of the -n parameter?

Sebastiano Laini
Web Developer

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed slow

[Sebastiano Laini]

I don't know how fast is but I have a whole dedicated server just for do that 
with a development URL that I'm using just to develop, is still configured in 
HTTP/1, looking to make some test on a new server to setup HTTP/2 with NGINX 
and possibly WMTS.

I'm seeding with a bash file and I can see (with TOP) that it's using about 
60/80% CPU usage and other HTTPD requests make it from the APACHE user.

Sebastiano Laini
Web Developer

From: Rahkonen Jukka (MML) [mailto:jukka.rahko...@maanmittauslaitos.fi]
Sent: 22 February 2019 15:03
To: Sebastiano Laini ; 
'mapserver-users@lists.osgeo.org' 
Subject: Re: mapcache seed slow

Hi,

Stupid question, but do you know how fast your WMS service is? How many GetMap 
requests/sec it can serve with the metatile size that you are seeding with? 
Have you some evidence that WMS is running idle while MapCache is busy with 
cutting the tiles and filling the cache?

-Jukka Rahkonen-

Lähettäjä: mapserver-users 
mailto:mapserver-users-boun...@lists.osgeo.org>>
 Puolesta Sebastiano Laini
Lähetetty: perjantai 22. helmikuuta 2019 13.35
Vastaanottaja: 'mapserver-users@lists.osgeo.org' 
mailto:mapserver-users@lists.osgeo.org>>
Aihe: [mapserver-users] mapcache seed slow

Hi all,

I'm trying to seed the whole UK at all zoom lv with these resolutions

280 140 70 28 14 7 6.16 2.8 1.4 0.7

But after 13 days it arrives at the 5th level where before I kill the process 
it seed just 900MB on that level.

This is my script

su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c 
/usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 
400" 2>&1>  /var/www/vhosts/$SITE/scripts/cache_log.txt

I know that in the documentation say that -n should be less or max the number 
of cpu in the server but is really slow if I put 2, so I tried with 100 and 
after 13 days that was the result so now I'm trying 400

Is there something else I can use to speed up the seed? I'm using the SQlite 
cache method

Should I use the -p (| -nprocesses: number of parallel processes that should be 
used to request tiles from the WMS source) Instead of the -n parameter?

Sebastiano Laini
Web Developer

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed slow

[Rahkonen Jukka (MML)]

Hi,

Stupid question, but do you know how fast your WMS service is? How many GetMap 
requests/sec it can serve with the metatile size that you are seeding with? 
Have you some evidence that WMS is running idle while MapCache is busy with 
cutting the tiles and filling the cache?

-Jukka Rahkonen-

Lähettäjä: mapserver-users  Puolesta 
Sebastiano Laini
Lähetetty: perjantai 22. helmikuuta 2019 13.35
Vastaanottaja: 'mapserver-users@lists.osgeo.org' 

Aihe: [mapserver-users] mapcache seed slow

Hi all,

I'm trying to seed the whole UK at all zoom lv with these resolutions

280 140 70 28 14 7 6.16 2.8 1.4 0.7

But after 13 days it arrives at the 5th level where before I kill the process 
it seed just 900MB on that level.

This is my script

su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c 
/usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 
400" 2>&1>  /var/www/vhosts/$SITE/scripts/cache_log.txt

I know that in the documentation say that -n should be less or max the number 
of cpu in the server but is really slow if I put 2, so I tried with 100 and 
after 13 days that was the result so now I'm trying 400

Is there something else I can use to speed up the seed? I'm using the SQlite 
cache method

Should I use the -p (| -nprocesses: number of parallel processes that should be 
used to request tiles from the WMS source) Instead of the -n parameter?

Sebastiano Laini
Web Developer

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed slow

[Sebastiano Laini]

I’m using this configuration

  
  
/var/www/vhosts/buchananmapping.co.uk/httpdocs/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3
  1
  1
  1573741823
  

  

  
image/png
miniscale
…/maps/bcmaps.map
  


  https://.../cgi-bin/mapserv?

  

  
bcmaps
cache_sqlite
bcgrid
PNG
4 4
0
2628000
  

  
   
  BC custom grid
   
   EPSG:27700
   128 128
   0 0 70 125
   280 140 70 28 14 7 6.16 2.8 1.4 0.7
  

  
   best
  

It supposed to have a limit that it’s the  full EPSG extent

Sebastiano Laini
Web Developer

From: Travis Kirstine [mailto:traviskirst...@gmail.com]
Sent: 22 February 2019 13:57
To: Sebastiano Laini 
Cc: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] mapcache seed slow

The time you have indicated seems excessive to seed the UK. My guess is that 
the current process it seeding the extents of your grid (world?) but default.  
I've seeded the province of ontario 1 million km2 z0-17 in about 2 weeks using 
a sqlite cache

Ramping up the -n will only bottle neck the process more (same with the -p 
flag) the limiting factor is SQLite and it limitation on concurrent writes (I 
use -n 4).  I've noticed that if you try to use other sqlite3 journal modes you 
can degrade performance 
(http://osgeo-org.1560.x6.nabble.com/mapcache-sqlite-cache-large-wal-files-and-blocked-checkpoints-when-seeding-td5377571.html).
  I would just use most basic cache config as a starting point.  BTW using SSD 
should increase the SQLite writes as well (if you are lucky enough to have them)

   
  ./{tileset}.sqlite3
   

There are a few things that you could do when seeding to speed up the process:

Use the "restricted_extent" parameter in the tileset grid to limit the number 
of tiles

Example
   
 ontario_2018_wms
 sqlite
 gmaps
 mixed
 5 5
   

Use a polygon shapefile (or other ogr datasource) to limit the seeding using 
the -d flag in mapcache_seed command (note the projection of the datasource 
should be the same as the grid)

Use a larger metatile when seeding (-M flag override in command line).  
Mapcache will "bulk" insert the resultant tiles based on the metatile - if 
5 5 mapcache will insert 25 tiles at a time if 8 
8 it will do 64 (I think this is the way it works).  The limiting 
factor here is trying to balance this with your source WMS performance.  I use 
8,8 when seeding

Regards















On Fri, 22 Feb 2019 at 06:35, Sebastiano Laini 
mailto:sebastiano.la...@buchanancomputing.co.uk>>
 wrote:
Hi all,

I’m trying to seed the whole UK at all zoom lv with these resolutions

280 140 70 28 14 7 6.16 2.8 1.4 0.7

But after 13 days it arrives at the 5th level where before I kill the process 
it seed just 900MB on that level.

This is my script

su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c 
/usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 –n 
400" 2>&1>  /var/www/vhosts/$SITE/scripts/cache_log.txt

I know that in the documentation say that –n should be less or max the number 
of cpu in the server but is really slow if I put 2, so I tried with 100 and 
after 13 days that was the result so now I’m trying 400

Is there something else I can use to speed up the seed? I’m using the SQlite 
cache method

Should I use the -p (| –nprocesses: number of parallel processes that should be 
used to request tiles from the WMS source) Instead of the –n parameter?

Sebastiano Laini
Web Developer

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org<mailto:mapserver-users@lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] mapcache seed slow

[Travis Kirstine]

The time you have indicated seems excessive to seed the UK. My guess is
that the current process it seeding the extents of your grid (world?) but
default.  I've seeded the province of ontario 1 million km2 z0-17 in about
2 weeks using a sqlite cache

Ramping up the -n will only bottle neck the process more (same with the -p
flag) the limiting factor is SQLite and it limitation on concurrent writes
(I use -n 4).  I've noticed that if you try to use other sqlite3 journal
modes you can degrade performance (
http://osgeo-org.1560.x6.nabble.com/mapcache-sqlite-cache-large-wal-files-and-blocked-checkpoints-when-seeding-td5377571.html).
I would just use most basic cache config as a starting point.  BTW using
SSD should increase the SQLite writes as well (if you are lucky enough to
have them)

   
  ./{tileset}.sqlite3
   

There are a few things that you could do when seeding to speed up the
process:

Use the "restricted_extent" parameter in the tileset grid to limit the
number of tiles

Example
   
 ontario_2018_wms
 sqlite
 gmaps
 mixed
 5 5
   

Use a polygon shapefile (or other ogr datasource) to limit the seeding
using the -d flag in mapcache_seed command (note the projection of the
datasource should be the same as the grid)

Use a larger metatile when seeding (-M flag override in command line).
Mapcache will "bulk" insert the resultant tiles based on the metatile -
if 5 5 mapcache will insert 25 tiles at a time if
8 8 it will do 64 (I think this is the way it works).
The limiting factor here is trying to balance this with your source WMS
performance.  I use 8,8 when seeding

Regards















On Fri, 22 Feb 2019 at 06:35, Sebastiano Laini <
sebastiano.la...@buchanancomputing.co.uk> wrote:

> Hi all,
>
>
>
> I’m trying to seed the whole UK at all zoom lv with these resolutions
>
>
>
> 280 140 70 28 14 7 6.16 2.8 1.4 0.7
>
>
>
> But after 13 days it arrives at the 5th level where before I kill the
> process it seed just 900MB on that level.
>
>
>
> This is my script
>
>
>
> su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c
> /usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z
> 5,6 –n 400" 2>&1>  /var/www/vhosts/$SITE/scripts/cache_log.txt
>
>
>
> I know that in the documentation say that –n should be less or max the
> number of cpu in the server but is really slow if I put 2, so I tried with
> 100 and after 13 days that was the result so now I’m trying 400
>
>
>
> Is there something else I can use to speed up the seed? I’m using the
> SQlite cache method
>
>
>
> Should I use the -p (| –nprocesses: number of parallel processes that
> should be used to request tiles from the WMS source) Instead of the –n
> parameter?
>
>
>
> Sebastiano Laini
>
> Web Developer
>
>
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] mapcache seed slow

[Sebastiano Laini]

Hi all,

I'm trying to seed the whole UK at all zoom lv with these resolutions

280 140 70 28 14 7 6.16 2.8 1.4 0.7

But after 13 days it arrives at the 5th level where before I kill the process 
it seed just 900MB on that level.

This is my script

su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c 
/usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 -n 
400" 2>&1>  /var/www/vhosts/$SITE/scripts/cache_log.txt

I know that in the documentation say that -n should be less or max the number 
of cpu in the server but is really slow if I put 2, so I tried with 100 and 
after 13 days that was the result so now I'm trying 400

Is there something else I can use to speed up the seed? I'm using the SQlite 
cache method

Should I use the -p (| -nprocesses: number of parallel processes that should be 
used to request tiles from the WMS source) Instead of the -n parameter?

Sebastiano Laini
Web Developer

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users