date:20210423

Re: [mapserver-users] Mapserver installation in cloud environments (kubernetes)

2021-04-23 Thread Wouter Visscher

Hi Andreas,

To answer your question: "Do you know of any work "

For the Dutch National Geodata Infrastructure (PDOK) we are running now
(over a year) hundreds of OGC WMS/WMTS/WFS servers (primarily Mapserver and
Mapproxy) on aks (Azure kubernetes).

The challenges you are describing sound very similar to ours :)

To give some small inside on some of our solutions regarding:
deployment: kustomize and operators
cloud optimization: (regarding data) geohashes over GPKG and PostGIS tables
and getting the data as close to the pods as possible.
load balancing: (infrastructure wise) 'standaard'
loadbalancing with traefik, (OGC wise) we are working on a
'ogc-webservice-proxy' for determining the size/duration/location,
basically what is going to be the impact of a request.
resource sharing: (infrastructure) ScaleSets for nodes, and ReplicaSets
based on load, (data) because we have most of the data in GPKG we 'copy'
the data around. So for a service that scales up to 4 pods we have 4 of the
same GPKG copied to that node.

I will share your knowledge with my colleagues, but it seems you reached a
lot of the same conclusions (so far I have read) we have.

On Fri, Apr 23, 2021 at 5:20 PM Andreas Neumann  wrote:

> Hi,
>
> For a small project as part of the Swiss National Geodata Infrastructure
> (grant project) several people worked on a study document called
> "Cloud-optimized OGC WMS Server" where we analyzed problems that can arise
> when you install an OGC web server in the cloud (e.g. docker image deployed
> via Kubernetes, OpenShift or the likes). This work had a focus on QGIS
> Server with it's own set of problems - but some of the issues studied in
> this document also matter for other OGC WMS servers, such as UMN Mapserver
> or Geoserver, such as the load balancing problem, how to share resources,
> etc.
>
> Here is the link to the document (not in final form yet, but close to
> being final):
> https://docs.google.com/document/d/1cOUWgzalRx7CHWTFgHz6-uyScsCcoaEmYC0VBHdZShQ/edit#heading=h.c7gq4lie7ys2
>
> I wonder if any similar work has been done specifically around problems,
> challenges and solutions when you deploy UMN Mapserver in cloud
> environments? Do you know of any work?
>
> One major problem that probably all installations of an OGC WMS server
> have is how to deploy a more intelligent load balancing system? Often, the
> default load balancer is some kind of round robin load balancer system, but
> often this leads to inferior results where "cheap and short" requests (such
> as a simple GetFeatureInfo or GetLegendGraphics request) can be queued
> behind a long-running GetMap request (potentially with many layers, many
> features and a high-dpi, such as 600dpi, where the request can take several
> seconds to process.
>
> In our production system we are currently separating the requests to
> dedicated instances for short requests and potentially long requests, to
> avoid the above mentioned scenario, but we are not so satisfied with the
> solution, as it is  a bit inflexible and also a bit harder to maintain.
> Ideally, we would like to have a more intelligent load balancer with
> incoming queue that holds back requests as long as all WMS server instances
> are busy. This would avoid the situation where a "less intelligent" load
> balancer would simply forward the requests to instances based on
> Round-Robin principle.
>
> Do you know of any work in the UMN Mapserver community regarding cloud
> deployment, cloud optimization, load balancing and resource sharing?
>
> In our study document I'd like to also include the perspective of other
> WMS servers besides QGIS server, so any input would be welcome.
>
> Thanks,
> Andreas
>
> --
> Andreas Neumann
> QGIS.ORG board member (treasurer)
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapserver installation in cloud environments (kubernetes)

2021-04-23 Thread Seth G

There was an interesting talk on running MapServer on Amazon Lambda - 
https://www.youtube.com/watch?v=g_WMDxsuCfk=2s
The CGI approach of MapServer scales well as each request can be run separately 
(at a slight overhead for database connections, PROJ startup etc). 

Seth

--
web:http://geographika.co.uk
twitter: @geographika


On Fri, Apr 23, 2021, at 5:20 PM, Andreas Neumann wrote:
> Hi,
> 
> For a small project as part of the Swiss National Geodata Infrastructure 
> (grant project) several people worked on a study document called 
> "Cloud-optimized OGC WMS Server" where we analyzed problems that can arise 
> when you install an OGC web server in the cloud (e.g. docker image deployed 
> via Kubernetes, OpenShift or the likes). This work had a focus on QGIS Server 
> with it's own set of problems - but some of the issues studied in this 
> document also matter for other OGC WMS servers, such as UMN Mapserver or 
> Geoserver, such as the load balancing problem, how to share resources, etc.
> 
> Here is the link to the document (not in final form yet, but close to being 
> final): 
> https://docs.google.com/document/d/1cOUWgzalRx7CHWTFgHz6-uyScsCcoaEmYC0VBHdZShQ/edit#heading=h.c7gq4lie7ys2
> 
> I wonder if any similar work has been done specifically around problems, 
> challenges and solutions when you deploy UMN Mapserver in cloud environments? 
> Do you know of any work?
> 
> One major problem that probably all installations of an OGC WMS server have 
> is how to deploy a more intelligent load balancing system? Often, the default 
> load balancer is some kind of round robin load balancer system, but often 
> this leads to inferior results where "cheap and short" requests (such as a 
> simple GetFeatureInfo or GetLegendGraphics request) can be queued behind a 
> long-running GetMap request (potentially with many layers, many features and 
> a high-dpi, such as 600dpi, where the request can take several seconds to 
> process.
> 
> In our production system we are currently separating the requests to 
> dedicated instances for short requests and potentially long requests, to 
> avoid the above mentioned scenario, but we are not so satisfied with the 
> solution, as it is  a bit inflexible and also a bit harder to maintain. 
> Ideally, we would like to have a more intelligent load balancer with incoming 
> queue that holds back requests as long as all WMS server instances are busy. 
> This would avoid the situation where a "less intelligent" load balancer would 
> simply forward the requests to instances based on Round-Robin principle.
> 
> Do you know of any work in the UMN Mapserver community regarding cloud 
> deployment, cloud optimization, load balancing and resource sharing?
> 
> In our study document I'd like to also include the perspective of other WMS 
> servers besides QGIS server, so any input would be welcome.
> 
> Thanks,
> Andreas
> 
> --
> Andreas Neumann
> QGIS.ORG  board member (treasurer)
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org 
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
> 
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapserver installation in cloud environments (kubernetes)

2021-04-23 Thread Rahkonen Jukka (MML)

Hi,

I believe that not much that kind of studies about load balancing WMS servers 
have been done. If there is high load then people very soon place some tile 
cache into front line and use WMS only for seeding the cache. Sharing load 
between tile servers is rather simple because each tile is about as expensive 
to serve. But users do want more different styles, possibility to select just 
some layer, wider support for coordinate systems etc. that may mean more demand 
for traditional WMS because caching all alternatives gets unpractical.

-Jukka Rahkonen-

Lähettäjä: mapserver-users  Puolesta 
Andreas Neumann
Lähetetty: perjantai 23. huhtikuuta 2021 18.21
Vastaanottaja: mapserver-users@lists.osgeo.org
Aihe: [mapserver-users] Mapserver installation in cloud environments 
(kubernetes)

Hi,

For a small project as part of the Swiss National Geodata Infrastructure (grant 
project) several people worked on a study document called "Cloud-optimized OGC 
WMS Server" where we analyzed problems that can arise when you install an OGC 
web server in the cloud (e.g. docker image deployed via Kubernetes, OpenShift 
or the likes). This work had a focus on QGIS Server with it's own set of 
problems - but some of the issues studied in this document also matter for 
other OGC WMS servers, such as UMN Mapserver or Geoserver, such as the load 
balancing problem, how to share resources, etc.

Here is the link to the document (not in final form yet, but close to being 
final): 
https://docs.google.com/document/d/1cOUWgzalRx7CHWTFgHz6-uyScsCcoaEmYC0VBHdZShQ/edit#heading=h.c7gq4lie7ys2

I wonder if any similar work has been done specifically around problems, 
challenges and solutions when you deploy UMN Mapserver in cloud environments? 
Do you know of any work?

One major problem that probably all installations of an OGC WMS server have is 
how to deploy a more intelligent load balancing system? Often, the default load 
balancer is some kind of round robin load balancer system, but often this leads 
to inferior results where "cheap and short" requests (such as a simple 
GetFeatureInfo or GetLegendGraphics request) can be queued behind a 
long-running GetMap request (potentially with many layers, many features and a 
high-dpi, such as 600dpi, where the request can take several seconds to process.

In our production system we are currently separating the requests to dedicated 
instances for short requests and potentially long requests, to avoid the above 
mentioned scenario, but we are not so satisfied with the solution, as it is  a 
bit inflexible and also a bit harder to maintain. Ideally, we would like to 
have a more intelligent load balancer with incoming queue that holds back 
requests as long as all WMS server instances are busy. This would avoid the 
situation where a "less intelligent" load balancer would simply forward the 
requests to instances based on Round-Robin principle.

Do you know of any work in the UMN Mapserver community regarding cloud 
deployment, cloud optimization, load balancing and resource sharing?

In our study document I'd like to also include the perspective of other WMS 
servers besides QGIS server, so any input would be welcome.

Thanks,
Andreas

--
Andreas Neumann
QGIS.ORG board member (treasurer)
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapserver installation in cloud environments (kubernetes)

2021-04-23 Thread David Hoese


Hi Andreas,

This is very interesting. I have my own MapServer + MapCache based 
application called GeoSphere: https://geosphere.ssec.wisc.edu/


We did the bare minimum of what we had to to get it running on an 
on-premise kubernetes cluster. So while I don't have any solutions for 
you, I'm very interested in what you're doing and what you might be able 
to find. We currently don't have a lot of users so we're still only 
using a single MapCache server talking to a single MapServer server, but 
want to look into more flexibility as we offer additional layers and 
features in the future. Thanks for the document, I'll be sure to read it.


Dave

On 4/23/21 10:20 AM, Andreas Neumann wrote:

Hi,

For a small project as part of the Swiss National Geodata Infrastructure 
(grant project) several people worked on a study document called 
"Cloud-optimized OGC WMS Server" where we analyzed problems that can 
arise when you install an OGC web server in the cloud (e.g. docker image 
deployed via Kubernetes, OpenShift or the likes). This work had a focus 
on QGIS Server with it's own set of problems - but some of the issues 
studied in this document also matter for other OGC WMS servers, such as 
UMN Mapserver or Geoserver, such as the load balancing problem, how to 
share resources, etc.


Here is the link to the document (not in final form yet, but close to 
being final): 
https://docs.google.com/document/d/1cOUWgzalRx7CHWTFgHz6-uyScsCcoaEmYC0VBHdZShQ/edit#heading=h.c7gq4lie7ys2 



I wonder if any similar work has been done specifically around problems, 
challenges and solutions when you deploy UMN Mapserver in cloud 
environments? Do you know of any work?


One major problem that probably all installations of an OGC WMS server 
have is how to deploy a more intelligent load balancing system? Often, 
the default load balancer is some kind of round robin load balancer 
system, but often this leads to inferior results where "cheap and short" 
requests (such as a simple GetFeatureInfo or GetLegendGraphics request) 
can be queued behind a long-running GetMap request (potentially with 
many layers, many features and a high-dpi, such as 600dpi, where the 
request can take several seconds to process.


In our production system we are currently separating the requests to 
dedicated instances for short requests and potentially long requests, to 
avoid the above mentioned scenario, but we are not so satisfied with the 
solution, as it is  a bit inflexible and also a bit harder to maintain. 
Ideally, we would like to have a more intelligent load balancer with 
incoming queue that holds back requests as long as all WMS server 
instances are busy. This would avoid the situation where a "less 
intelligent" load balancer would simply forward the requests to 
instances based on Round-Robin principle.


Do you know of any work in the UMN Mapserver community regarding cloud 
deployment, cloud optimization, load balancing and resource sharing?


In our study document I'd like to also include the perspective of other 
WMS servers besides QGIS server, so any input would be welcome.


Thanks,
Andreas

--
Andreas Neumann
QGIS.ORG  board member (treasurer)

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Mapserver installation in cloud environments (kubernetes)

2021-04-23 Thread Andreas Neumann

Hi,

For a small project as part of the Swiss National Geodata Infrastructure
(grant project) several people worked on a study document called
"Cloud-optimized OGC WMS Server" where we analyzed problems that can arise
when you install an OGC web server in the cloud (e.g. docker image deployed
via Kubernetes, OpenShift or the likes). This work had a focus on QGIS
Server with it's own set of problems - but some of the issues studied in
this document also matter for other OGC WMS servers, such as UMN Mapserver
or Geoserver, such as the load balancing problem, how to share resources,
etc.

Here is the link to the document (not in final form yet, but close to being
final):
https://docs.google.com/document/d/1cOUWgzalRx7CHWTFgHz6-uyScsCcoaEmYC0VBHdZShQ/edit#heading=h.c7gq4lie7ys2

I wonder if any similar work has been done specifically around problems,
challenges and solutions when you deploy UMN Mapserver in cloud
environments? Do you know of any work?

One major problem that probably all installations of an OGC WMS server have
is how to deploy a more intelligent load balancing system? Often, the
default load balancer is some kind of round robin load balancer system, but
often this leads to inferior results where "cheap and short" requests (such
as a simple GetFeatureInfo or GetLegendGraphics request) can be queued
behind a long-running GetMap request (potentially with many layers, many
features and a high-dpi, such as 600dpi, where the request can take several
seconds to process.

In our production system we are currently separating the requests to
dedicated instances for short requests and potentially long requests, to
avoid the above mentioned scenario, but we are not so satisfied with the
solution, as it is  a bit inflexible and also a bit harder to maintain.
Ideally, we would like to have a more intelligent load balancer with
incoming queue that holds back requests as long as all WMS server instances
are busy. This would avoid the situation where a "less intelligent" load
balancer would simply forward the requests to instances based on
Round-Robin principle.

Do you know of any work in the UMN Mapserver community regarding cloud
deployment, cloud optimization, load balancing and resource sharing?

In our study document I'd like to also include the perspective of other WMS
servers besides QGIS server, so any input would be welcome.

Thanks,
Andreas

--
Andreas Neumann
QGIS.ORG board member (treasurer)
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] GeoJSON Date Anomalies

2021-04-23 Thread Seth G

Hi,

I'm getting some strange date formatting anomalies when returning GeoJSON from 
MapServer. It might be more of a GDAL issue than MapServer. 

Using MapServer from the main branch (7-7) and gdal302.dll - the exact same 
MapServer setup returns dates with different formatting dependent on the 
machine. 

Dates on a local machine have T, dates on the server do not:

- Windows 10 local: "LastUpdatedDateUtc": "2021-04-20T10:33:31.626"
- Windows 2019 Server: "LastUpdatedDateUtc": "2021-04-20 10:33:31.6259376" 

Data is coming from the MSSQL Driver and database requests are identical. WFS 
requests are also identical for the test. 

OUTPUTFORMAT block below:

OUTPUTFORMAT
NAME "geojson"
DRIVER "OGR/GEOJSON"
MIMETYPE "application/json; subtype=geojson; charset=utf-8"
FORMATOPTION "FORM=SIMPLE"
FORMATOPTION "STORAGE=memory"
FORMATOPTION "LCO:NATIVE_MEDIA_TYPE=application/vnd.geo+json"
END

Has anyone else experienced this? Does a machine locale affect date formats?


Seth
--
web:http://geographika.co.uk
twitter: @geographika
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] show cluster region?

2021-04-23 Thread WhereGroup

Hi,

is it possible to visualize the cluster regions of a clustered layer?
Aim is to get a visual information about which points are clustered to
which cluster-point.

regards,
Jörg

-- 
 
 Aufwind durch Wissen!
 Web-Seminare und Online-Schulungen
 bei der www.foss-academy.com
 



Jörg Thomsen
WhereGroup GmbH
Bundesallee 23
10717 Berlin
Germany

Fon: +49 (0)30 / 5130 278 74
Fax: +49 (0)30 / 5130 278 11

joerg.thom...@wheregroup.com
www.wheregroup.com

Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885

---
Folgen Sie der WhereGroup auf twitter: http://twitter.com/WhereGroup_com
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Mapserver installation in cloud environments (kubernetes)

Re: [mapserver-users] Mapserver installation in cloud environments (kubernetes)

Re: [mapserver-users] Mapserver installation in cloud environments (kubernetes)

Re: [mapserver-users] Mapserver installation in cloud environments (kubernetes)

[mapserver-users] Mapserver installation in cloud environments (kubernetes)

[mapserver-users] GeoJSON Date Anomalies

[mapserver-users] show cluster region?

7 matches

Site Navigation

Mail list logo

Footer information