Re: [mapserver-users] Hello from Finland

[Søren Laursen via mapserver-users]

Hi Jukka,


You might want to look at:

https://www.pydap.org/en/latest/index.html


It is the backend of ifm.fcoo.dk using netcdf files. grib can be converted to 
netcdf.


Søren


FORSVARSMINISTERIETS MATERIEL- OG INDKØBSSTYRELSE

DANISH MINISTRY OF DEFENCE ACQUISITION AND LOGISTICS ORGANISATION


Joint GEOMETOC Support Centre

Lautrupbjerg 1-5, DK-2750 Ballerup

TLF: +45 7281 5627
MOBIL: +45 2961 9652

E-mail: fmi-ma-op...@mil.dk

www.fmi.dk





Fra: mapserver-users  på vegne af 
Rahkonen Jukka (MML) 
Sendt: 16. marts 2021 11:54:13
Til: Stefan Gofferje; mapserver-users@lists.osgeo.org
Emne: Re: [mapserver-users] Hello from Finland

Hi Stefan,

When it comes to memory usage with raster files, Mapserver is reading rasters 
with GDAL so in this case with https://gdal.org/drivers/raster/grib.html. I 
have never been playing with GRIB files and I do not know how well the format 
is optimized for features that are needed for making fast map services: fast 
access to subregions and overview levels. I have no idea how well GRIB files 
are indexed for random access, it there is difference between png and JPEG2000 
encoding, and how well the GDAL GRIB driver can utilize the possibilities that 
the format offer.

The most common raster format that Mapserver services are using is certainly 
GeoTIFF and when they are tiled, compressed, and have overviews they work 
pretty well. It is certainly an advantage if you can use the GRIB files 
directly without conversions. I see that you are using Mapcache in front of 
Mapserver and it will help a lot if you pre-seed the tiles. If you do not 
pre-seed then each time when the user is the first visitor on some site the 
bottle neck will be Mapserver.

The Ls/Hs idea is interesting. I do not believe there is anything ready 
available but I know that my colleagues at the Finnish Geodetic Institute have 
made quite nice things with Mapserver and shell scripts running some GDAL 
commands. Should they be computed on-the-fly for the area that is visible on 
the map?

I wonder if it could be possible to query the CONTOUR layer that you already 
has https://www.mapserver.org/input/vector/contour.html and search the min/max 
values from there. Or then to improve the contouring algorithm a bit because it 
is already processing all the pixels.

-Jukka Rahkonen-

-Alkuperäinen viesti-

Lähettäjä: mapserver-users  Puolesta 
Stefan Gofferje
Lähetetty: tiistai 16. maaliskuuta 2021 12.11
Vastaanottaja: mapserver-users@lists.osgeo.org
Aihe: Re: [mapserver-users] Hello from Finland

Hi Jeff,

On 3/15/21 2:35 PM, Jeff McKenna wrote:
 > Thanks Stefan for this wonderful introduction, it's nice to hear of  > 
 > positive experiences to give us all more motivation to continue to  > 
 > helping to improve MapServer!  Welcome to both you and Iliya to the  > 
 > community.  Keep us posted on your activities!
 >
 > Oh, once your MapServer backend is plugged-in be sure to add your  > weather 
 > site to the MapServer gallery, so we can help new users see  > what can be 
 > done...

Thanks for your nice words! I now have mapserver and mapcache running 
successfully and talking to each other in my home lab. I have set up a number 
of test layers from different input sources pretty successfully.
The documentation and the configuration required some adaption of my thinking 
processes because it's pretty different from anything else I have ever done 
before but I love this kind of challenges!

Currently I have only two issues which prevent me from doing the swap right 
now. Thanks to NOAA using a 0-360° grid instead of (-180)-180°, there's some 
artefacts when reprojecting to epsg:3857. For raster maps this is already 
addressed in #6106. I ran into an issue which LINE maps with a CONTOUR source 
which probably is related and opened #6261.

Then I have a question... Although it's not strictly necessary, I would like to 
paint those Hs and Ls in my atmospheric pressure map. Is there a way to run 
scripts or something like that in mapserver. I could probably run some GDAL 
stuff externally over the grid files in my download script and create some 
point overlays which could be in a GROUP with the isoline layer but doing this 
inside of mapserver would kinda be cooler.

And a second question: How is mapserver handling raster files memory-wise? Does 
it try to load the whole file to the memory at once?
My biggest issue during the whole 14 years I run this website always has been 
balancing preprocessing and map rendering. The more preprocessing I have to do, 
the longer the whole process of map generation takes but of course I couldn't 
afford to buy a $100k top-of-the-line server with terabytes of RAM and plenty 
of CPU cores for the rendering...
I'm thinking about my input file strategy at the moment...

Re: [mapserver-users] Forward time and other parameters to a layer defined as wms client

[Søren Laursen]

That is a very nice and simple solution. Already startet to code and put it in 
the staging enviroment.

I will put our code on gitlab when we have something that works to create the 
files etc.

Best regards

Søren



Joint GeoMETOC Support Center

Lautrupbjerg 1-5 - 2750 Ballerup
Danish Defence Acquisition and Logistics Organisation
TLF: +45 7281 5627
MOBIL: +45 2961 9652
FIIN: FMI-MA-OPD07
http://WWW.FCOO.DK/


Fra: tron...@main.crusaders.no [tron...@main.crusaders.no] p vegne af 
Trond Michelsen [trondmm-mapserver+2...@crusaders.no]
Sendt: 8. august 2019 12:59
Til: Søren Laursen; Mapserver
Emne: Re: [mapserver-users] Forward time and other parameters to a layer 
defined as wms client

On Thu, Aug 08, 2019 at 10:12:58AM +, Søren Laursen wrote:
> Right now I am hacking some code to get GetCapabilities to work in
> our client, because we need some information about the timespan in
> the layer, which mapserver is not forwarding or know any information
> about.

I've made an outputfilter that does the same thing :)

But, you can do it with the regular mapfile, too. I'm using an include
file, so I don't need to rewrite the full mapfile repeatedly.

  METADATA
"wms_title"  "Radar Nowcast"
"wms_extent" "-79 754 759000 9398000"
"wms_srs""EPSG:3857 EPSG:3575 EPSG:4326 EPSG:32633 EPSG:32761 
EPSG:25833 CRS:84"
INCLUDE "/var/tmp/timeextent.txt"
  END

Then timeextent.txt looks like this:

--8<--
"wms_dimensionlist" "modeltime"
"wms_modeltime_item""foo"
"wms_modeltime_extent"  "2019-08-08T10:45:00Z"
"wms_modeltime_default" "2019-08-08T10:45:00Z"
"wms_modeltime_units"   "ISO8601"
"wms_timeextent" 
"2019-08-08T10:45:00Z,2019-08-08T10:50:00Z,2019-08-08T10:55:00Z,2019-08-08T11:00:00Z,2019-08-08T11:05:00Z,2019-08-08T11:10:00Z,2019-08-08T11:15:00Z,2019-08-08T11:20:00Z,2019-08-08T11:25:00Z,2019-08-08T11:30:00Z,2019-08-08T11:35:00Z,2019-08-08T11:40:00Z,2019-08-08T11:45:00Z,2019-08-08T11:50:00Z,2019-08-08T11:55:00Z,2019-08-08T12:00:00Z,2019-08-08T12:05:00Z,2019-08-08T12:10:00Z,2019-08-08T12:15:00Z,2019-08-08T12:20:00Z,2019-08-08T12:25:00Z,2019-08-08T12:30:00Z,2019-08-08T12:35:00Z,2019-08-08T12:40:00Z"
--8<--


This results in this capabilities-output

--8<--
2019-08-08T10:45:00Z,2019-08-08T10:50:00Z,2019-08-08T10:55:00Z,2019-08-08T11:00:00Z,2019-08-08T11:05:00Z,2019-08-08T11:10:00Z,2019-08-08T11:15:00Z,2019-08-08T11:20:00Z,2019-08-08T11:25:00Z,2019-08-08T11:30:00Z,2019-08-08T11:35:00Z,2019-08-08T11:40:00Z,2019-08-08T11:45:00Z,2019-08-08T11:50:00Z,2019-08-08T11:55:00Z,2019-08-08T12:00:00Z,2019-08-08T12:05:00Z,2019-08-08T12:10:00Z,2019-08-08T12:15:00Z,2019-08-08T12:20:00Z,2019-08-08T12:25:00Z,2019-08-08T12:30:00Z,2019-08-08T12:35:00Z,2019-08-08T12:40:00Z
2019-08-08T10:45:00Z
--8<--

Just make sure replacing the old timeextent.txt is an atomic
operation. Otherwise you risk requests coming in just when you've
deleted the old, and before the new is in place. That will cause a 500
Internal Server Error.

--
Trond Michelsen
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Forward time and other parameters to a layer defined as wms client

[Søren Laursen]

Yep,

Got something working now with runtime substitution. Will post a minimum 
configuration that works, need some clean up, and have to verify how validate 
works, because you need it in the layer definition.

Right now I am hacking some code to get GetCapabilities to work in our client, 
because we need some information about the timespan in the layer, which 
mapserver is not forwarding or know any information about.

Søreb

Joint GeoMETOC Support Center

Lautrupbjerg 1-5 - 2750 Ballerup
Danish Defence Acquisition and Logistics Organisation
TLF: +45 7281 5627
MOBIL: +45 2961 9652
FIIN: FMI-MA-OPD07
http://WWW.FCOO.DK/


Fra: mapserver-users [mapserver-users-boun...@lists.osgeo.org] p vegne af 
Trond Michelsen [trondmm-mapserver+2...@crusaders.no]
Sendt: 8. august 2019 11:34
Til: Seth G; Mapserver
Emne: Re: [mapserver-users] Forward time and other parameters to a layer 
defined as wms client

You're right, Runtime substitution is possible, but there were a
couple of issues.

One of them is probably not relevant for Søren. Different layers could
use different dimensions, like pressure and temperature. So we needed
something that would pass all possible dimensions to the backend. They
_could_ be hardcoded on a layer to layer basis, but it simplified the
setup if we could use a single connection string for all layers.

The second was that I couldn't find a way to make the runtime
substitution case insensitive. Using %TIME% in the mapfile won't
substitute anything if the request says =...

The third issue we had at the time was that we needed default values,
but I see that this has been fixed a long time ago.

On Thu, Aug 08, 2019 at 08:56:13AM +0200, Seth G wrote:
> A related approach to the one suggested by Trond is the one
> mentioned in https://github.com/mapserver/mapserver/issues/4797

> MAP->WEB: METADATA (values only, not keys) are supported as runtime
> substitution parameters - see
> https://mapserver.org/cgi/runsub.html#parameters-supported

> So if hardcoding wms_time works then in theory this could be
> replaced with a dynamic value from a URL.  I'm unsure if TIME would
> need to be added to the VALIDATION block as it is already a standard
> WMS parameter.

> On Thu, Aug 8, 2019, at 2:19 AM, Trond Michelsen wrote:
>> I made a workaround for this a while ago (I think we were using
>> Mapserver 4.10 at the time). It's possible there are better ways to
>> solve this now, but it still works :)
>>
>> I can give more details tomorrow, but the gist of it is that I've made
>> an Apache handler (mod_perl) that modifies the incoming request, and
>> adds the TIME, LEVEL and all DIM_* parameters to an environment
>> variable I've called MS_PASSTHROUGH, and then I add %MS_PASSTHROUGH%&
>> to the connection parameter in the layer configuration.
>>
>> I can send you a copy of my Apache handler and config tomorrow, if you
>> like.
>>
>> On Wed, Aug 07, 2019 at 05:49:20PM +, Søren Laursen wrote:
>>> We have been using mapserver to display static layers, and are now
>>> in progress of using it to display dynamic (time bound) layers.
>>
>>> We have a wms server that serve forecast, and we use a shape file to
>>> mask out areas, for example cut a forecast so it follow landmass.
>>
>>> This works perfect, but when we want to include the TIME parameter
>>> to show a forecast at a specific point in time, we cannot get
>>> mapserver to forward the parameter.
>>
>>> Later on we will have the same problem with forecast that are bound
>>> by for example attitude and time.
>>
>>> We are using mapserver with apache, and in production we have a
>>> varnish in front (and banning urls when new forecast are ready).
>>
>>> I have look at time documentation of both the time wms settings and
>>> the runtime subtitionen but cannot figure out how we can make a
>>> solution.
>>
>>> When we hardcode a wms_time parameter in the layer definition
>>> everything works.

--
Trond Michelsen
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Forward time and other parameters to a layer defined as wms client

[Søren Laursen]

Yes, this is how the client is defined:

 LAYER
CONNECTION "http://wms-server/dataset.wms?;
CONNECTIONTYPE WMS
DEBUG 4
METADATA
  "wms_srs" "EPSG:3857"
  "wms_exceptions_format"   "XML"
  "wms_cmap""Current_cms_WGYR_11colors_1.0"
  "wms_transparent" "TRUE"
  "wms_name""UCUR:VCUR"
  "wms_timeitem" "TIME"
#  "wms_timedefault""2019-06-25T12:00:00.000Z"
#  "wms_time" "2019-06-25T12:00:00Z"
#  "wms_timeformat" "-MM-DDTHH:MM:SSZ"
  "wms_format"  "image/png"
  "wms_server_version"  "1.3.0"
  "wms_style"   
"plot_method=color_quiver2;legend=Current_cms_WGYR_11colors_1.0"
  "wms_crs" "EPSG:3857"
  "wms_enable_request" "*"
END # METADATA
NAME "Current"
PROCESSING "CLOSE_CONNECTION=NORMAL"
PROCESSING "LOAD_WHOLE_IMAGE=YES"
PROJECTION
  "init=epsg:3857"
END # PROJECTION
STATUS ON
TILEITEM "location"
TYPE RASTER
UNITS METERS
MASK "waterpolygons"
  END # LAYER


Joint GeoMETOC Support Center
[FMI logo]
Lautrupbjerg 1-5 - 2750 Ballerup
Danish Defence Acquisition and Logistics Organisation
TLF: +45 7281 5627
MOBIL: +45 2961 9652
FIIN: FMI-MA-OPD07
http://WWW.FCOO.DK/

Fra: mapserver-users [mapserver-users-boun...@lists.osgeo.org] på vegne af Seth 
G [se...@geographika.co.uk]
Sendt: 7. august 2019 21:00
Til: mapserver-users@lists.osgeo.org
Emne: Re: [mapserver-users] Forward time and other parameters to a layer 
defined as wms client

Hi Søren,

Just to clarify you are using MapServer as a WMS Client as outlined here 
https://mapserver.org/ogc/wms_client.html
With  "CONNECTIONTYPE WMS"?

Seth

--
web:http://geographika.co.uk
twitter: @geographika


On Wed, Aug 7, 2019, at 7:55 PM, Søren Laursen wrote:
Hi,

We have been using mapserver to display static layers, and are now in progress 
of using it to display dynamic (time bound) layers.

We have a wms server that serve forecast, and we use a shape file to mask out 
areas, for example cut a forecast so it follow landmass.

This works perfect, but when we want to include the TIME parameter to show a 
forecast at a specific point in time, we cannot get mapserver to forward the 
parameter.

Later on we will have the same problem with forecast that are bound by for 
example attitude and time.

We are using mapserver with apache, and in production we have a varnish in 
front (and banning urls when new forecast are ready).

I have look at time documentation of both the time wms settings and the runtime 
subtitionen but cannot figure out how we can make a solution.

When we hardcode a wms_time parameter in the layer definition everything works.

Best regards

Søren




Joint GeoMETOC Support Center
[FMI logo]
Lautrupbjerg 1-5 - 2750 Ballerup
Danish Defence Acquisition and Logistics Organisation
TLF: +45 7281 5627
FIIN: FMI-MA-OPD07
http://WWW.FCOO.DK/
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Forward time and other parameters to a layer defined as wms client

[Søren Laursen]

Hi,

We have been using mapserver to display static layers, and are now in progress 
of using it to display dynamic (time bound) layers.

We have a wms server that serve forecast, and we use a shape file to mask out 
areas, for example cut a forecast so it follow landmass.

This works perfect, but when we want to include the TIME parameter to show a 
forecast at a specific point in time, we cannot get mapserver to forward the 
parameter.

Later on we will have the same problem with forecast that are bound by for 
example attitude and time.

We are using mapserver with apache, and in production we have a varnish in 
front (and banning urls when new forecast are ready).

I have look at time documentation of both the time wms settings and the runtime 
subtitionen but cannot figure out how we can make a solution.

When we hardcode a wms_time parameter in the layer definition everything works.

Best regards

Søren




Joint GeoMETOC Support Center
[FMI logo]
Lautrupbjerg 1-5 - 2750 Ballerup
Danish Defence Acquisition and Logistics Organisation
TLF: +45 7281 5627
FIIN: FMI-MA-OPD07
http://WWW.FCOO.DK/
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users