Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

[Rahkonen Jukka]

Hi,

Mapserver is not clever at all when it finds many overlapping images. It 
renders them all one by one. If the list of images is sorted correctly then the 
end result is what was desired, but with all too much work. See 
https://github.com/MapServer/MapServer/issues/5899.
We use two tricks:
1) We have computed the time range of validity for each image and we save the 
start data and end date into two attributes in the tileindex table in a 
database. Start date is the acquisition date of the image, and end date is the 
date when the last pixel of the image gets covered by some newer image. If the 
requested time is not within the range of validity then the image is skipped. 
Skipping is based on the SQL query that is used as DATA for the tileindex layer.
2) Because time is not supported as a dimension we have written a small utility 
that is manipulating the incoming http GetCoverage requests before they are 
sent to Mapserver. The utility converts the time subset, for example 
SUBSET=time("2012-12-31T02:00:00Z") into something like 
"_date<2012-12-31T02:00:00Z_date>2012-12-31T02:00:00Z. The start_date 
and end_date variables are defined and validated in the mapfile.

As a result Mapserver usually does not get overlapping images from the 
tileindex so it does not need to spend time for multiple rendering either. We 
have typically about 10 aerial photos from different times at each place and 
without the above mentioned tweaks the service was all too slow.

-Jukka Rahkonen-

Lähettäjä: MapServer-users  Puolesta 
Marcin Niemyjski via MapServer-users
Lähetetty: torstai 27. heinäkuuta 2023 12.26
Vastaanottaja: MapServer Users 
Aihe: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS 
service?

Hello Jukka,

Yup I guess that's the case, I assumed that WCS has the same capabilities as 
WMS when it comes to handling time dimension and it would use the default time 
from mapfile while requesting the data.
This is my mapserv log file from one getCoverage request (very small, that's 
why It took only 30 sec to process it). It would be great if you check it :
https://s3.waw3-1.cloudferro.com/swift/v1/Julien/log2.txt

as you can see mapserver lists all of tile item objects (or at least I guess it 
does haha) which intersect with requested BBOX. So, this is the problem, that's 
why it takes so long to generate file. I've got to pass somehow time parameter 
to request, or at least to query(?) to reduce number of returning shapes.

You mentioned your own supporting time WCS service, is it open source? If not, 
can you give me advice on how to start writing my own? Just some basics ^^

Thanks for your complex answer it helped a lot,
Marcin



From: Rahkonen Jukka 
mailto:jukka.rahko...@maanmittauslaitos.fi>>
Sent: Thursday, July 27, 2023 12:05 AM
To: Marcin Niemyjski 
mailto:mniemyj...@cloudferro.com>>; MapServer Users 
mailto:mapserver-users@lists.osgeo.org>>
Subject: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up 
WCS service?


Hi,



Does the tileindex query ” DATA 'geometry from (select * from plua order by 
"maxCC" desc) “ return several images from the requested BBOX?



I know that MapServer WCS can be fast and we have also a demo at 
https://beta-karttakuva.maanmittauslaitos.fi/demo/ortokuvat_ja_korkeusmallit/#15.05/67.01553/27.16873

but unfortunately the user interface is only in Finnish. The source images 
(maybe 10 TB or so) are GeoTIFFs and they are not in the cloud. The time for 
downloading a small GetCoverage response as GeoTIFF seems to take around 400 
milliseconds. We had to write a small service that makes our WCS service to 
support time as a dimension because MapServer does not support it with WCS 
https://www.mail-archive.com/mapserver-users@lists.osgeo.org/msg26265.html.



-Jukka Rahkonen-





Lähettäjä: MapServer-users 
mailto:mapserver-users-boun...@lists.osgeo.org>>
 Puolesta Marcin Niemyjski via MapServer-users
Lähetetty: keskiviikko 26. heinäkuuta 2023 15.06
Vastaanottaja: MapServer Users 
mailto:mapserver-users@lists.osgeo.org>>
Aihe: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS 
service?



Hi Seth,



as you suggested, I've added the Extent parameter to all of my layers, and it 
resulted in decreasing time to 5m33.534s. But still, it's quite a lot of time 
for generating 1.5 MiB file.



Any other suggestions?
Maybe there is something wrong with projection so beyond just serving data 
mapserver have to reproject it multiple times? (more than one time by using 
tilesrs option)?



Thanks,

Marcin





From: Seth G mailto:se...@geographika.co.uk>>
Sent: Wednesday, July 26, 2023 12:58 PM
To: Marcin Niemyjski 
mailto:mniemyj...@cloudferro.com>>; MapServer Users 
mailto:mapserver-users@lists.osgeo.org>>
Subject: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up 
WCS service?



Hi Marcin,



As a first test I'd recommend adding in EXTENTs 

Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

[Marcin Niemyjski via MapServer-users]

Hello Jukka,

Yup I guess that's the case, I assumed that WCS has the same capabilities as 
WMS when it comes to handling time dimension and it would use the default time 
from mapfile while requesting the data.
This is my mapserv log file from one getCoverage request (very small, that's 
why It took only 30 sec to process it). It would be great if you check it :
https://s3.waw3-1.cloudferro.com/swift/v1/Julien/log2.txt

as you can see mapserver lists all of tile item objects (or at least I guess it 
does haha) which intersect with requested BBOX. So, this is the problem, that's 
why it takes so long to generate file. I've got to pass somehow time parameter 
to request, or at least to query(?) to reduce number of returning shapes.

You mentioned your own supporting time WCS service, is it open source? If not, 
can you give me advice on how to start writing my own? Just some basics ^^

Thanks for your complex answer it helped a lot,
Marcin







From: Rahkonen Jukka 
Sent: Thursday, July 27, 2023 12:05 AM
To: Marcin Niemyjski ; MapServer Users 

Subject: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up 
WCS service?


Hi,



Does the tileindex query ” DATA 'geometry from (select * from plua order by 
"maxCC" desc) “ return several images from the requested BBOX?



I know that MapServer WCS can be fast and we have also a demo at 
https://beta-karttakuva.maanmittauslaitos.fi/demo/ortokuvat_ja_korkeusmallit/#15.05/67.01553/27.16873

but unfortunately the user interface is only in Finnish. The source images 
(maybe 10 TB or so) are GeoTIFFs and they are not in the cloud. The time for 
downloading a small GetCoverage response as GeoTIFF seems to take around 400 
milliseconds. We had to write a small service that makes our WCS service to 
support time as a dimension because MapServer does not support it with WCS 
https://www.mail-archive.com/mapserver-users@lists.osgeo.org/msg26265.html.



-Jukka Rahkonen-





Lähettäjä: MapServer-users  Puolesta 
Marcin Niemyjski via MapServer-users
Lähetetty: keskiviikko 26. heinäkuuta 2023 15.06
Vastaanottaja: MapServer Users 
Aihe: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS 
service?



Hi Seth,



as you suggested, I've added the Extent parameter to all of my layers, and it 
resulted in decreasing time to 5m33.534s. But still, it's quite a lot of time 
for generating 1.5 MiB file.



Any other suggestions?
Maybe there is something wrong with projection so beyond just serving data 
mapserver have to reproject it multiple times? (more than one time by using 
tilesrs option)?



Thanks,

Marcin





From: Seth G mailto:se...@geographika.co.uk>>
Sent: Wednesday, July 26, 2023 12:58 PM
To: Marcin Niemyjski 
mailto:mniemyj...@cloudferro.com>>; MapServer Users 
mailto:mapserver-users@lists.osgeo.org>>
Subject: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up 
WCS service?



Hi Marcin,



As a first test I'd recommend adding in EXTENTs to all your layers (and 
ensuring they are correct):



EXTENT 1334552.857598 5342648.587634 4715044.821842 7547620.486088



If these are missing MapServer will often try and calculate these from the 
datasource automatically. Depending on the request type this calculation could 
be made multiple times.



Seth



--

web:https://geographika.net & 
https://mapserverstudio.net

twitter: @geographika



On Wed, Jul 26, 2023, at 11:53 AM, Marcin Niemyjski via MapServer-users wrote:

Hello,



I have a problem, Namely, I have created a WCS and here is its mapfile:

https://pastecode.io/s/6v3qfssb



For a better understanding, here is a view of the tileindex through which the 
data is served:

https://pasteboard.co/ydgvDiqxz4lf.png



I am connecting to files located in a private bucket, so I am using vsis3.



The WMS loads quite fast, but the issue arises with the time it takes to 
generate a requested WCS tile:

 https://pasteboard.co/8G8uF41jqgHc.png



It takes almost 5 minutes to generate the tile, and the received data is 1.5 
MiB (which is it's actual size).



Additionally, I am sending you URLs to sample data:

https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T34UFD_20230708T093549_TCI_60m.jp2

https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T37TCN_20220605T083559_TCI_60m.jp2



The case is - how to speed it up? Am I making any goofy mistakes?

I would be grateful for any recommendations or if you notice any errors, I 
might be making :)



Best,

Marcin

___

MapServer-users mailing list

MapServer-users@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/mapserver-users




___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

[Rahkonen Jukka]

Hi,

Does the tileindex query " DATA 'geometry from (select * from plua order by 
"maxCC" desc) " return several images from the requested BBOX?

I know that MapServer WCS can be fast and we have also a demo at 
https://beta-karttakuva.maanmittauslaitos.fi/demo/ortokuvat_ja_korkeusmallit/#15.05/67.01553/27.16873
but unfortunately the user interface is only in Finnish. The source images 
(maybe 10 TB or so) are GeoTIFFs and they are not in the cloud. The time for 
downloading a small GetCoverage response as GeoTIFF seems to take around 400 
milliseconds. We had to write a small service that makes our WCS service to 
support time as a dimension because MapServer does not support it with WCS 
https://www.mail-archive.com/mapserver-users@lists.osgeo.org/msg26265.html.

-Jukka Rahkonen-


Lähettäjä: MapServer-users  Puolesta 
Marcin Niemyjski via MapServer-users
Lähetetty: keskiviikko 26. heinäkuuta 2023 15.06
Vastaanottaja: MapServer Users 
Aihe: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS 
service?

Hi Seth,

as you suggested, I've added the Extent parameter to all of my layers, and it 
resulted in decreasing time to 5m33.534s. But still, it's quite a lot of time 
for generating 1.5 MiB file.

Any other suggestions?
Maybe there is something wrong with projection so beyond just serving data 
mapserver have to reproject it multiple times? (more than one time by using 
tilesrs option)?

Thanks,
Marcin


From: Seth G mailto:se...@geographika.co.uk>>
Sent: Wednesday, July 26, 2023 12:58 PM
To: Marcin Niemyjski 
mailto:mniemyj...@cloudferro.com>>; MapServer Users 
mailto:mapserver-users@lists.osgeo.org>>
Subject: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up 
WCS service?

Hi Marcin,

As a first test I'd recommend adding in EXTENTs to all your layers (and 
ensuring they are correct):

EXTENT 1334552.857598 5342648.587634 4715044.821842 7547620.486088

If these are missing MapServer will often try and calculate these from the 
datasource automatically. Depending on the request type this calculation could 
be made multiple times.

Seth

--
web:https://geographika.net & 
https://mapserverstudio.net
twitter: @geographika

On Wed, Jul 26, 2023, at 11:53 AM, Marcin Niemyjski via MapServer-users wrote:
Hello,

I have a problem, Namely, I have created a WCS and here is its mapfile:
https://pastecode.io/s/6v3qfssb

For a better understanding, here is a view of the tileindex through which the 
data is served:
https://pasteboard.co/ydgvDiqxz4lf.png

I am connecting to files located in a private bucket, so I am using vsis3.

The WMS loads quite fast, but the issue arises with the time it takes to 
generate a requested WCS tile:
 https://pasteboard.co/8G8uF41jqgHc.png

It takes almost 5 minutes to generate the tile, and the received data is 1.5 
MiB (which is it's actual size).

Additionally, I am sending you URLs to sample data:
https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T34UFD_20230708T093549_TCI_60m.jp2
https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T37TCN_20220605T083559_TCI_60m.jp2

The case is - how to speed it up? Am I making any goofy mistakes?
I would be grateful for any recommendations or if you notice any errors, I 
might be making :)

Best,
Marcin
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users


___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

[Marcin Niemyjski via MapServer-users]

Hi Seth,

as you suggested, I've added the Extent parameter to all of my layers, and it 
resulted in decreasing time to 5m33.534s. But still, it's quite a lot of time 
for generating 1.5 MiB file.

Any other suggestions?
Maybe there is something wrong with projection so beyond just serving data 
mapserver have to reproject it multiple times? (more than one time by using 
tilesrs option)?

Thanks,
Marcin





From: Seth G 
Sent: Wednesday, July 26, 2023 12:58 PM
To: Marcin Niemyjski ; MapServer Users 

Subject: Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up 
WCS service?

Hi Marcin,

As a first test I'd recommend adding in EXTENTs to all your layers (and 
ensuring they are correct):

EXTENT 1334552.857598 5342648.587634 4715044.821842 7547620.486088

If these are missing MapServer will often try and calculate these from the 
datasource automatically. Depending on the request type this calculation could 
be made multiple times.

Seth

--
web:https://geographika.net & https://mapserverstudio.net
twitter: @geographika

On Wed, Jul 26, 2023, at 11:53 AM, Marcin Niemyjski via MapServer-users wrote:
Hello,

I have a problem, Namely, I have created a WCS and here is its mapfile:
https://pastecode.io/s/6v3qfssb

For a better understanding, here is a view of the tileindex through which the 
data is served:
https://pasteboard.co/ydgvDiqxz4lf.png

I am connecting to files located in a private bucket, so I am using vsis3.

The WMS loads quite fast, but the issue arises with the time it takes to 
generate a requested WCS tile:
 https://pasteboard.co/8G8uF41jqgHc.png

It takes almost 5 minutes to generate the tile, and the received data is 1.5 
MiB (which is it's actual size).

Additionally, I am sending you URLs to sample data:
https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T34UFD_20230708T093549_TCI_60m.jp2
https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T37TCN_20220605T083559_TCI_60m.jp2

The case is - how to speed it up? Am I making any goofy mistakes?
I would be grateful for any recommendations or if you notice any errors, I 
might be making :)

Best,
Marcin
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users


___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

[Seth G]

Hi Marcin,

As a first test I'd recommend adding in EXTENTs to all your layers (and 
ensuring they are correct):

EXTENT 1334552.857598 5342648.587634 4715044.821842 7547620.486088

If these are missing MapServer will often try and calculate these from the 
datasource automatically. Depending on the request type this calculation could 
be made multiple times.

Seth

--
web:https://geographika.net & https://mapserverstudio.net
twitter: @geographika

On Wed, Jul 26, 2023, at 11:53 AM, Marcin Niemyjski via MapServer-users wrote:
> Hello,
> 
> I have a problem, Namely, I have created a WCS and here is its mapfile:
> https://pastecode.io/s/6v3qfssb
> 
> For a better understanding, here is a view of the tileindex through which the 
> data is served:
> https://pasteboard.co/ydgvDiqxz4lf.png
> 
> I am connecting to files located in a private bucket, so I am using vsis3.
> 
> The WMS loads quite fast, but the issue arises with the time it takes to 
> generate a requested WCS tile:
>  https://pasteboard.co/8G8uF41jqgHc.png
> 
> It takes almost 5 minutes to generate the tile, and the received data is 1.5 
> MiB (which is it's actual size).
> 
> Additionally, I am sending you URLs to sample data:
> https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T34UFD_20230708T093549_TCI_60m.jp2
> https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T37TCN_20220605T083559_TCI_60m.jp2
> 
> The case is - how to speed it up? Am I making any goofy mistakes?
> I would be grateful for any recommendations or if you notice any errors, I 
> might be making :)
> 
> Best,
> Marcin
> ___
> MapServer-users mailing list
> MapServer-users@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
> 
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users

[MapServer-users] Speed of WMS vs Speed of WCS; How to speed up WCS service?

[Marcin Niemyjski via MapServer-users]

Hello,

I have a problem, Namely, I have created a WCS and here is its mapfile:
https://pastecode.io/s/6v3qfssb

For a better understanding, here is a view of the tileindex through which the 
data is served:
https://pasteboard.co/ydgvDiqxz4lf.png

I am connecting to files located in a private bucket, so I am using vsis3.

The WMS loads quite fast, but the issue arises with the time it takes to 
generate a requested WCS tile:
 https://pasteboard.co/8G8uF41jqgHc.png

It takes almost 5 minutes to generate the tile, and the received data is 1.5 
MiB (which is it's actual size).

Additionally, I am sending you URLs to sample data:
https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T34UFD_20230708T093549_TCI_60m.jp2
https://s3.waw3-1.cloudferro.com/swift/v1/Julien/Data/T37TCN_20220605T083559_TCI_60m.jp2

The case is - how to speed it up? Am I making any goofy mistakes?
I would be grateful for any recommendations or if you notice any errors, I 
might be making :)

Best,
Marcin
___
MapServer-users mailing list
MapServer-users@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/mapserver-users