Re: [Geoserver-users] Best practice for large amounts of orthophotos

[Andrew Miskelly]

Hi Mats,

One option is to make some simple modifications to the geotiffs as you
receive them. A single invocation of gdal_translate could introduce
compression, which would reduce the file size significantly, and tiling,
which would improve performance. If in doubt, use a couple of invocations
and turn them into cloud optimised geotiffs. This will ensure you get good
overall performance (and, subsequently, you'll have the option of serving
them from cloud storage). See
https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF#HowtogenerateitwithGDAL
.

Then you could use the approach with ImageMosaic you outlined above.

To optimise the files you'd need to set up whatever Windows' equivalent of
a cron job is - something which periodically lists files in the input
directory, processes them, outputs the results to the directory from which
the files will be served and cleans up the original.

Regards,

Andrew

On Fri, 11 Oct 2019 at 20:33, Mats Elfström 
wrote:

> Hi all!
>
> I would like to discuss best practice for serving rather large amounts of
> aerial imagery (ie orthophotos). My setup is Geoserver 2.16 on a local
> Windows Server. I also have PostgreSQL 10.5/Postgis 2.4.4 running on the
> same server, but that is not used for imagery at this point.
>
> Imagery is stored in uncompressed geotifs, ~300MB each, and there are
> 150-250 of these per year so it adds up to hundreds of GB’s. Disk space on
> the server is already an issue but can be augmented. I have FME and lots of
> experience working with aerial imagery, but less experience from imagery on
> Geoserver.
>
> I have figured out and tested three approaches and will list them with the
> pros and cons I have found so far. They are not in order of preference or
> anything. Less disk space is more important than necessary bandwidth, but
> overall performance is important.
>
>  1/ Use ImageMosaic and serve one folder of tiffs per year.
>
> Pros: No file processing necessary, deliveries can simply be dumped in a
> folder. Folder doubles as data storage if original files need to be
> retrieved. Original resolution is unchanged.
>
> Cons: Will need huge amounts of disk space. Despite some tiles tweaking,
> very slow performance, esp at small scales.
>
> 2/ Mosaic the image files into one ECW file with the original resolution
> and serve that from an ECW store.
>
> Pros: Will take up significantly (1/5-1/10) less disk space. Very fast
> performance, regardless of scale.
>
> Cons: Time consuming file processing, but a one-time job. Original files
> will need storage elsewhere. Possible licensing limitations on the ecw
> software.
>
> 3/ Mosaic the image files into one JP2k file and serve that from a JP2ECW
> store.
>
> Pros: Will take up much less disk space, but more than ECW. No licensing
> issues.
>
> Cons: Time consuming file processing, but a one-time job. Original files
> need storage elsewhere. Slightly slower performance than ECW.
>
> In a case like this, what is a generally agreed best practice? All hints
> and suggestions are most welcome. Maybe there are better alternatives than
> the above? PostGIS raster storage maybe?
>
> Thanks in advance and best regards, Mats.E
>
>
>
> *GisKraft*, GIS och webbkonsult
> Mats Elfström, Väpplingvägen 21, SE-227 38 LUND, Sweden
> tel: +46 70 595 39 35 / web: www.giskraft.com
> ___
> Geoserver-users mailing list
>
> Please make sure you read the following two resources before posting to
> this list:
> - Earning your support instead of buying it, but Ian Turton:
> http://www.ianturton.com/talks/foss4g.html#/
> - The GeoServer user list posting guidelines:
> http://geoserver.org/comm/userlist-guidelines.html
>
> If you want to request a feature or an improvement, also see this:
> https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer
>
>
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>


-- 
*Andrew Miskelly*
Senior Developer
*T *  +61 2 9965 9200
*E   *amiske...@weatherzone.com.au



www.weatherzone.com.au | business.weatherzone.com.au
supp...@weatherzone.com.au | sa...@weatherzone.com.au

  
___
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this 
list:
- Earning your support instead of buying it, but Ian Turton: 
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: 
http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: 
https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Re: [Geoserver-users] Best practice for large amounts of orthophotos

[Andrea Aime]

On Fri, Oct 11, 2019 at 11:40 AM Mats Elfström 
wrote:

> 2/ Mosaic the image files into one ECW file with the original resolution
> and serve that from an ECW store.
>
> Pros: Will take up significantly (1/5-1/10) less disk space. Very fast
> performance, regardless of scale.
>
> Cons: Time consuming file processing, but a one-time job. Original files
> will need storage elsewhere. Possible licensing limitations on the ecw
> software.
>
> 3/ Mosaic the image files into one JP2k file and serve that from a JP2ECW
> store.
>
> Pros: Will take up much less disk space, but more than ECW. No licensing
> issues.
>

I think one has to pay a license any time the ECW libraries are used server
side, no matter what format
is being read using these libraries...  I'm not sure, but I'd warmly
suggest to check with Hexagon.

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information. == Ing. Andrea Aime @geowolf Technical Lead
GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39
0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549
http://www.geo-solutions.it http://twitter.com/geosolutions_it
--- *Con riferimento
alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 -
Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni
circostanza inerente alla presente email (il suo contenuto, gli eventuali
allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i
destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per
errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le
sarei comunque grato se potesse darmene notizia. This email is intended
only for the person or entity to which it is addressed and may contain
information that is privileged, confidential or otherwise protected from
disclosure. We remind that - as provided by European Regulation 2016/679
“GDPR” - copying, dissemination or use of this e-mail or the information
herein by anyone other than the intended recipient is prohibited. If you
have received this email by mistake, please notify us immediately by
telephone or e-mail.*
___
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this 
list:
- Earning your support instead of buying it, but Ian Turton: 
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: 
http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: 
https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Re: [Geoserver-users] Best practice for large amounts of orthophotos

[Daniele Romagnoli]

Dear Mats,
Did you consider reprocessing your files as GeoTIFF with internal JPEG
compression?
At the end, if you are not against reprocessing your files, that could be
an additional option.

By this way, you can:
- re-layout your data as you prefer, by internally retiling your GeoTIFFs
for a quicker access, and optionally aggregate multiple files into BigTIFFs
if you aim to have a file per year.
- add internal JPEG compression to tiles in order to heavily reduce disk
usage. (you can play with quality parameters to setup the desired trade-off
between file size and visual quality)
- add internal overviews for fast performances at other scales
You will also get rid of any licensing concern and third party library
installation/compatibility potential issue.

Moreover, depending on how many "files" you need to access on a single
view, you may also consider pyramid.
Suppose your files compose a whole country and the client will allow to
both zoom in to cities details as well as zoom out to continents or whole
world. In the latter case (zooming out to whole world), having a single
file to be opened instead of 1000 files with their internal overviews to be
accessed to compose that world view could be better (so the top level of a
pyramid OR a BigTIFF).

Regards,
Daniele







On Fri, Oct 11, 2019 at 11:40 AM Mats Elfström 
wrote:

> Hi all!
>
> I would like to discuss best practice for serving rather large amounts of
> aerial imagery (ie orthophotos). My setup is Geoserver 2.16 on a local
> Windows Server. I also have PostgreSQL 10.5/Postgis 2.4.4 running on the
> same server, but that is not used for imagery at this point.
>
> Imagery is stored in uncompressed geotifs, ~300MB each, and there are
> 150-250 of these per year so it adds up to hundreds of GB’s. Disk space on
> the server is already an issue but can be augmented. I have FME and lots of
> experience working with aerial imagery, but less experience from imagery on
> Geoserver.
>
> I have figured out and tested three approaches and will list them with the
> pros and cons I have found so far. They are not in order of preference or
> anything. Less disk space is more important than necessary bandwidth, but
> overall performance is important.
>
>  1/ Use ImageMosaic and serve one folder of tiffs per year.
>
> Pros: No file processing necessary, deliveries can simply be dumped in a
> folder. Folder doubles as data storage if original files need to be
> retrieved. Original resolution is unchanged.
>
> Cons: Will need huge amounts of disk space. Despite some tiles tweaking,
> very slow performance, esp at small scales.
>
> 2/ Mosaic the image files into one ECW file with the original resolution
> and serve that from an ECW store.
>
> Pros: Will take up significantly (1/5-1/10) less disk space. Very fast
> performance, regardless of scale.
>
> Cons: Time consuming file processing, but a one-time job. Original files
> will need storage elsewhere. Possible licensing limitations on the ecw
> software.
>
> 3/ Mosaic the image files into one JP2k file and serve that from a JP2ECW
> store.
>
> Pros: Will take up much less disk space, but more than ECW. No licensing
> issues.
>
> Cons: Time consuming file processing, but a one-time job. Original files
> need storage elsewhere. Slightly slower performance than ECW.
>
> In a case like this, what is a generally agreed best practice? All hints
> and suggestions are most welcome. Maybe there are better alternatives than
> the above? PostGIS raster storage maybe?
>
> Thanks in advance and best regards, Mats.E
>
>
>
> *GisKraft*, GIS och webbkonsult
> Mats Elfström, Väpplingvägen 21, SE-227 38 LUND, Sweden
> tel: +46 70 595 39 35 / web: www.giskraft.com
> ___
> Geoserver-users mailing list
>
> Please make sure you read the following two resources before posting to
> this list:
> - Earning your support instead of buying it, but Ian Turton:
> http://www.ianturton.com/talks/foss4g.html#/
> - The GeoServer user list posting guidelines:
> http://geoserver.org/comm/userlist-guidelines.html
>
> If you want to request a feature or an improvement, also see this:
> https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer
>
>
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>


-- 
Regards,
Daniele Romagnoli
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:  +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE
2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si
precisa che ogni circostanza 

Re: [Geoserver-users] Best practice for large amounts of orthophotos

[Nikola Jankovic]

Hi Mats,

I'm dealing with similar stuff, and I will drop a couple of links below for 
reference.

In case of 1. consider adding tiling and inner overviews for each of your 
images with gdal_translate & gdaladdo (also look up cloud optimized geotiffs), 
and storing them in a CRS that will mostly be requested (less reprojection 
overhead). If the data is meant for visualization also consider lossy 
compression to save space. If you need to serve raw data with WCS, go with some 
lossless compression at a lower level. Although this might mean getting read of 
the original data or duplicating data, and reprojecting also causes data loss 
(which I believe for orthophotos should be negligible).

For 2 & 3 I cannot speak since I need to deal with lots of files without too 
much influence on them so I didn't do research there.

https://www.cogeo.org/ - for http range requests but i notice also in general 
good for data optimization.
https://geoserver.geo-solutions.it/edu/en/enterprise/raster.html - skim through 
here for some good reference on tweaking/optimization/caching etc.

Cheers,
Nikola

From: Mats Elfström 
Sent: 11 October 2019 11:32
To: geoserver-users@lists.sourceforge.net 

Subject: [Geoserver-users] Best practice for large amounts of orthophotos


Hi all!

I would like to discuss best practice for serving rather large amounts of 
aerial imagery (ie orthophotos). My setup is Geoserver 2.16 on a local Windows 
Server. I also have PostgreSQL 10.5/Postgis 2.4.4 running on the same server, 
but that is not used for imagery at this point.

Imagery is stored in uncompressed geotifs, ~300MB each, and there are 150-250 
of these per year so it adds up to hundreds of GB’s. Disk space on the server 
is already an issue but can be augmented. I have FME and lots of experience 
working with aerial imagery, but less experience from imagery on Geoserver.

I have figured out and tested three approaches and will list them with the pros 
and cons I have found so far. They are not in order of preference or anything. 
Less disk space is more important than necessary bandwidth, but overall 
performance is important.

 1/ Use ImageMosaic and serve one folder of tiffs per year.

Pros: No file processing necessary, deliveries can simply be dumped in a 
folder. Folder doubles as data storage if original files need to be retrieved. 
Original resolution is unchanged.

Cons: Will need huge amounts of disk space. Despite some tiles tweaking, very 
slow performance, esp at small scales.

2/ Mosaic the image files into one ECW file with the original resolution and 
serve that from an ECW store.

Pros: Will take up significantly (1/5-1/10) less disk space. Very fast 
performance, regardless of scale.

Cons: Time consuming file processing, but a one-time job. Original files will 
need storage elsewhere. Possible licensing limitations on the ecw software.

3/ Mosaic the image files into one JP2k file and serve that from a JP2ECW store.

Pros: Will take up much less disk space, but more than ECW. No licensing issues.

Cons: Time consuming file processing, but a one-time job. Original files need 
storage elsewhere. Slightly slower performance than ECW.

In a case like this, what is a generally agreed best practice? All hints and 
suggestions are most welcome. Maybe there are better alternatives than the 
above? PostGIS raster storage maybe?

Thanks in advance and best regards, Mats.E

 
[https://docs.google.com/uc?export=download=0B6X-OqqLVTjPM2pwU05zTXNZWkU=0B6X-OqqLVTjPRzhtZUZyMDg3eFBoejRubGlkaGx3bVhNTHJrPQ]

GisKraft, GIS och webbkonsult
Mats Elfström, Väpplingvägen 21, SE-227 38 LUND, Sweden
tel: +46 70 595 39 35 / web: www.giskraft.com
___
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this 
list:
- Earning your support instead of buying it, but Ian Turton: 
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: 
http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: 
https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users