Hi! I recently faced a similar task. Hundreds of high resolution geotiffs over a limited area that I wanted to serve on Geoserver. This is 2.16 but I do not think that matters. I also have several image sets from different years. In all, this amounts to several tens of gigabytes. I tried to use image mosaic, but performance was horrible. A suggestion from this list was to degrade the tiffs and compress them further. This would effectively duplicate an already large file storage and I hesitated to do that. Also, I do not want the Geoserver machine to double as a file storage. I solved the task by mosaicking the yearly file sets into jpeg200, one file per year. These I then serve, one store and layer per year. No need for tiling either. That is taken care of in the file format. Normally I use ecw for purposes like these, but there is a licensing issue with ecw in a server environment. JPEG2000 is true open source, and is almost as efficient as ecw. Files are slightly larger. Mosaicking, projection and file conversion was of course done with FME.
*GisKraft*, Geodatakonsult Mats Elfström, Marsvinsholms stationsväg 80, 271 93 Ystad, Sweden tel: +46 70 595 39 35 / web: www.giskraft.com Den sön 24 nov. 2019 kl 16:14 skrev David I <persso...@gmail.com>: > Hello Simone, > > thanks for the concrete piece of advice (more than 30-40 files then > imagemosaic might be unsuitable, find some other method, like using the > pyramids plugin). In regards to whether one might feed our collection of > 500MB tiffs (tiled for one scale, that is) to gdal_retile or if to try to > produce something that isn't fantastically big with gdal_merge and then > give > that to gdal_retile or gdalbuildvrt on them and then gdal_translate to jpeg > compress and produce and one large (but not super large) tif and feed that > to gdal_retile - I shall tinker (the Geoserver docs give one big as the > input in the example but the tool supposedly can handle multiple, including > what's already tiled) (though if anyone has strong recommendations either > way then don't be afraid to holler :) ) > > Won't open a ticket for the timeout as we are currently on an older version > of Geoserver. An update is being planned but this tif-sink thing required > more immediate attention. > > Thanks again, > > Regards > David. > == > GeoServer Professional Services from the experts! > Visit http://bit.ly/gs-services > < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fbit.ly%2Fgs-services&dID=1574517951672&linkName=http://bit.ly/gs-services> > ; > for more information. > == > Ing. Simone Giannecchini > @simogeo > Founder/Director > > GeoSolutions S.A.S. > Via di Montramito 3/A > 55054 Massarosa (LU) > Italy > phone: +39 0584 962313 > fax: +39 0584 1660272 > mob: +39 333 8128928 > > http://www.geo-solutions.it > < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.geo-solutions.it&dID=1574517951672&linkName=http://www.geo-solutions.it> > ; > http://twitter.com/geosolutions_it > < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Ftwitter.com%2Fgeosolutions_it&dID=1574517951672&linkName=http://twitter.com/geosolutions_it> > ; > > ------------------------------------------------------- > Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE > 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si > precisa che ogni circostanza inerente alla presente email (il suo > contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è > riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il > messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra > operazione è illecita. Le sarei comunque grato se potesse darmene notizia. > > This email is intended only for the person or entity to which it is > addressed and may contain information that is privileged, confidential or > otherwise protected from disclosure. We remind that - as provided by > European Regulation 2016/679 “GDPR” - copying, dissemination or use of this > e-mail or the information herein by anyone other than the intended > recipient is prohibited. If you have received this email by mistake, please > notify us immediately by telephone or e-mail. > > > On Sat, Nov 23, 2019 at 2:26 PM David Persson <perssond9@> wrote: > > > Hi, thanks for your replies. > > > > We have not employed the image pyramid plugin. The current troublesome > > source material is an orthophoto delivered as a collection of rgb tiffs > > with .tfw reference files, 8cm resolution, 1000x1000m per tiff. Each tiff > > is 500MB, total size 324GB. > > > > The tiffs were prepped thusly: for %%i in (*.tif) do gdal_translate -co > > COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES %%i <folder>%%~ni.tif > > which reduced the size in MB of many tiffs by rather a lot. Then it was > > published via the Image Mosaic plugin in Geoserver. > > Would I need to gdal_merge into one gigantic tiff first in order to run > > gdal_retile? Or would you advice preparing this source material in some > > other way? > > > > The key point for the imagemosaic is the number of files it has to open to > answer a request. > If you mosaic is comprised of more than 30 to 40 files as you zoom out > GeoServer will suffer a lot to answer, up to a point where it will probably > die for one reason or the other. > > So we usually use pyramids or layergroup with different layers plus styling > rules based on the scale to exclude some layers. > > > > > > (Eventually we might run a standalone gwc as some things should only be > > employed cached but we still need to learn how to prep and publish raster > > properly in Geoserver in order for gwc to have something to feed on :) ) > > > > (Haven't found a bug report about getMap requests that ignore the > > Geoserver WMS timeout set time, yet) > > > > You can open that, this is true only for raster data atm. > > > > > > Respectfully, > > David. > > > > On Mon, Nov 11, 2019 at 1:37 PM Simone Giannecchini < > > simone.giannecchini@> wrote: > > > >> Dear David, > >> I would take a different path and understand more about this "large > >> rasters": > >> > >> - tell us more about format and size > >> - tell us more about how you preprocessed them for performance > >> > >> The first step with serving (large) raster data is choosing the right > >> format then preprocessing, see here (slide 8 and onward): > >> < > https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fwww.slideshare.net%2Fgeosolutions%2Fgeoserver-in-production-we-do-it-here-is-how-foss4g-2016&dID=1573475791127&linkName=https://www.slideshare.net/geosolutions/geoserver-in-production-we-do-it-here-is-how-foss4g-2016> > ; > >> https://www > >> < > https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fwww&dID=1574517951672&linkName=https://www> > ; > >> .slideshare.net/geosolutions/ > >> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fslideshare.net%2Fgeosolutions%2F&dID=1574517951672&linkName=slideshare.net/geosolutions/> > ; > >> geoserver-in-production-we-do-it-here-is-how-foss4g-2016 > >> Everything else comes (way) afterwards. > >> > >> Regards, > >> Simone Giannecchini > >> == > >> GeoServer Professional Services from the experts! > >> Visit http://bit.ly/gs-services > >> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fbit.ly%2Fgs-services&dID=1573475791127&linkName=http://bit.ly/gs-services> > ; > >> for more information. > >> == > >> Ing. Simone Giannecchini > >> @simogeo > >> Founder/Director > >> > >> GeoSolutions S.A.S. > >> Via di Montramito 3/A > >> 55054 Massarosa (LU) > >> Italy > >> phone: +39 0584 962313 > >> fax: +39 0584 1660272 > >> mob: +39 333 8128928 > >> > >> http://www.geo-solutions.it > >> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.geo-solutions.it&dID=1573475791127&linkName=http://www.geo-solutions.it> > ; > >> > >> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Ftwitter.com%2Fgeosolutions_it&dID=1573475791127&linkName=http://twitter.com/geosolutions_it> > ; > >> http://twitter.com/ > >> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Ftwitter.com%2F&dID=1574517951672&linkName=http://twitter.com/> > ; > >> geosolutions_it > >> > >> ------------------------------------------------------- > >> Con riferimento alla normativa sul trattamento dei dati personali (Reg. > >> UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si > >> precisa che ogni circostanza inerente alla presente email (il suo > >> contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è > >> riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il > >> messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra > >> operazione è illecita. Le sarei comunque grato se potesse darmene > >> notizia. > >> > >> This email is intended only for the person or entity to which it is > >> addressed and may contain information that is privileged, confidential > or > >> otherwise protected from disclosure. We remind that - as provided by > >> European Regulation 2016/679 “GDPR” - copying, dissemination or use of > >> this > >> e-mail or the information herein by anyone other than the intended > >> recipient is prohibited. If you have received this email by mistake, > >> please > >> notify us immediately by telephone or e-mail. > >> > >> > >> On Sun, Nov 10, 2019 at 3:47 AM David I <perssond9@> wrote: > >> > >>> Hi, > >>> > >>> We're currently running 2.11.5 (upgrade of the whole server scheduled > >>> for > >>> early next year) and are experiencing stability issues with the Jetty > >>> 9.3 > >>> hosting Geoserver (and a Geoexplorer). For instance large ortho rasters > >>> previewed in Geoserver try to render for a long time and eventually > fail > >>> and > >>> if several of those heavy getMaps happen in parallell Jetty becomes > slow > >>> to > >>> respond and will either end up in state where it doesn't appear to > >>> recover, > >>> at least for 10 minutes, or it recovers. Lesser rasters preview ok and > >>> do > >>> not cause > >>> > >>> (This is a problem when using these WMS layers in desktop applications, > >>> it's > >>> not that lots of people preview them from Geoserver's /web) > >>> > >>> > >>> Latest Jetty 9.3, latest Corretto JRE 8. > >>> > >>> Java options: > >>> Marlin 0.9.4.1, > >>> -XX:+UseParallelGC > >>> -XX:SoftRefLRUPolicyMSPerMB=36000 > >>> > >>> 6144MB initial heap and 8192 max, thread stack size 256KB. > >>> > >>> When Jetty becomes partially or totally unresponsive the heap via > >>> jmapheap > >>> -heap looks like so: > >>> > >>> Heap Configuration: > >>> MinHeapFreeRatio = 0 > >>> MaxHeapFreeRatio = 100 > >>> MaxHeapSize = 8589934592 (8192.0MB) > >>> NewSize = 2147483648 (2048.0MB) > >>> MaxNewSize = 2863136768 (2730.5MB) > >>> OldSize = 4294967296 (4096.0MB) > >>> NewRatio = 2 > >>> SurvivorRatio = 8 > >>> MetaspaceSize = 21807104 (20.796875MB) > >>> CompressedClassSpaceSize = 1073741824 (1024.0MB) > >>> MaxMetaspaceSize = 17592186044415 MB > >>> G1HeapRegionSize = 0 (0.0MB) > >>> > >>> Heap Usage: > >>> PS Young Generation > >>> Eden Space: > >>> capacity = 1982332928 (1890.5MB) > >>> used = 1982332928 (1890.5MB) > >>> free = 0 (0.0MB) > >>> 100.0% used > >>> From Space: > >>> capacity = 437256192 (417.0MB) > >>> used = 0 (0.0MB) > >>> free = 437256192 (417.0MB) > >>> 0.0% used > >>> To Space: > >>> capacity = 439353344 (419.0MB) > >>> used = 0 (0.0MB) > >>> free = 439353344 (419.0MB) > >>> 0.0% used > >>> PS Old Generation > >>> capacity = 5726797824 (5461.5MB) > >>> used = 5726435656 (5461.154609680176MB) > >>> free = 362168 (0.34539031982421875MB) > >>> 99.99367590735469% used > >>> > >>> In addition Geoserver doesn't appear to adhere to the max rendering > time > >>> under WMS settings: 35s, this is what the Monitor plugin captured > >>> shortly > >>> before Jetty stopped responding earlier today: > >>> > >>> > >>> /<Request id="3,417,511"> > >>> <Operation>GetMap</Operation> > >>> <Resources>ortofoto:orto_tatort2018</Resources> > >>> <Path>/ows</Path> > >>> > >>> > >>> > <QueryString>SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&LAYERS=ortofoto:orto_tatort2018&STYLES=&CRS=EPSG:3010&FORMAT=image/jpeg&TRANSPARENT=TRUE&WIDTH=502&HEIGHT=502&BBOX=6575000,141000,6579016,145016</QueryString> > >>> <Body> > >>> > >>> </Body> > >>> <StartTime>2019-11-07T07:06:21.469Z</StartTime> > >>> <EndTime>2019-11-07T07:13:26.694Z</EndTime> > >>> <TotalTime>425,225</TotalTime> > >>> <ResponseStatus>200</ResponseStatus> > >>> <Failed>false</Failed> > >>> </Request> / > >>> > >>> Geoserver's (production9 or jetty (dev) log do not mention > >>> java.lang.outofmemoryerror, but Geoserver will sometimes reply with > just > >>> that : heap space, when in its unresponsive state and a wfs getFeature > >>> is > >>> sent. (logging typically stops when the servlet becomes unresponsive). > >>> > >>> Would a different garbage collector be of use? > >>> Setting MaxPermSize? > >>> Somehow optimizing the tifs of the mosaic stores? > >>> Is there a way to not allow these WMS services to be used except with > >>> tiled=true -> their cached variants? > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> Sent from: > >>> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fosgeo-org.1560.x6.nabble.com%2FGeoServer-User-f3786390.html&dID=1574517951672> > ; > >>> http://osgeo-org.1560.x6 > >>> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fosgeo-org.1560.x6&dID=1574517951672&linkName=http://osgeo-org.1560.x6> > ; > >>> .nabble.com/GeoServer-User- > >>> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fnabble.com%2FGeoServer-User-&dID=1574517951672&linkName=nabble.com/GeoServer-User-> > ; > >>> f3786390.html > >>> > >>> > >>> _______________________________________________ > >>> Geoserver-users mailing list > >>> > >>> Please make sure you read the following two resources before posting to > >>> this list: > >>> - Earning your support instead of buying it, but Ian Turton: > >>> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.ianturton.com%2Ftalks%2Ffoss4g.html%23%2F&dID=1574517951672> > ; > >>> http://www.ianturton.com/ > >>> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.ianturton.com%2F&dID=1574517951672&linkName=http://www.ianturton.com/> > ; > >>> talks/foss4g.html#/ > >>> - The GeoServer user list posting guidelines: > >>> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fgeoserver.org%2Fcomm%2Fuserlist-guidelines.html&dID=1574517951672> > ; > >>> http://geoserver.org/comm/ > >>> < > https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fgeoserver.org%2Fcomm%2F&dID=1574517951672&linkName=http://geoserver.org/comm/> > ; > >>> userlist-guidelines.html > >>> > >>> If you want to request a feature or an improvement, also see this: > >>> < > https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fgithub.com%2Fgeoserver%2Fgeoserver%2Fwiki%2FSuccessfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer&dID=1574517951672> > ; > >>> https://github.com/geoserver/ > >>> < > https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fgithub.com%2Fgeoserver%2F&dID=1574517951672&linkName=https://github.com/geoserver/> > ; > >>> > geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer > >>> > >>> > >>> Geoserver-users@.sourceforge > >>> > >>> < > https://gnldr.online/tracker/click?redirect=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fgeoserver-users&dID=1574517951672> > ; > >>> https://lists.sourceforge.net/ > >>> < > https://gnldr.online/tracker/click?redirect=https%3A%2F%2Flists.sourceforge.net%2F&dID=1574517951672&linkName=https://lists.sourceforge.net/> > ; > >>> lists/listinfo/geoserver-users > >>> > >> > > > _______________________________________________ > Geoserver-users mailing list > > Please make sure you read the following two resources before posting to > this > list: > - Earning your support instead of buying it, but Ian Turton: > http://www.ianturton.com/talks/foss4g.html#/ > - The GeoServer user list posting guidelines: > http://geoserver.org/comm/userlist-guidelines.html > > If you want to request a feature or an improvement, also see this: > > https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer > > > Geoserver-users@.sourceforge > https://lists.sourceforge.net/lists/listinfo/geoserver-users > > > > > > -- > Sent from: > http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html > > > _______________________________________________ > Geoserver-users mailing list > > Please make sure you read the following two resources before posting to > this list: > - Earning your support instead of buying it, but Ian Turton: > http://www.ianturton.com/talks/foss4g.html#/ > - The GeoServer user list posting guidelines: > http://geoserver.org/comm/userlist-guidelines.html > > If you want to request a feature or an improvement, also see this: > https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer > > > Geoserver-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geoserver-users >
_______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
