Hello Simone, thanks for the concrete piece of advice (more than 30-40 files then imagemosaic might be unsuitable, find some other method, like using the pyramids plugin). In regards to whether one might feed our collection of 500MB tiffs (tiled for one scale, that is) to gdal_retile or if to try to produce something that isn't fantastically big with gdal_merge and then give that to gdal_retile or gdalbuildvrt on them and then gdal_translate to jpeg compress and produce and one large (but not super large) tif and feed that to gdal_retile - I shall tinker (the Geoserver docs give one big as the input in the example but the tool supposedly can handle multiple, including what's already tiled) (though if anyone has strong recommendations either way then don't be afraid to holler :) )
Won't open a ticket for the timeout as we are currently on an older version of Geoserver. An update is being planned but this tif-sink thing required more immediate attention. Thanks again, Regards David. == GeoServer Professional Services from the experts! Visit http://bit.ly/gs-services <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fbit.ly%2Fgs-services&dID=1574517951672&linkName=http://bit.ly/gs-services> for more information. == Ing. Simone Giannecchini @simogeo Founder/Director GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 333 8128928 http://www.geo-solutions.it <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.geo-solutions.it&dID=1574517951672&linkName=http://www.geo-solutions.it> http://twitter.com/geosolutions_it <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Ftwitter.com%2Fgeosolutions_it&dID=1574517951672&linkName=http://twitter.com/geosolutions_it> ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail. On Sat, Nov 23, 2019 at 2:26 PM David Persson <perssond9@> wrote: > Hi, thanks for your replies. > > We have not employed the image pyramid plugin. The current troublesome > source material is an orthophoto delivered as a collection of rgb tiffs > with .tfw reference files, 8cm resolution, 1000x1000m per tiff. Each tiff > is 500MB, total size 324GB. > > The tiffs were prepped thusly: for %%i in (*.tif) do gdal_translate -co > COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES %%i <folder>%%~ni.tif > which reduced the size in MB of many tiffs by rather a lot. Then it was > published via the Image Mosaic plugin in Geoserver. > Would I need to gdal_merge into one gigantic tiff first in order to run > gdal_retile? Or would you advice preparing this source material in some > other way? > The key point for the imagemosaic is the number of files it has to open to answer a request. If you mosaic is comprised of more than 30 to 40 files as you zoom out GeoServer will suffer a lot to answer, up to a point where it will probably die for one reason or the other. So we usually use pyramids or layergroup with different layers plus styling rules based on the scale to exclude some layers. > > (Eventually we might run a standalone gwc as some things should only be > employed cached but we still need to learn how to prep and publish raster > properly in Geoserver in order for gwc to have something to feed on :) ) > > (Haven't found a bug report about getMap requests that ignore the > Geoserver WMS timeout set time, yet) > You can open that, this is true only for raster data atm. > > Respectfully, > David. > > On Mon, Nov 11, 2019 at 1:37 PM Simone Giannecchini < > simone.giannecchini@> wrote: > >> Dear David, >> I would take a different path and understand more about this "large >> rasters": >> >> - tell us more about format and size >> - tell us more about how you preprocessed them for performance >> >> The first step with serving (large) raster data is choosing the right >> format then preprocessing, see here (slide 8 and onward): >> <https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fwww.slideshare.net%2Fgeosolutions%2Fgeoserver-in-production-we-do-it-here-is-how-foss4g-2016&dID=1573475791127&linkName=https://www.slideshare.net/geosolutions/geoserver-in-production-we-do-it-here-is-how-foss4g-2016> >> https://www >> <https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fwww&dID=1574517951672&linkName=https://www> >> .slideshare.net/geosolutions/ >> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fslideshare.net%2Fgeosolutions%2F&dID=1574517951672&linkName=slideshare.net/geosolutions/> >> geoserver-in-production-we-do-it-here-is-how-foss4g-2016 >> Everything else comes (way) afterwards. >> >> Regards, >> Simone Giannecchini >> == >> GeoServer Professional Services from the experts! >> Visit http://bit.ly/gs-services >> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fbit.ly%2Fgs-services&dID=1573475791127&linkName=http://bit.ly/gs-services> >> for more information. >> == >> Ing. Simone Giannecchini >> @simogeo >> Founder/Director >> >> GeoSolutions S.A.S. >> Via di Montramito 3/A >> 55054 Massarosa (LU) >> Italy >> phone: +39 0584 962313 >> fax: +39 0584 1660272 >> mob: +39 333 8128928 >> >> http://www.geo-solutions.it >> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.geo-solutions.it&dID=1573475791127&linkName=http://www.geo-solutions.it> >> >> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Ftwitter.com%2Fgeosolutions_it&dID=1573475791127&linkName=http://twitter.com/geosolutions_it> >> http://twitter.com/ >> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Ftwitter.com%2F&dID=1574517951672&linkName=http://twitter.com/> >> geosolutions_it >> >> ------------------------------------------------------- >> Con riferimento alla normativa sul trattamento dei dati personali (Reg. >> UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si >> precisa che ogni circostanza inerente alla presente email (il suo >> contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è >> riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il >> messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra >> operazione è illecita. Le sarei comunque grato se potesse darmene >> notizia. >> >> This email is intended only for the person or entity to which it is >> addressed and may contain information that is privileged, confidential or >> otherwise protected from disclosure. We remind that - as provided by >> European Regulation 2016/679 “GDPR” - copying, dissemination or use of >> this >> e-mail or the information herein by anyone other than the intended >> recipient is prohibited. If you have received this email by mistake, >> please >> notify us immediately by telephone or e-mail. >> >> >> On Sun, Nov 10, 2019 at 3:47 AM David I <perssond9@> wrote: >> >>> Hi, >>> >>> We're currently running 2.11.5 (upgrade of the whole server scheduled >>> for >>> early next year) and are experiencing stability issues with the Jetty >>> 9.3 >>> hosting Geoserver (and a Geoexplorer). For instance large ortho rasters >>> previewed in Geoserver try to render for a long time and eventually fail >>> and >>> if several of those heavy getMaps happen in parallell Jetty becomes slow >>> to >>> respond and will either end up in state where it doesn't appear to >>> recover, >>> at least for 10 minutes, or it recovers. Lesser rasters preview ok and >>> do >>> not cause >>> >>> (This is a problem when using these WMS layers in desktop applications, >>> it's >>> not that lots of people preview them from Geoserver's /web) >>> >>> >>> Latest Jetty 9.3, latest Corretto JRE 8. >>> >>> Java options: >>> Marlin 0.9.4.1, >>> -XX:+UseParallelGC >>> -XX:SoftRefLRUPolicyMSPerMB=36000 >>> >>> 6144MB initial heap and 8192 max, thread stack size 256KB. >>> >>> When Jetty becomes partially or totally unresponsive the heap via >>> jmapheap >>> -heap looks like so: >>> >>> Heap Configuration: >>> MinHeapFreeRatio = 0 >>> MaxHeapFreeRatio = 100 >>> MaxHeapSize = 8589934592 (8192.0MB) >>> NewSize = 2147483648 (2048.0MB) >>> MaxNewSize = 2863136768 (2730.5MB) >>> OldSize = 4294967296 (4096.0MB) >>> NewRatio = 2 >>> SurvivorRatio = 8 >>> MetaspaceSize = 21807104 (20.796875MB) >>> CompressedClassSpaceSize = 1073741824 (1024.0MB) >>> MaxMetaspaceSize = 17592186044415 MB >>> G1HeapRegionSize = 0 (0.0MB) >>> >>> Heap Usage: >>> PS Young Generation >>> Eden Space: >>> capacity = 1982332928 (1890.5MB) >>> used = 1982332928 (1890.5MB) >>> free = 0 (0.0MB) >>> 100.0% used >>> From Space: >>> capacity = 437256192 (417.0MB) >>> used = 0 (0.0MB) >>> free = 437256192 (417.0MB) >>> 0.0% used >>> To Space: >>> capacity = 439353344 (419.0MB) >>> used = 0 (0.0MB) >>> free = 439353344 (419.0MB) >>> 0.0% used >>> PS Old Generation >>> capacity = 5726797824 (5461.5MB) >>> used = 5726435656 (5461.154609680176MB) >>> free = 362168 (0.34539031982421875MB) >>> 99.99367590735469% used >>> >>> In addition Geoserver doesn't appear to adhere to the max rendering time >>> under WMS settings: 35s, this is what the Monitor plugin captured >>> shortly >>> before Jetty stopped responding earlier today: >>> >>> >>> /<Request id="3,417,511"> >>> <Operation>GetMap</Operation> >>> <Resources>ortofoto:orto_tatort2018</Resources> >>> <Path>/ows</Path> >>> >>> >>> <QueryString>SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&LAYERS=ortofoto:orto_tatort2018&STYLES=&CRS=EPSG:3010&FORMAT=image/jpeg&TRANSPARENT=TRUE&WIDTH=502&HEIGHT=502&BBOX=6575000,141000,6579016,145016</QueryString> >>> <Body> >>> >>> </Body> >>> <StartTime>2019-11-07T07:06:21.469Z</StartTime> >>> <EndTime>2019-11-07T07:13:26.694Z</EndTime> >>> <TotalTime>425,225</TotalTime> >>> <ResponseStatus>200</ResponseStatus> >>> <Failed>false</Failed> >>> </Request> / >>> >>> Geoserver's (production9 or jetty (dev) log do not mention >>> java.lang.outofmemoryerror, but Geoserver will sometimes reply with just >>> that : heap space, when in its unresponsive state and a wfs getFeature >>> is >>> sent. (logging typically stops when the servlet becomes unresponsive). >>> >>> Would a different garbage collector be of use? >>> Setting MaxPermSize? >>> Somehow optimizing the tifs of the mosaic stores? >>> Is there a way to not allow these WMS services to be used except with >>> tiled=true -> their cached variants? >>> >>> >>> >>> >>> >>> -- >>> Sent from: >>> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fosgeo-org.1560.x6.nabble.com%2FGeoServer-User-f3786390.html&dID=1574517951672> >>> http://osgeo-org.1560.x6 >>> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fosgeo-org.1560.x6&dID=1574517951672&linkName=http://osgeo-org.1560.x6> >>> .nabble.com/GeoServer-User- >>> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fnabble.com%2FGeoServer-User-&dID=1574517951672&linkName=nabble.com/GeoServer-User-> >>> f3786390.html >>> >>> >>> _______________________________________________ >>> Geoserver-users mailing list >>> >>> Please make sure you read the following two resources before posting to >>> this list: >>> - Earning your support instead of buying it, but Ian Turton: >>> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.ianturton.com%2Ftalks%2Ffoss4g.html%23%2F&dID=1574517951672> >>> http://www.ianturton.com/ >>> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.ianturton.com%2F&dID=1574517951672&linkName=http://www.ianturton.com/> >>> talks/foss4g.html#/ >>> - The GeoServer user list posting guidelines: >>> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fgeoserver.org%2Fcomm%2Fuserlist-guidelines.html&dID=1574517951672> >>> http://geoserver.org/comm/ >>> <https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fgeoserver.org%2Fcomm%2F&dID=1574517951672&linkName=http://geoserver.org/comm/> >>> userlist-guidelines.html >>> >>> If you want to request a feature or an improvement, also see this: >>> <https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fgithub.com%2Fgeoserver%2Fgeoserver%2Fwiki%2FSuccessfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer&dID=1574517951672> >>> https://github.com/geoserver/ >>> <https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fgithub.com%2Fgeoserver%2F&dID=1574517951672&linkName=https://github.com/geoserver/> >>> geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer >>> >>> >>> Geoserver-users@.sourceforge >>> >>> <https://gnldr.online/tracker/click?redirect=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fgeoserver-users&dID=1574517951672> >>> https://lists.sourceforge.net/ >>> <https://gnldr.online/tracker/click?redirect=https%3A%2F%2Flists.sourceforge.net%2F&dID=1574517951672&linkName=https://lists.sourceforge.net/> >>> lists/listinfo/geoserver-users >>> >> _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@.sourceforge https://lists.sourceforge.net/lists/listinfo/geoserver-users -- Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
