Hi there, thanks for your reply! Just so that I understand you correctly, did you after creating one jpeg2k file per store set them up as imagemosaic stores in Geoserver? As ImageMosaic can be used with jpeg2k files after a GDAL extension is installed for Geoserver and I didn't see you explicitly mentioning another type of data store and the description for the ImageMosaic type is that it mosaics 'two or more' images. Anyways, have currently ordered FME to 322GB (~750) tifs -> 1 jpeg2k, will be interesting to see what it produces : )
Regards, David matself wrote > Hi! > I recently faced a similar task. > Hundreds of high resolution geotiffs over a limited area that I wanted to > serve on Geoserver. This is 2.16 but I do not think that matters. I also > have several image sets from different years. In all, this amounts to > several tens of gigabytes. > I tried to use image mosaic, but performance was horrible. A suggestion > from this list was to degrade the tiffs and compress them further. This > would effectively duplicate an already large file storage and I hesitated > to do that. Also, I do not want the Geoserver machine to double as a file > storage. > I solved the task by mosaicking the yearly file sets into jpeg200, one > file > per year. These I then serve, one store and layer per year. No need for > tiling either. That is taken care of in the file format. > Normally I use ecw for purposes like these, but there is a licensing issue > with ecw in a server environment. JPEG2000 is true open source, and is > almost as efficient as ecw. Files are slightly larger. > Mosaicking, projection and file conversion was of course done with FME. > > > *GisKraft*, Geodatakonsult > > Mats Elfström, Marsvinsholms stationsväg 80, 271 93 Ystad, Sweden > tel: +46 70 595 39 35 / web: www.giskraft.com > > > Den sön 24 nov. 2019 kl 16:14 skrev David I < > perssond9@ > >: > >> Hello Simone, >> >> thanks for the concrete piece of advice (more than 30-40 files then >> imagemosaic might be unsuitable, find some other method, like using the >> pyramids plugin). In regards to whether one might feed our collection of >> 500MB tiffs (tiled for one scale, that is) to gdal_retile or if to try to >> produce something that isn't fantastically big with gdal_merge and then >> give >> that to gdal_retile or gdalbuildvrt on them and then gdal_translate to >> jpeg >> compress and produce and one large (but not super large) tif and feed >> that >> to gdal_retile - I shall tinker (the Geoserver docs give one big as the >> input in the example but the tool supposedly can handle multiple, >> including >> what's already tiled) (though if anyone has strong recommendations either >> way then don't be afraid to holler :) ) >> >> Won't open a ticket for the timeout as we are currently on an older >> version >> of Geoserver. An update is being planned but this tif-sink thing required >> more immediate attention. >> >> Thanks again, >> >> Regards >> David. >> == >> GeoServer Professional Services from the experts! >> Visit http://bit.ly/gs-services >> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fbit.ly%2Fgs-services&dID=1574517951672&linkName=http://bit.ly/gs-services> >> ; >> for more information. >> == >> Ing. Simone Giannecchini >> @simogeo >> Founder/Director >> >> GeoSolutions S.A.S. >> Via di Montramito 3/A >> 55054 Massarosa (LU) >> Italy >> phone: +39 0584 962313 >> fax: +39 0584 1660272 >> mob: +39 333 8128928 >> >> http://www.geo-solutions.it >> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.geo-solutions.it&dID=1574517951672&linkName=http://www.geo-solutions.it> >> ; >> http://twitter.com/geosolutions_it >> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Ftwitter.com%2Fgeosolutions_it&dID=1574517951672&linkName=http://twitter.com/geosolutions_it> >> ; >> >> ------------------------------------------------------- >> Con riferimento alla normativa sul trattamento dei dati personali (Reg. >> UE >> 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si >> precisa che ogni circostanza inerente alla presente email (il suo >> contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è >> riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il >> messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra >> operazione è illecita. Le sarei comunque grato se potesse darmene >> notizia. >> >> This email is intended only for the person or entity to which it is >> addressed and may contain information that is privileged, confidential or >> otherwise protected from disclosure. We remind that - as provided by >> European Regulation 2016/679 “GDPR” - copying, dissemination or use of >> this >> e-mail or the information herein by anyone other than the intended >> recipient is prohibited. If you have received this email by mistake, >> please >> notify us immediately by telephone or e-mail. >> >> >> On Sat, Nov 23, 2019 at 2:26 PM David Persson <perssond9@> wrote: >> >> > Hi, thanks for your replies. >> > >> > We have not employed the image pyramid plugin. The current troublesome >> > source material is an orthophoto delivered as a collection of rgb tiffs >> > with .tfw reference files, 8cm resolution, 1000x1000m per tiff. Each >> tiff >> > is 500MB, total size 324GB. >> > >> > The tiffs were prepped thusly: for %%i in (*.tif) do gdal_translate -co >> > COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES %%i > <folder> > %%~ni.tif >> > which reduced the size in MB of many tiffs by rather a lot. Then it was >> > published via the Image Mosaic plugin in Geoserver. >> > Would I need to gdal_merge into one gigantic tiff first in order to run >> > gdal_retile? Or would you advice preparing this source material in some >> > other way? >> > >> >> The key point for the imagemosaic is the number of files it has to open >> to >> answer a request. >> If you mosaic is comprised of more than 30 to 40 files as you zoom out >> GeoServer will suffer a lot to answer, up to a point where it will >> probably >> die for one reason or the other. >> >> So we usually use pyramids or layergroup with different layers plus >> styling >> rules based on the scale to exclude some layers. >> >> >> > >> > (Eventually we might run a standalone gwc as some things should only be >> > employed cached but we still need to learn how to prep and publish >> raster >> > properly in Geoserver in order for gwc to have something to feed on :) >> ) >> > >> > (Haven't found a bug report about getMap requests that ignore the >> > Geoserver WMS timeout set time, yet) >> > >> >> You can open that, this is true only for raster data atm. >> >> >> > >> > Respectfully, >> > David. >> > >> > On Mon, Nov 11, 2019 at 1:37 PM Simone Giannecchini < >> > simone.giannecchini@> wrote: >> > >> >> Dear David, >> >> I would take a different path and understand more about this "large >> >> rasters": >> >> >> >> - tell us more about format and size >> >> - tell us more about how you preprocessed them for performance >> >> >> >> The first step with serving (large) raster data is choosing the right >> >> format then preprocessing, see here (slide 8 and onward): >> >> < >> https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fwww.slideshare.net%2Fgeosolutions%2Fgeoserver-in-production-we-do-it-here-is-how-foss4g-2016&dID=1573475791127&linkName=https://www.slideshare.net/geosolutions/geoserver-in-production-we-do-it-here-is-how-foss4g-2016> >> ; >> >> https://www >> >> < >> https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fwww&dID=1574517951672&linkName=https://www> >> ; >> >> .slideshare.net/geosolutions/ >> >> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fslideshare.net%2Fgeosolutions%2F&dID=1574517951672&linkName=slideshare.net/geosolutions/> >> ; >> >> geoserver-in-production-we-do-it-here-is-how-foss4g-2016 >> >> Everything else comes (way) afterwards. >> >> >> >> Regards, >> >> Simone Giannecchini >> >> == >> >> GeoServer Professional Services from the experts! >> >> Visit http://bit.ly/gs-services >> >> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fbit.ly%2Fgs-services&dID=1573475791127&linkName=http://bit.ly/gs-services> >> ; >> >> for more information. >> >> == >> >> Ing. Simone Giannecchini >> >> @simogeo >> >> Founder/Director >> >> >> >> GeoSolutions S.A.S. >> >> Via di Montramito 3/A >> >> 55054 Massarosa (LU) >> >> Italy >> >> phone: +39 0584 962313 >> >> fax: +39 0584 1660272 >> >> mob: +39 333 8128928 >> >> >> >> http://www.geo-solutions.it >> >> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.geo-solutions.it&dID=1573475791127&linkName=http://www.geo-solutions.it> >> ; >> >> >> >> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Ftwitter.com%2Fgeosolutions_it&dID=1573475791127&linkName=http://twitter.com/geosolutions_it> >> ; >> >> http://twitter.com/ >> >> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Ftwitter.com%2F&dID=1574517951672&linkName=http://twitter.com/> >> ; >> >> geosolutions_it >> >> >> >> ------------------------------------------------------- >> >> Con riferimento alla normativa sul trattamento dei dati personali >> (Reg. >> >> UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), >> si >> >> precisa che ogni circostanza inerente alla presente email (il suo >> >> contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è >> >> riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il >> >> messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra >> >> operazione è illecita. Le sarei comunque grato se potesse darmene >> >> notizia. >> >> >> >> This email is intended only for the person or entity to which it is >> >> addressed and may contain information that is privileged, confidential >> or >> >> otherwise protected from disclosure. We remind that - as provided by >> >> European Regulation 2016/679 “GDPR” - copying, dissemination or use of >> >> this >> >> e-mail or the information herein by anyone other than the intended >> >> recipient is prohibited. If you have received this email by mistake, >> >> please >> >> notify us immediately by telephone or e-mail. >> >> >> >> >> >> On Sun, Nov 10, 2019 at 3:47 AM David I <perssond9@> wrote: >> >> >> >>> Hi, >> >>> >> >>> We're currently running 2.11.5 (upgrade of the whole server scheduled >> >>> for >> >>> early next year) and are experiencing stability issues with the Jetty >> >>> 9.3 >> >>> hosting Geoserver (and a Geoexplorer). For instance large ortho >> rasters >> >>> previewed in Geoserver try to render for a long time and eventually >> fail >> >>> and >> >>> if several of those heavy getMaps happen in parallell Jetty becomes >> slow >> >>> to >> >>> respond and will either end up in state where it doesn't appear to >> >>> recover, >> >>> at least for 10 minutes, or it recovers. Lesser rasters preview ok >> and >> >>> do >> >>> not cause >> >>> >> >>> (This is a problem when using these WMS layers in desktop >> applications, >> >>> it's >> >>> not that lots of people preview them from Geoserver's /web) >> >>> >> >>> >> >>> Latest Jetty 9.3, latest Corretto JRE 8. >> >>> >> >>> Java options: >> >>> Marlin 0.9.4.1, >> >>> -XX:+UseParallelGC >> >>> -XX:SoftRefLRUPolicyMSPerMB=36000 >> >>> >> >>> 6144MB initial heap and 8192 max, thread stack size 256KB. >> >>> >> >>> When Jetty becomes partially or totally unresponsive the heap via >> >>> jmapheap >> >>> -heap looks like so: >> >>> >> >>> Heap Configuration: >> >>> MinHeapFreeRatio = 0 >> >>> MaxHeapFreeRatio = 100 >> >>> MaxHeapSize = 8589934592 (8192.0MB) >> >>> NewSize = 2147483648 (2048.0MB) >> >>> MaxNewSize = 2863136768 (2730.5MB) >> >>> OldSize = 4294967296 (4096.0MB) >> >>> NewRatio = 2 >> >>> SurvivorRatio = 8 >> >>> MetaspaceSize = 21807104 (20.796875MB) >> >>> CompressedClassSpaceSize = 1073741824 (1024.0MB) >> >>> MaxMetaspaceSize = 17592186044415 MB >> >>> G1HeapRegionSize = 0 (0.0MB) >> >>> >> >>> Heap Usage: >> >>> PS Young Generation >> >>> Eden Space: >> >>> capacity = 1982332928 (1890.5MB) >> >>> used = 1982332928 (1890.5MB) >> >>> free = 0 (0.0MB) >> >>> 100.0% used >> >>> From Space: >> >>> capacity = 437256192 (417.0MB) >> >>> used = 0 (0.0MB) >> >>> free = 437256192 (417.0MB) >> >>> 0.0% used >> >>> To Space: >> >>> capacity = 439353344 (419.0MB) >> >>> used = 0 (0.0MB) >> >>> free = 439353344 (419.0MB) >> >>> 0.0% used >> >>> PS Old Generation >> >>> capacity = 5726797824 (5461.5MB) >> >>> used = 5726435656 (5461.154609680176MB) >> >>> free = 362168 (0.34539031982421875MB) >> >>> 99.99367590735469% used >> >>> >> >>> In addition Geoserver doesn't appear to adhere to the max rendering >> time >> >>> under WMS settings: 35s, this is what the Monitor plugin captured >> >>> shortly >> >>> before Jetty stopped responding earlier today: >> >>> >> >>> >> >>> / > <Request id="3,417,511"> >> >>> > <Operation> > GetMap > </Operation> >> >>> > <Resources> > ortofoto:orto_tatort2018 > </Resources> >> >>> > <Path> > /ows > </Path> >> >>> >> >>> >> >>> >> > <QueryString> > SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&LAYERS=ortofoto:orto_tatort2018&STYLES=&CRS=EPSG:3010&FORMAT=image/jpeg&TRANSPARENT=TRUE&WIDTH=502&HEIGHT=502&BBOX=6575000,141000,6579016,145016 > </QueryString> >> >>> > <Body> >> >>> >> >>> > </Body> >> >>> > <StartTime> > 2019-11-07T07:06:21.469Z > </StartTime> >> >>> > <EndTime> > 2019-11-07T07:13:26.694Z > </EndTime> >> >>> > <TotalTime> > 425,225 > </TotalTime> >> >>> > <ResponseStatus> > 200 > </ResponseStatus> >> >>> > <Failed> > false > </Failed> >> >>> > </Request> > / >> >>> >> >>> Geoserver's (production9 or jetty (dev) log do not mention >> >>> java.lang.outofmemoryerror, but Geoserver will sometimes reply with >> just >> >>> that : heap space, when in its unresponsive state and a wfs >> getFeature >> >>> is >> >>> sent. (logging typically stops when the servlet becomes >> unresponsive). >> >>> >> >>> Would a different garbage collector be of use? >> >>> Setting MaxPermSize? >> >>> Somehow optimizing the tifs of the mosaic stores? >> >>> Is there a way to not allow these WMS services to be used except with >> >>> tiled=true -> their cached variants? >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Sent from: >> >>> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fosgeo-org.1560.x6.nabble.com%2FGeoServer-User-f3786390.html&dID=1574517951672> >> ; >> >>> http://osgeo-org.1560.x6 >> >>> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fosgeo-org.1560.x6&dID=1574517951672&linkName=http://osgeo-org.1560.x6> >> ; >> >>> .nabble.com/GeoServer-User- >> >>> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fnabble.com%2FGeoServer-User-&dID=1574517951672&linkName=nabble.com/GeoServer-User-> >> ; >> >>> f3786390.html >> >>> >> >>> >> >>> _______________________________________________ >> >>> Geoserver-users mailing list >> >>> >> >>> Please make sure you read the following two resources before posting >> to >> >>> this list: >> >>> - Earning your support instead of buying it, but Ian Turton: >> >>> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.ianturton.com%2Ftalks%2Ffoss4g.html%23%2F&dID=1574517951672> >> ; >> >>> http://www.ianturton.com/ >> >>> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fwww.ianturton.com%2F&dID=1574517951672&linkName=http://www.ianturton.com/> >> ; >> >>> talks/foss4g.html#/ >> >>> - The GeoServer user list posting guidelines: >> >>> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fgeoserver.org%2Fcomm%2Fuserlist-guidelines.html&dID=1574517951672> >> ; >> >>> http://geoserver.org/comm/ >> >>> < >> https://gnldr.online/tracker/click?redirect=http%3A%2F%2Fgeoserver.org%2Fcomm%2F&dID=1574517951672&linkName=http://geoserver.org/comm/> >> ; >> >>> userlist-guidelines.html >> >>> >> >>> If you want to request a feature or an improvement, also see this: >> >>> < >> https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fgithub.com%2Fgeoserver%2Fgeoserver%2Fwiki%2FSuccessfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer&dID=1574517951672> >> ; >> >>> https://github.com/geoserver/ >> >>> < >> https://gnldr.online/tracker/click?redirect=https%3A%2F%2Fgithub.com%2Fgeoserver%2F&dID=1574517951672&linkName=https://github.com/geoserver/> >> ; >> >>> >> geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer >> >>> >> >>> >> >>> Geoserver-users@.sourceforge >> >>> >> >>> < >> https://gnldr.online/tracker/click?redirect=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fgeoserver-users&dID=1574517951672> >> ; >> >>> https://lists.sourceforge.net/ >> >>> < >> https://gnldr.online/tracker/click?redirect=https%3A%2F%2Flists.sourceforge.net%2F&dID=1574517951672&linkName=https://lists.sourceforge.net/> >> ; >> >>> lists/listinfo/geoserver-users >> >>> >> >> >> >> >> _______________________________________________ >> Geoserver-users mailing list >> >> Please make sure you read the following two resources before posting to >> this >> list: >> - Earning your support instead of buying it, but Ian Turton: >> http://www.ianturton.com/talks/foss4g.html#/ >> - The GeoServer user list posting guidelines: >> http://geoserver.org/comm/userlist-guidelines.html >> >> If you want to request a feature or an improvement, also see this: >> >> https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer >> >> >> Geoserver-users@.sourceforge >> https://lists.sourceforge.net/lists/listinfo/geoserver-users >> >> >> >> >> >> -- >> Sent from: >> http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html >> >> >> _______________________________________________ >> Geoserver-users mailing list >> >> Please make sure you read the following two resources before posting to >> this list: >> - Earning your support instead of buying it, but Ian Turton: >> http://www.ianturton.com/talks/foss4g.html#/ >> - The GeoServer user list posting guidelines: >> http://geoserver.org/comm/userlist-guidelines.html >> >> If you want to request a feature or an improvement, also see this: >> https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer >> >> >> > Geoserver-users@.sourceforge >> https://lists.sourceforge.net/lists/listinfo/geoserver-users >> > > > _______________________________________________ > Geoserver-users mailing list > > Please make sure you read the following two resources before posting to > this list: > - Earning your support instead of buying it, but Ian Turton: > http://www.ianturton.com/talks/foss4g.html#/ > - The GeoServer user list posting guidelines: > http://geoserver.org/comm/userlist-guidelines.html > > If you want to request a feature or an improvement, also see this: > https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer > > Geoserver-users@.sourceforge > https://lists.sourceforge.net/lists/listinfo/geoserver-users -- Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-User-f3786390.html _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
