Hi Andrea, thank you for your time and your suggestion! However, I don´t think the problem has to do with the *metatiling lock*, but instead with the locks that are used to read and write the " *metadata.properties.gz*" file for the layer... Since we always had metatiling set at 4x4 and never had problems with performance before.
However, our layer is using viewparams, and as we understand, that is what causing the creation and use of the metadata.properties.gz file. We spent long hours investigating the issue, and trying all sorts of things... we finally decided to download the code and try to patch it. We commented the call to "persistParameterMap" on line 480 of FileBlobStore.java and recompiled gwc-core.jar ignoring all test cases errors. Link to the exact line here: https://github.com/GeoWebCache/geowebcache/blob/main/geowebcache/core/src/main/java/org/geowebcache/storage/blobstore/file/FileBlobStore.java#L480 That prevented the reading and writing of the metadata.properties.gz file and so we have no longer locks on that file. After doing that, GWC started working fine, answering really fast and caching everything as usual! So now we have our server back online after several days of downtime. Of course this means that the metadata.properties.gz is no longer being generated and that worries us. We tried to determine what that file is used for, to determine if we could be generating a future problem but couldn't find a clear answer by looking at the code. Seems like this file is a reverse hash map of some kind to be able to determine what parameters in the original request generated what folder in the cache. But at least in our use case we don't see any use for it... We think this may be used on truncate operations, but we have truncate operations on our system, and that seems to be working fine. Would you be so kind to discuss with us about the need of that file, and possible consequences for disabling it? Do you think there could be a bug with the locks that causes the service to stop responding. Or in case that file is not absolutely necessary in all use cases, maybe there could be a flag to enable or disable it more easily without having to modify the source code? I know this is somehow a complex matter and and usual we really thank you and everyone for any help and also hope this can help resolve the issue for future versions! Best Regards! Sebastián El lun, 27 mar 2023 a las 13:04, Andrea Aime (< andrea.a...@geosolutionsgroup.com>) escribió: > Hi Seba, > the metatiling lock is taken to prevent multiple requests from building > the same meta-tile, so one is actually building > the output, while all others are waiting for the tiles to be produced. > > Given your explanation, maybe you have a very large meta-tiling factor in > your layer, and the production of the full meta-tile (single large image > containing all tiles) > really takes that much time? For that time, all other requests hitting the > same meta-tile will be blocked, waiting. > > If that's the case, then a quick way to get the server to respond faster, > is to reduce the meta-tiling factor > > Cheers > Andrea > > > On Fri, Mar 17, 2023 at 6:34 PM Seba Oszlak <sosz...@gmail.com> wrote: > >> Hello Everyone! >> >> A couple of days ago we started experiencing severe downtime on our >> Geoserver instance that was working fine for a long time before this. >> For context we have Geo 2.22.0 running on Tomcat 9 on a Windows Server >> 2019. >> >> All our requests go to GWC gmaps service with an URL in this format: >> >> >> /geoserver/gwc/service/gmaps?FORMAT=image/png8&LAYERS=skycop:Referencias+Vectorial+2&ZOOM=17&X=44602&Y=75080&VIEWPARAMS=Perfil:USERNAME;ShowLabels:1&_swc=1678964828059 >> >> In the latest version of our application, we have added a new parameter >> in the VIEWPARAMS for the requests (ShowLabels) and that of course in turn >> provoked a complete reseed of the Cache. Since then, when the server is >> under normal load, threads take a very long time to resolve (500/600 sec), >> or never resolve at all. So after a little while all Tomcat threads become >> busy and unresponsive. >> >> We have no errors recorded on the logs, no matter what log level we set. >> >> Our DB (SQL Server) is configured with a 64 connections pool. Is working >> normally and responding plenty fast to Geoserver, so we have mostly >> discarded that as the bottleneck. >> >> Investigating further, we see many of the threads are mostly idle waiting >> for something, so after some thread dumps, we found that most of the treads >> are stuck on this line of code where it seems to be waiting on a lock to be >> released for the metadata.properties.gz file: >> >> >> https://github.com/geoserver/geoserver/blob/main/src/gwc/src/main/java/org/geoserver/gwc/layer/GeoServerTileLayer.java#L594 >> >> I should add that during the night, with light load, the server is >> responding fine to requests in no more than 2/3 seconds. (And much faster >> obviously if the cache is hit). >> >> If anyone can give us a clue on where to look further or any other >> information or possible solution... we will be very very grateful! >> >> Thank you very much! >> >> Sebastian Oszlak >> >> _______________________________________________ >> Geoserver-users mailing list >> >> Please make sure you read the following two resources before posting to >> this list: >> - Earning your support instead of buying it, but Ian Turton: >> http://www.ianturton.com/talks/foss4g.html#/ >> - The GeoServer user list posting guidelines: >> http://geoserver.org/comm/userlist-guidelines.html >> >> If you want to request a feature or an improvement, also see this: >> https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer >> >> >> Geoserver-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/geoserver-users >> > > > -- > > Regards, > > Andrea Aime > > == > GeoServer Professional Services from the experts! > > Visit http://bit.ly/gs-services-us for more information. > == > > Ing. Andrea Aime > @geowolf > Technical Lead > > GeoSolutions Group > phone: +39 0584 962313 > > fax: +39 0584 1660272 > > mob: +39 339 8844549 > > https://www.geosolutionsgroup.com/ > > http://twitter.com/geosolutions_it > > ------------------------------------------------------- > > Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE > 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si > precisa che ogni circostanza inerente alla presente email (il suo > contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è > riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il > messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra > operazione è illecita. Le sarei comunque grato se potesse darmene notizia. > > This email is intended only for the person or entity to which it is > addressed and may contain information that is privileged, confidential or > otherwise protected from disclosure. We remind that - as provided by > European Regulation 2016/679 “GDPR” - copying, dissemination or use of this > e-mail or the information herein by anyone other than the intended > recipient is prohibited. If you have received this email by mistake, please > notify us immediately by telephone or e-mail >
_______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
