Using PostGIS for indexing seems to have solved my problems. Thanks! Ashley Mort Sr. Software Engineer 972-638-6317 office 814-571-3039 cell
From: Andrea Aime <andrea.a...@geosolutionsgroup.com> Sent: Wednesday, November 17, 2021 9:16 AM To: Ashley Mort <ashley_c_m...@raytheon.com> Cc: GeoServer Mailing List List <geoserver-users@lists.sourceforge.net> Subject: [External] Re: [Geoserver-users] GeoServer hangs when POSTing external.imagemosaic with concurrent WMS requests Ashley, I strongly recommend against using Shapefile for holding the index of a mosaic that needs to be updated at runtime. The shapefile update is not a quick in-place process, it builds a new shapefile and then replaces the old one. While it would be possible to make concurrent edit better, in the best scenario, it would still be very slow. Do yourself a favour and switch to PostGIS Cheers Andrea On Wed, Nov 17, 2021 at 3:06 AM Ashley Mort via Geoserver-users <geoserver-users@lists.sourceforge.net<mailto:geoserver-users@lists.sourceforge.net>> wrote: So thanks to Simone helping me find some sample JMeter data, I was able to piece together a JMeter to find what has been causing my GeoServer 2.17.2 to become non-responsive. The JMeter test is (5threads/users repeated for 10 iterations) basically as follows. I will attach it here. I hope that’s ok as it’s only 15KB. GET WMS request GET WMS request GET WMS request POST external.imagemosaic (given a directory of images: /data/browse) Note: the image mosaic is ~100 JPEG granules each w/ a corresponding .jgw worldfile. We add a few images a few times each day. It seems once this JMeter is iterated a few times, I will eventually get an error like below (usually missing .shp or missing .qix file) then my GeoServer WMS requests will all hang and GeoServer Java process utilization will stay high. A restart of GeoServer sometimes totally resolves the issue and sometimes the Shapefile data files of my ImageMosaic Store get hosed and I have to manually delete and let GeoServer re-create them. Is there something I could be doing better here? If I could provide the exact image file path for each external.imagemosaic POST update instead of just giving it the parent directory of my images, would that help? Any thoughts are appreciated. Thanks! 17 Nov 01:16:27 ERROR [geoserver.ows] - org.geoserver.platform.ServiceException: Error rendering coverage on the fast path at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:362) at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:275) at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:135) at org.geoserver.wms.GetMap.executeInternal(GetMap.java:749) at org.geoserver.wms.GetMap.run(GetMap.java:300) at org.geoserver.wms.GetMap.run(GetMap.java:123) at org.geoserver.wms.DefaultWebMapService.getMap(DefaultWebMapService.java:246) at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.geoserver.kml.WebMapServiceKmlInterceptor.invoke(WebMapServiceKmlInterceptor.java:38) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.geoserver.gwc.wms.CacheSeedingWebMapService.invoke(CacheSeedingWebMapService.java:55) at org.geoserver.gwc.wms.CacheSeedingWebMapService.invoke(CacheSeedingWebMapService.java:31) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.geoserver.gwc.wms.CachingWebMapService.invoke(CachingWebMapService.java:61) at org.geoserver.gwc.wms.CachingWebMapService.invoke(CachingWebMapService.java:41) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.geoserver.ows.util.RequestObjectLogger.invoke(RequestObjectLogger.java:50) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy104.getMap(Unknown Source) at sun.reflect.GeneratedMethodAccessor310.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.geoserver.ows.Dispatcher.execute(Dispatcher.java:877) at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:265) at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177) at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at org.geoserver.filters.ThreadLocalsCleanupFilter.doFilter(ThreadLocalsCleanupFilter.java:26) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:69) at org.geoserver.wms.animate.AnimatorFilter.doFilter(AnimatorFilter.java:70) at org.geoserver.filters.SpringDelegatingFilter$Chain.doFilter(SpringDelegatingFilter.java:66) at org.geoserver.filters.SpringDelegatingFilter.doFilter(SpringDelegatingFilter.java:41) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.geoserver.platform.AdvancedDispatchFilter.doFilter(AdvancedDispatchFilter.java:37) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74) at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74) at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.geoserver.security.filter.GeoServerAnonymousAuthenticationFilter.doFilter(GeoServerAnonymousAuthenticationFilter.java:51) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74) at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91) at org.geoserver.security.filter.GeoServerBasicAuthenticationFilter.doFilter(GeoServerBasicAuthenticationFilter.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:70) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.geoserver.security.filter.GeoServerSecurityContextPersistenceFilter$1.doFilter(GeoServerSecurityContextPersistenceFilter.java:52) at org.geoserver.security.filter.GeoServerCompositeFilter$NestedFilterChain.doFilter(GeoServerCompositeFilter.java:74) at org.geoserver.security.filter.GeoServerCompositeFilter.doFilter(GeoServerCompositeFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) at org.geoserver.security.GeoServerSecurityFilterChainProxy.doFilter(GeoServerSecurityFilterChainProxy.java:142) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:101) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.geoserver.filters.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:77) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:53) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.geoserver.filters.SessionDebugFilter.doFilter(SessionDebugFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.geoserver.filters.FlushSafeFilter.doFilter(FlushSafeFilter.java:42) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:311) at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:265) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.lang.Thread.run(Thread.java:748) Caused by: org.geoserver.platform.ServiceException: java.lang.RuntimeException: java.io.FileNotFoundException: /mapdata/geoserver_data_dir/data/ArchiveQuery/BrowseImageMosaicStore/BrowseImageMosaicStore.shp (No such file or directory) at org.geoserver.wms.map.RenderedImageMapOutputFormat.directRasterRender(RenderedImageMapOutputFormat.java:1211) at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:359) ... 129 more Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /mapdata/geoserver_data_dir/data/ArchiveQuery/BrowseImageMosaicStore/BrowseImageMosaicStore.shp (No such file or directory) at org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:165) at org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:52) at org.geotools.data.DataUtilities.bounds(DataUtilities.java:2725) at org.geotools.gce.imagemosaic.RasterLayerRequest.computeCoverageBoundingBox(RasterLayerRequest.java:383) at org.geotools.gce.imagemosaic.RasterLayerRequest.<init>(RasterLayerRequest.java:288) at org.geotools.gce.imagemosaic.RasterManager.read(RasterManager.java:1315) at org.geotools.gce.imagemosaic.ImageMosaicReader.read(ImageMosaicReader.java:905) at org.geotools.gce.imagemosaic.ImageMosaicReader.read(ImageMosaicReader.java:886) at org.geoserver.catalog.SingleGridCoverage2DReader.read(SingleGridCoverage2DReader.java:132) at org.geoserver.catalog.CoverageDimensionCustomizerReader.read(CoverageDimensionCustomizerReader.java:234) at org.geoserver.catalog.CoverageDimensionCustomizerReader.read(CoverageDimensionCustomizerReader.java:222) at org.geotools.renderer.lite.gridcoverage2d.GridCoverageReaderHelper.readSingleCoverage(GridCoverageReaderHelper.java:642) at org.geotools.renderer.lite.gridcoverage2d.GridCoverageReaderHelper.readCoverageInEnvelope(GridCoverageReaderHelper.java:371) at org.geotools.renderer.lite.gridcoverage2d.GridCoverageReaderHelper.readCoverages(GridCoverageReaderHelper.java:247) at org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer.renderImage(GridCoverageRenderer.java:690) at org.geotools.renderer.lite.gridcoverage2d.GridCoverageRenderer.renderImage(GridCoverageRenderer.java:639) at org.geoserver.wms.map.RenderedImageMapOutputFormat.directRasterRender(RenderedImageMapOutputFormat.java:1059) ... 130 more Caused by: java.io.FileNotFoundException: /mapdata/geoserver_data_dir/data/ArchiveQuery/BrowseImageMosaicStore/BrowseImageMosaicStore.shp (No such file or directory) at java.io.RandomAccessFile.open0(Native Method) at java.io.RandomAccessFile.open(RandomAccessFile.java:316) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) at org.geotools.data.shapefile.files.ShpFiles.getReadChannel(ShpFiles.java:746) at org.geotools.data.shapefile.shp.ShapefileReader.<init>(ShapefileReader.java:261) at org.geotools.data.shapefile.ShapefileSetManager.openShapeReader(ShapefileSetManager.java:64) at org.geotools.data.shapefile.ShapefileFeatureSource.getReaderInternal(ShapefileFeatureSource.java:375) at org.geotools.data.shapefile.ShapefileFeatureStore.getReaderInternal(ShapefileFeatureStore.java:133) at org.geotools.data.store.ContentFeatureSource.getReader(ContentFeatureSource.java:619) at org.geotools.data.store.ContentFeatureCollection.features(ContentFeatureCollection.java:163) ... 146 more 17 Nov 01:16:35 INFO [geoserver.wms] - Request: getServiceInfo Ashley Mort Sr. Software Engineer 972-638-6317 office 814-571-3039 cell _______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net<mailto:Geoserver-users@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/geoserver-users -- Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://bit.ly/gs-services-us for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions Group phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 333 8128928 https://www.geosolutionsgroup.com/ http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail
_______________________________________________ Geoserver-users mailing list Please make sure you read the following two resources before posting to this list: - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/ - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer Geoserver-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-users
