Regarding the "Too many open files" issue - I took another brief look at that one about a month ago, and couldn't make any further headway. From the perspective of the OS configuration it should be working fine, so it is almost certainly caused by a leak in GeoServer, Jenkins, or Java.
I did find this <https://wiki.jenkins.io/display/JENKINS/I%27m+getting+too+many+open+files+error>, which provides a good reference on how to track down exactly where it is coming from (or at least rule out Jenkins), however debugging it properly is going to require a reasonably large chunk of uninterrupted time (or a bit of luck) in order to catch the server in a state where there are currently too many open files and in a state where there aren't, and I haven't really had that time lately. For the GWC test failure on 2.14 - I can't recall whether that issue came up shortly after the change or later on either. It looks like you've added an ostensible fix <https://github.com/geoserver/geoserver/blame/2.14.x/src/gwc/src/test/java/org/geoserver/gwc/GWCIntegrationTest.java#L931> for the seeding issue on 2.14 and newer, although that evidently didn't quite fix the issue, so I guess there might be something else going on. I'm not really sure what that issue might be, especially since all the other similar tests that one test was based on (e.g. testAutomaticTruncationLayerStyleContentsChange <https://github.com/geoserver/geoserver/blob/2.14.x/src/gwc/src/test/java/org/geoserver/gwc/GWCIntegrationTest.java#L816> () ) still all seem to be passing. Given that, this may actually represent a bug specific to the WFS TransactionListener and how it interacts with GWC (possibly just an additional delay in handling truncation). The next build of the 2.14 job passed, so the issue continues to be sporadic, which is going to make tracking down the root cause difficult. Cheers, Torben On Wed, May 1, 2019 at 12:23 AM Andrea Aime <[email protected]> wrote: > Hi Torben, > great stuff, thanks for fixing it! > > Checking the buid server, the java 11 geoserver builds are stil plagued by > the too many open files issues (although they are intermittent now). > I googled around once more, found a few issues related to QA tools but all > seem fixed in the versions we're using: > https://github.com/google/error-prone/issues/737 > https://github.com/spotbugs/spotbugs/pull/591 > and also one related to mvn assembly: > http://ksoong.org/troubleshooting/descriptor/mvn-too-many-file-open.html > Also, intermittent is probably not the right word, as failures and > successes are clustered, as if there was some sort of global leak > accumulating. Wondering if the bunch of green builds are associated with a > node restart or something similar. > > About the 2.14.x failure, it's one intermittent failure that we have seen > mostly on the windows build server on pull requests, but occasionally pops > up > on linux too: > > testAutomaticTruncationFeatureChange(org.geoserver.gwc.GWCIntegrationTest) > Time elapsed: 2464 sec <<< FAILURE! > java.lang.AssertionError: > Expected: equalToIgnoringCase("MISS") > but: was HIT > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) > at org.junit.Assert.assertThat(Assert.java:865) > at org.junit.Assert.assertThat(Assert.java:832) > at > org.geoserver.gwc.GWCIntegrationTest.testAutomaticTruncationFeatureChange(GWCIntegrationTest.java:936) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) > > > You're the author of the test failing, but I don't recall if the random > failure started happening right away (e.g., it came with the test), or > later down the road (e.g., it popped up as a result of another change). > As far as I remember, when I looked into it, it seemed to be caused by the > seeding tasks being background and thus not reliable to do their job within > a given timeframe in a testing environment > > Cheers > Andrea > > On Tue, Apr 30, 2019 at 11:35 PM Torben Barsballe < > [email protected]> wrote: > >> Okay, I think I've got this working now - tests are passing again for >> master and 2.15 (2.14 appears to have an unrelated failure in GWC). >> >> Torben >> >> On Tue, Apr 30, 2019 at 9:13 AM Torben Barsballe < >> [email protected]> wrote: >> >>> That looks like it was caused by me moving over the PostGIS config to >>> build-slave-01 (see earlier emails on the failing app-schema-online tests) >>> >>> As far as I could tell, prior to that there wasn't any version of GDAL >>> installed (so I don't think any of the GDAL/OGR tests were even running), >>> and I think the PostGIS install pulled in a fairly old version, so that >>> could explain the failure. >>> I can take a look at getting a working version of GDAL installed on the >>> server. >>> >>> Torben >>> >>> On Tue, Apr 30, 2019 at 3:54 AM Andrea Aime < >>> [email protected]> wrote: >>> >>>> Hi, >>>> 2.15.x has been failing for a number of builds, starting on a seemingly >>>> unrelated change: >>>> https://build.geoserver.org/job/geoserver-2.15.x/83/ >>>> >>>> Checking the failure, it seems like a system installation problem on >>>> the server, see this: >>>> >>>> Tests run: 10, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 1.934 sec >>>> <<< FAILURE! >>>> testSimpleCSV(org.geoserver.wfs.response.Ogr2OgrFormatTest) Time elapsed: >>>> 978 sec <<< ERROR! >>>> org.geoserver.platform.ServiceException: Exception occurred during output >>>> generation >>>> at >>>> org.geoserver.wfs.response.Ogr2OgrOutputFormat.write(Ogr2OgrOutputFormat.java:307) >>>> at >>>> org.geoserver.wfs.WFSGetFeatureOutputFormat.write(WFSGetFeatureOutputFormat.java:200) >>>> at >>>> org.geoserver.wfs.response.Ogr2OgrFormatTest.testSimpleCSV(Ogr2OgrFormatTest.java:196) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>> at >>>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) >>>> at >>>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) >>>> at >>>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) >>>> at >>>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) >>>> at >>>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) >>>> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) >>>> at >>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) >>>> at >>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) >>>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) >>>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) >>>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) >>>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) >>>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) >>>> at org.junit.runners.ParentRunner.run(ParentRunner.java:363) >>>> at >>>> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) >>>> at >>>> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) >>>> at >>>> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>> at >>>> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) >>>> at >>>> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) >>>> at >>>> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) >>>> at >>>> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) >>>> at >>>> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)*Caused >>>> by: java.io.IOException: ogr2ogr did not terminate successfully, exit >>>> code 1. Was trying to run: [ogr2ogr, -f, CSV, -a_srs, >>>> /tmp/ogrtmpin0.367406913451224/srs8240695920266311468wkt, >>>> /tmp/ogrtmpout0.18567878589886866/Buildings.csv, >>>> /tmp/ogrtmpin0.367406913451224/Buildings.gml] >>>> Resulted in: >>>> >>>> ERROR 1: Unable to create Xerces C++ or Expat based GML reader, Xerces or >>>> Expat support >>>> not configured into GDAL/OGR. >>>> ERROR 1: File /tmp/ogrtmpin0.367406913451224/Buildings.gml appears to be >>>> GML but the GML reader can't >>>> be instantiated, likely because Xerces or Expat support wasn't >>>> configured in. >>>> FAILURE:* >>>> >>>> Maybe some updates occurred April 29th that broke the ogr2ogr >>>> installation? >>>> >>>> Cheers >>>> Andrea >>>> >>>> == >>>> >>>> GeoServer Professional Services from the experts! Visit >>>> http://goo.gl/it488V for more information. == Ing. Andrea Aime >>>> @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 >>>> Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 >>>> 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it >>>> ------------------------------------------------------- *Con >>>> riferimento alla normativa sul trattamento dei dati personali (Reg. UE >>>> 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si >>>> precisa che ogni circostanza inerente alla presente email (il suo >>>> contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è >>>> riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il >>>> messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra >>>> operazione è illecita. Le sarei comunque grato se potesse darmene notizia. >>>> This email is intended only for the person or entity to which it is >>>> addressed and may contain information that is privileged, confidential or >>>> otherwise protected from disclosure. We remind that - as provided by >>>> European Regulation 2016/679 “GDPR” - copying, dissemination or use of this >>>> e-mail or the information herein by anyone other than the intended >>>> recipient is prohibited. If you have received this email by mistake, please >>>> notify us immediately by telephone or e-mail.* >>>> >>> > > -- > > Regards, Andrea Aime == GeoServer Professional Services from the experts! > Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime > @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 > Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 > 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it > ------------------------------------------------------- *Con riferimento > alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - > Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni > circostanza inerente alla presente email (il suo contenuto, gli eventuali > allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i > destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per > errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le > sarei comunque grato se potesse darmene notizia. This email is intended > only for the person or entity to which it is addressed and may contain > information that is privileged, confidential or otherwise protected from > disclosure. We remind that - as provided by European Regulation 2016/679 > “GDPR” - copying, dissemination or use of this e-mail or the information > herein by anyone other than the intended recipient is prohibited. If you > have received this email by mistake, please notify us immediately by > telephone or e-mail.* >
_______________________________________________ Geoserver-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geoserver-devel
