Hi Keith, On one node it has been failing since at least the 7th, but quite likely since its run on the 3rd. There looks to be 3 commits that would have been in that run on the 3rd, r1688825, r1688862, and r1688958, the last of which was for QPID-6613 test log changes.
Another build has run after your revert and just passed on that node, which is interesting in itself since there didn't seem to be any change in behaviour resulting from the original commit on the 13th, so some other changes must be interacting too. In runs before and after your commit on the 13th the first test to fail was LocalTransactionTest.testStoreEnqueuesCausesExceptions due to being unable to load some classes, with the following tests mostly doing the same except those in TCPandSSLTransportTest (which look to be the last to run), with its tests instead failing to open a Selector during SelectorThread init while starting the NetworkConnectionScheduler. If it happens again I'll let you know. Not sure what details I can get from the box other than the logs, anything specific you were thinking of? Robbie On 16 July 2015 at 09:03, Keith W <[email protected]> wrote: > Hi Robbie > > QPID-6613 has been making some changes to test logging. One area of change > is the generation of per-test log files which is now handled by Logback's > SiftingAppender. > > I made a commit (r1690741) on Monday (13th) (to help investigate a > seemingly new sporadical issue where the Java Broker was failing to close > down completely because the Selector-Port thread remains running). This > commit would mean the JVM running the tests would hold on to more file > handles for longer. I have reverted this change as I expect that it is > contributing to the issue you are seeing. I have just reverted this change. > > I wonder if the issue you were observing from around 2nd/3rd July is the > same as my Selector-Port thread issue. If you see a recurrence, can you > post some details? I am having problems reproducing on the kit I have > available to me. > > cheers, Keith. > > > On 15 July 2015 at 15:24, Robbie Gemmell <[email protected]> wrote: > >> Hi folks, >> >> Irina mentioned seeing some failures for the java build in a CI system >> she runs. It has been failing consistently on one box, and >> intermittent on another. Taking a quick peek it seems like the root >> cause are the "Too many open files" exceptions happening at various >> points. One example is failing to open a Selector, another is Logback >> failing to create a file for its appender during setUp, and another >> was failing to find the class under test. The failures start part way >> through the broker-core module. >> >> It isn't clear exactly when the issue started as some of the >> intervening reports have already been removed, but it seems like the >> last successfull build was from July 2nd, and the first with issues >> *may* have been on July 3rd as a result. There looks to have been a >> bunch of logging related changes around then and subsequently, so I >> thought I'd mention it in case it does signal an issue (it may >> alternatively indicate a need for increasing the open files limit). >> FWIW, the build passed locally when I tried and seems as happy as >> usual on the ASF Jenkins nodes. >> >> Robbie >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
