Thanks. I'll set the rig up again and run some volume through with these changes.
Jon On Mon, Jan 7, 2019 at 8:46 PM Doychin Bondzhev <[email protected]> wrote: > Hi Jon, > > Please check this PR: > https://github.com/apache/tomee/pull/360 > > We can discuss the details there. > > I also found this bug report which is still open: > https://issues.apache.org/jira/browse/TOMEE-2257 > > Is that the problem you describe? > > On 7.1.2019 г. 18:51, jgallimore [via TomEE & OpenEJB] wrote: > > This test was a pain to write, and I am grateful for the feedback. Please > do send a PR and I'll take a look. > > Some background to the problem here is that the the AutoConnectionTracker > failed in some circumstances when servers were under heavy load with lots > of GC happening - there appeared to be a timing issue where a resource > that > was actually in use was being forcefully closed by the auto connection > tracker, but because something else still had a hold of it, it ended up > going back in the pool and a broken resource being served up to the next > thing that requested it. The requests for GC were quite key in reproducing > the actual issue originally, so removing them might not actually be the > right approach. > > It was particularly hard to reproduce and pin down and I ended up going > through something like 40GB of heap dumps and logs to figure out what was > happening. Previously there was no test, which made this even harder to > pin > down. I'm definitely happy to explore some options to make this more > solid. > > Jon > > On Sun, Jan 6, 2019 at 8:28 PM Doychin Bondzhev <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=4687328&i=0>> > wrote: > > > I use IDE to run test multiple times. It failed only once here. > > > > I also did some changes in the test code. > > > > I removed the line > > > > > > > https://github.com/apache/tomee/blob/7318a1db025878b3f1f56587c60b47a14d55ef68/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L162 > > > > This made test many times faster. It is not needed. This tests non > leaking > > business method so we expect everything to be cleared. > > > > Also I removed the all the code that waits for log messages. I left only > > the line > > > > assertEquals(message, times, logCapture.find(message).size()); > > > > > > There is no need to wait for anything. All the waiting is already done > for > > the first and second tests in > > > > runTest method > > > > Regarding > > > > System.gc(); > > > > > > It is needed only for the tests that leak so I removed it from all other > > tests. > > > > Third and fourth test are run in the main thread and there is no need to > > wait before processing the log to. > > You can see the change here : > > > > > https://github.com/doychin/tomee/commit/76e090b482825682a859d6a1637f572d8128f30a > > > > In case you like it I can create PR. > > > > On 6.1.2019 г. 19:47, David Blevins-2 [via TomEE & OpenEJB] wrote: > > > > > On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email] > > <http:///user/SendEmail.jtp?type=node&node=4687291&i=0>> wrote: > > > > > > I would like to investigate this issue: > > https://issues.apache.org/jira/browse/TOMEE-2403 > > > > > > Where I can get more detailed error log regarding this issue? > > > > > > I tried to run the test locally but it works. > > > > Here's a technique I use sometimes to check random failures: > > > > $ for n in {1..50}; do mvn clean install > > -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done > > > > So far 3 out 17 have failed. Debugging them will still be a pain in the > > butt, but at least there's some hope. > > > > > > My shallow observation is that the approach taken in the test itself > looks > > conspicuous. Particularly, I'd say this test is not "correct" until it > can > > pass without the System.gc() statements. > > > > - > > > https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197 > > > > More importantly, this method is using thread sleeps and retries to wait > > for log output which *will* cause random failures. We should have a > > discussion on if there is a better way to make this more deterministic. > > > > - > > > https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216 > > > > Testing log output can work, however the test logic is ultimately "if > you > > do or don't see this message in n seconds, we're good." It needs to be > "if > > you do or don't see this message *when the test is over*, we're good." > > > > Since it's looking in log output, one way to do that might be to simply > > log a "test complete" statement (perhaps with a random number on it), > and > > then wait for that in the log output. > > > > > > -David > > > > > > > > > > > > *signature.asc* (499 bytes) Download Attachment > > < > http://tomee-openejb.979440.n4.nabble.com/attachment/4687291/0/signature.asc> > > > > > > > ------------------------------ > > If you reply to this email, your message will be added to the discussion > > below: > > > > > http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687291.html > > To start a new topic under TomEE Dev, email > > [hidden email] <http:///user/SendEmail.jtp?type=node&node=4687328&i=1> > > To unsubscribe from TomEE Dev, click here > > < > > . > > NAML > > < > http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > > > > > -- > > Doychin Bondzhev > > dSoft-Bulgaria Ltd. > > PowerPro - billing & provisioning solution for Service providershttp:// > www.dsoft-bg.com/ > > Mobile: +359888243116 > > > > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687328.html > To start a new topic under TomEE Dev, email > [email protected] > To unsubscribe from TomEE Dev, click here > <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=982480&code=ZG95Y2hpbkBkc29mdC1iZy5jb218OTgyNDgwfC0xNTI2MjM5MTA3> > . > NAML > <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > -- > Doychin Bondzhev > dSoft-Bulgaria Ltd. > PowerPro - billing & provisioning solution for Service > providershttp://www.dsoft-bg.com/ > Mobile: +359888243116 > >
