Happy to see JMeter helped you diagnose an issue in your architecture. And many thanks for feedback.
Regards Philippe M. @philmdot On Thu, Oct 30, 2014 at 3:31 PM, Brian Devaney <[email protected]> wrote: > The problem is definitely the load balancer. I added some more logging and > monitoring and found the cause of the problem with JMeter. It does not > happen at the same time every night, it's completely random and occurred 5 > times last night. A segmentation fault is reported in the load balancer > error log at precisely the second after JMeter sufferes its one second > error. Now I just need to track down what is causing the segmentation > fault. > > Thanks for all your help. > > > On Wed, Oct 29, 2014 at 7:03 PM, Shmuel Krakower <[email protected]> > wrote: > > > Sounds like a problem with your load balancer. > > > > 1. Is it the same timestamp every night or happen only once per night, > > every night on a different time? > > 2. Do you have any error logs or monitoring on your load balancer? > > 3. Do you have any other metrics on the load balancer? > > 4. Could you try running similar tests in parallel from another host and > > see if both tests result with this behavior on the exact time frame? > > > > Best > > > > www.beatsoo.org - free application performance monitoring from world > wide > > locations. > > On Oct 29, 2014 10:31 PM, "Brian Devaney" <[email protected]> > wrote: > > > > > I am not sure that this is even a JMeter problem, but I will post > anyway > > > because JMeter seems to be the only way I can cause and detect the > > problem. > > > > > > I have a number of JMeter scripts that get run in a test overnight. > Each > > > script is run three times with increasing numbers of users each time. > > The > > > script is hitting a cluster of two nodes with a load balancer in front. > > > The scripts run perfectly over the course of the night, except in one > or > > > two cases, a number of errors will occur, which is where I get the > > > NoHttpResponseException. At what seems to be a completely random > moment, > > > for one second, all requests that are getting sent to the load balancer > > > will get this error. According to the jmeter.log it always seems to be > > > within one second. It will not, for example, start halfway through > > > 10:05:19 and finish halfway through 10:05:20. Every log entry with the > > > error will be within the same second of 10:05:19. > > > > > > The stack trace that gets generated from one of these errors is: > > > org.apache.commons.httpclient.NoHttpResponseException: The server > > > 10.10.2.179 failed to respond > > > at > > > > > > > > > org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1976) > > > at > > > > > > > > > org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) > > > at > > > > > > > > > org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) > > > at > > > > > > > > > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) > > > at > > > > > > > > > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > > > at > > > > > > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > > > at > > > > > > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > > > at > > > > > > > > > org.apache.jmeter.protocol.http.sampler.SoapSampler.sample(SoapSampler.java:272) > > > at > > > > > > > > > org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1105) > > > at > > > > > > > > > org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1094) > > > at > > > > > > > > > org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:429) > > > at > org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257) > > > at java.lang.Thread.run(Thread.java:722) > > > > > > > > > As part of my testing, I have turned off any timeouts in JMeter that I > > can, > > > or made them long enough that I will get a response eventually. Both > > nodes > > > in the cluster, the load balancer and the server running the JMeter > test > > > all have ntp installed in order to keep their clocks in sync. As a > test > > I > > > ran one of my JMeter scripts while monitoring the time drift to see if > > the > > > clock getting reset during the test could be the culprit. The > exception > > > did not show up on that run. Another possibility was that the cluster > > > could not handle the load, but in my tests, I have seen the case where > a > > > script with 300 users has the error and the same script with 500 users > > > works perfectly with no configuration change other than the number of > > users > > > running the script. > > > > > > Like I said, this may be outside of a JMeter problem, but if anyone has > > any > > > ideas, I would greatly appreciate it. Thank you. > > > > > > -- > > > > > > -- > > > Brian Devaney > > > DocFinity® by Optical Image Technology, Inc. > > > 100 Oakwood Avenue, State College, PA 16803 > > > ph: 814.238.0006 ext. 274 > > > fax: 814.238.0011 > > > mailto:[email protected] > > > http://www.docfinity.com/ > > > > > > > > > DISCLAIMER: This email may contain proprietary information, some or all > > > of which may be legally privileged. It is for the intended recipient > > > only. If an addressing or transmission error has misdirected this > email, > > > please notify the author by replying to this email. If you are not the > > > intended recipient, you may not use, disclose, distribute, copy, print, > > > or rely on this email. > > > > > > > > > -- > > -- > Brian Devaney > DocFinity® by Optical Image Technology, Inc. > 100 Oakwood Avenue, State College, PA 16803 > ph: 814.238.0006 ext. 274 > fax: 814.238.0011 > mailto:[email protected] > http://www.docfinity.com/ > > > DISCLAIMER: This email may contain proprietary information, some or all > of which may be legally privileged. It is for the intended recipient > only. If an addressing or transmission error has misdirected this email, > please notify the author by replying to this email. If you are not the > intended recipient, you may not use, disclose, distribute, copy, print, > or rely on this email. > -- Cordialement. Philippe Mouawad.
