Is apache reverse proxy in your test stack? If so, there may be some tuning necessary there to deal with low concurrency issues. L
On Jun 6, 2012, at 4:29 PM, John King wrote: > Hi Alan, > > Got VisualVM attached to a firewalled remote Nakamura using Mark Triggs' > procedure. Thanks for your help also. > > In answer to your questions, our initial load testing work has two goals: > 1) Get in-house Ops experience with Tsung. Jon Felder our Ops Engineer did > the setup and has been running the tests although now Ray and I are starting > to run them ourselves as well. > 2) Support Ray's work comparing the performance of the application when using > the embedded Solr server or a standalone Solr server. > > Jon Felder will be the Sakai conference next week. If you're going I hope > you two connect and discuss things with Ray as well. I know that Anthony > White is planning a BOF for performance testing there. Unfortunately, I > won't be going but will pick Jon's and Ray's brains when they get back. I am > also aware that Kent has been tasked by Anthony with providing load testing > infrastructure as a community resource. Maybe our initial work will be > useful in this domain. > > The initial work on #2 was done using Tsung sessions that use mainly search > triggering URL's to isolate as much as possible the search performance as Ray > as been making many fixes in this area above and beyond configuring and using > the standalone Solr server. > > The results from that initial work are somewhat concerning as we are seeing a > lot of server 503 responses when using only a few (4 or more) concurrent > users. We're pretty sure they are caused, under even this small load, by > the fact that the number of "Very Slow Solr Queries" goes way up with a > number of them taking 20 seconds or more to return. Hence the desire to > attach a profiling instrument to get more info on bottlenecks and the hope > that Ray's on-going fixes in this area will reduce or eliminate them. Ray is > currently running some Tsung tests to compare various Solr caching config > settings as he thinks there may be some low hanging performance wins in that > area. > > I think that some of performance issues are due to the virtual machine > servers we are using. They may turn out to be under-powered, or > under-optimized, for this application but that is just speculation for now. > > I will be doing, in your terms, "burn tests" with VisualVM attached using > both Tsung load tests and the new OAE_model_loader client side module to > stress the system. No results on that yet but we hope that being able to > attach VisualVM will show us the "loudly shouting" problem areas. > > I am sure we, Jon, Ray and I, will be posting results as we go. And I hope a > lot more good info on load and burn tests come out of the conference as well > as good news about all the Solr fixes and optimizations. > > I will be in touch regarding your experiences in this area after the > conference. Thanks for your offer of help in that regard. > > Best, > > John > > On 6/6/12 1:13 AM, Berg, Alan wrote: >> >> Hi John and Jonathan, >> >> I hope it is going well with your adventures with JMX. I read my e-mail >> written with my thumb on my telephone last night. It was brief and probably >> does not contain the details you need. Judging by the on list help, there is >> more than one way to get things working. If you are still having issues I do >> not mind having a brief Skype call to compare details. >> >> I am curious to understand your performance test plans around Sakai OAE. I >> am occasionally busy in my day Job at UvA with stress testing Java >> applications to capture resource leaks etc. Again, you are very welcome >> (good for my own education) to compare notes. >> >> Regards, >> >> Alan >> >> >> Alan Berg >> >> Group Education and Research Services >> Central Computer Services >> University of Amsterdam >> From: Berg, Alan >> Sent: 05 June 2012 22:27 >> To: John King >> Cc: Felder, Jonathan >> Subject: Re: [oae-dev] Problems attaching the VisualVM Java profiler to a >> remote firewalled OAE Application >> >> I use jmx on localhost and tunnel with ssh -x >> >> What is the difficulty? It works reliably with jconsole >> >> John King <jo...@media.berkeley.edu> wrote: >> >> Folks, >> >> As part of our ongoing performance testing here at Berkeley, we want to use >> the VisualVM Java profiler to watch the app, which is running remotely >> behind a firewall, while running load tests. We have run into a fundamental >> problem doing this and I am writing to the list to ask if anyone has solved >> this particular problem in a simple, easily configurable manner. >> >> The problem is that we are being hindered by the nature of JMX (management >> beans) communication over RMI - the default protocol. The difficulty is >> that this protocol uses two ports, one defined by server configuration and a >> second port randomly assigned at runtime. Obviously the second port >> presents a problem in configuring a firewall or SSH tunnel as the second >> port will change when the app is restarted. >> >> Our Ops Engineer, Jon Felder, and I have been researching solutions to this >> problem and have found some possible approaches. - see [1] and [2] >> >> [1] recommends using an alternative communications protocol, JMXMP, which is >> JMX over TCP instead of RMI. As this appears to be the simplest solution, >> I am going to try this out today and will let the list know if it works. >> [2] involves a fair amount of custom coding based on adapting the example >> shown. >> >> A third approach involves parsing the response from the initial handshake >> for the randomly assigned port and configuring the SSH tunnel to use or >> opening that port in the firewall Jon Felder has made good progress on doin >> this but, as mentioned above, it will be unwieldy. >> >> Has anyone on the list solved this particular problem? Any suggestions as >> to the easiest solution will be much appreciated. >> >> [1] http://blog.markfeeney.com/2010/10/jmx-through-ssh-tunnel.html >> [2] >> https://blogs.oracle.com/jmxetc/entry/connecting_through_firewall_using_jmx >> -- >> John King >> Applications Programmer >> Learning Systems Group >> Educational Technology Services >> 9 Dwinelle Hall - Mail >> 117 Dwinelle Hall - Office >> University of California >> Berkeley, CA 94720-2535 >> Phone: 510-529-5074 >> Email: jo...@media.berkeley.edu >> > > > -- > John King > Applications Programmer > Learning Systems Group > Educational Technology Services > 9 Dwinelle Hall - Mail > 117 Dwinelle Hall - Office > University of California > Berkeley, CA 94720-2535 > Phone: 510-529-5074 > Email: jo...@media.berkeley.edu > > _______________________________________________ > oae-dev mailing list > oae-dev@collab.sakaiproject.org > http://collab.sakaiproject.org/mailman/listinfo/oae-dev
_______________________________________________ oae-dev mailing list oae-dev@collab.sakaiproject.org http://collab.sakaiproject.org/mailman/listinfo/oae-dev
