Hi Alan,
Got VisualVM attached to a firewalled remote Nakamura using Mark Triggs'
procedure. Thanks for your help also.
In answer to your questions, our initial load testing work has two goals:
1) Get in-house Ops experience with Tsung. Jon Felder our Ops Engineer
did the setup and has been running the tests although now Ray and I are
starting to run them ourselves as well.
2) Support Ray's work comparing the performance of the application when
using the embedded Solr server or a standalone Solr server.
Jon Felder will be the Sakai conference next week. If you're going I
hope you two connect and discuss things with Ray as well. I know that
Anthony White is planning a BOF for performance testing there.
Unfortunately, I won't be going but will pick Jon's and Ray's brains
when they get back. I am also aware that Kent has been tasked by
Anthony with providing load testing infrastructure as a community
resource. Maybe our initial work will be useful in this domain.
The initial work on #2 was done using Tsung sessions that use mainly
search triggering URL's to isolate as much as possible the search
performance as Ray as been making many fixes in this area above and
beyond configuring and using the standalone Solr server.
The results from that initial work are somewhat concerning as we are
seeing a lot of server 503 responses when using only a few (4 or more)
concurrent users. We're pretty sure they are caused, under even this
small load, by the fact that the number of "Very Slow Solr Queries" goes
way up with a number of them taking 20 seconds or more to return. Hence
the desire to attach a profiling instrument to get more info on
bottlenecks and the hope that Ray's on-going fixes in this area will
reduce or eliminate them. Ray is currently running some Tsung tests to
compare various Solr caching config settings as he thinks there may be
some low hanging performance wins in that area.
I think that some of performance issues are due to the virtual machine
servers we are using. They may turn out to be under-powered, or
under-optimized, for this application but that is just speculation for now.
I will be doing, in your terms, "burn tests" with VisualVM attached
using both Tsung load tests and the new OAE_model_loader client side
module to stress the system. No results on that yet but we hope that
being able to attach VisualVM will show us the "loudly shouting" problem
areas.
I am sure we, Jon, Ray and I, will be posting results as we go. And I
hope a lot more good info on load and burn tests come out of the
conference as well as good news about all the Solr fixes and optimizations.
I will be in touch regarding your experiences in this area after the
conference. Thanks for your offer of help in that regard.
Best,
John
On 6/6/12 1:13 AM, Berg, Alan wrote:
Hi John and Jonathan,
I hope it is going well with your adventures with JMX. I read my
e-mail written with my thumb on my telephone last night. It was brief
and probably does not contain the details you need. Judging by the on
list help, there is more than one way to get things working. If you
are still having issues I do not mind having a brief Skype call to
compare details.
I am curious to understand your performance test plans around Sakai
OAE. I am occasionally busy in my day Job at UvA with stress testing
Java applications to capture resource leaks etc. Again, you are very
welcome (good for my own education) to compare notes.
Regards,
Alan
Alan Berg
Group Education and Research Services
Central Computer Services
University of Amsterdam
------------------------------------------------------------------------
*From:* Berg, Alan
*Sent:* 05 June 2012 22:27
*To:* John King
*Cc:* Felder, Jonathan
*Subject:* Re: [oae-dev] Problems attaching the VisualVM Java profiler
to a remote firewalled OAE Application
I use jmx on localhost and tunnel with ssh -x
What is the difficulty? It works reliably with jconsole
John King<jo...@media.berkeley.edu> wrote:
Folks,
As part of our ongoing performance testing here at Berkeley, we want
to use the VisualVM Java profiler to watch the app, which is running
remotely behind a firewall, while running load tests. We have run
into a fundamental problem doing this and I am writing to the list to
ask if anyone has solved this particular problem in a simple, easily
configurable manner.
The problem is that we are being hindered by the nature of JMX
(management beans) communication over RMI - the default protocol.
The difficulty is that this protocol uses two ports, one defined by
server configuration and a second port randomly assigned at runtime.
Obviously the second port presents a problem in configuring a
firewall or SSH tunnel as the second port will change when the app is
restarted.
Our Ops Engineer, Jon Felder, and I have been researching solutions to
this problem and have found some possible approaches. - see [1] and [2]
[1] recommends using an alternative communications protocol, JMXMP,
which is JMX over TCP instead of RMI. As this appears to be the
simplest solution, I am going to try this out today and will let the
list know if it works.
[2] involves a fair amount of custom coding based on adapting the
example shown.
A third approach involves parsing the response from the initial
handshake for the randomly assigned port and configuring the SSH
tunnel to use or opening that port in the firewall Jon Felder has made
good progress on doin this but, as mentioned above, it will be unwieldy.
Has anyone on the list solved this particular problem? Any
suggestions as to the easiest solution will be much appreciated.
[1] http://blog.markfeeney.com/2010/10/jmx-through-ssh-tunnel.html
[2]
https://blogs.oracle.com/jmxetc/entry/connecting_through_firewall_using_jmx
--
John King
Applications Programmer
Learning Systems Group
Educational Technology Services
9 Dwinelle Hall - Mail
117 Dwinelle Hall - Office
University of California
Berkeley, CA 94720-2535
Phone: 510-529-5074
Email:jo...@media.berkeley.edu
--
John King
Applications Programmer
Learning Systems Group
Educational Technology Services
9 Dwinelle Hall - Mail
117 Dwinelle Hall - Office
University of California
Berkeley, CA 94720-2535
Phone: 510-529-5074
Email: jo...@media.berkeley.edu
_______________________________________________
oae-dev mailing list
oae-dev@collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/oae-dev
