On 07.10.2017 00:05, Jerry Malcolm wrote:
On 10/6/2017 5:15 AM, André Warnier (tomcat) wrote:
On 06.10.2017 11:57, Suvendu Sekhar Mondal wrote:
On Fri, Oct 6, 2017 at 2:06 AM, Jerry Malcolm <techst...@malcolms.com> wrote:
I am running TC 8.0 on WinServer8 on a commercially hosted platform with a
WAMP environment. I am running around 10 virtual hosts. 2 hosts are
dedicated to JSPWiki. The other 8 are running variations of the same custom
application with around 10-15 individual webapps each.
When I am running 7 of the 10 hosts, performance is great. I get JSP
response time under a second. But when I add just a couple of more of the
hosts, my page response time on all of the apps goes from an acceptable
under a second to horrible at around 15-20 seconds per page. It doesn't
seem to be a specific host causing the problem. Reducing overall hosts in
any order makes the problem go away.
I've looked at the the processor utilization during the good times and slow
times, and don't see a significant difference. I have 16GB of memory, and
it consistently shows about 35% utilization. I also checked mySQL response
time, and the per-query SQL response does not vary. So it doesn't appear to
be a db problem.
I suspect there is some TC configuration parameter such as heap, etc that I
need to tweak. (But I'm not getting OutOfMemory errors). But I don't know
which one, and I don't know a formula to use to figure out what to set it
to. So I just need a little education. What tools can I use to help me
figure out what is going south slowing everything to a crawl when the extra
hosts come online? And what parameters should I be looking at (and how
should I calculate the proper values based on number of hosts)?
Suggestions?
Interesting problem. Couple of questions:
Is the slowness sporadic or persistent? What happens when you again
shutdown some of the hosts? Does response time comes back to normal?
Is all transactions are slow or some of them which are routed to
specific host(s)?
Are you fronting Tomcat instances with Apache? If yes, then please
post BalancerMember configuration of the Tomcat cluster here.
Taking the OP's observations above at face value, I think that first thing that
I would
do, is to configure the Java JVM that runs this tomcat, to do some logging of
the
Garbage Collection events. And then I would look at that logfile, and find out
if there
is some significant difference (in frequency of GC, or in the mounts of memory
mentioned
for each event), depending on the number of Hosts enabled.
You would have to look at the appropriate command-line switches to use for the
particular JVM being used, for how to set this up.
And then (since this seems to be on a Windows platform, running tomcat as a
service),
you should use the "tomcat(version)w.exe" program (a GUI), to set these
parameters
properly.
See e.g.
https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html#Tomcat_service_application
--> "Tomcat monitor application"
Thanks for the responses. Answers to several questions in responses:
1) The 16GB is physical OS memory, not JVM memory
Ok. And how much /is/ the JVM memory ? (size of the Heap e.g.).
(for this, you may need to examine the command-line switches used for this JVM, using the
above-mentioned tomcat8w.exe.)
2) Re: 'balancermember config'.... I am running one instance of Apache with
mod_jk to one
one instance of Tomcat containing all of the 10 virtual hosts. I'm not
familiar with
'balancermember'. But it sounds like something to do with a cluster config.
If it still
applies to my config, let me know where I can learn more about it.
3) I'll look into turning on garbage collection logging
4) The problem is constant when I have 'too many' hosts. And the problem is
pervasive to
all running hosts. I've found the magic number is 7 hosts. 7 hosts and life is
good. 8+
hosts and everything clogs up big-time.
I failed to mention that none of these host sites are anywhere close to
'high-volume'.
When a user hits the site, there will be flurry of page activity for a few
minutes. But
I'd say each of the sites averages one or two user hits per hour on average.
So bringing
sites 8, 9, and 10 online is not bringing massive additional
tomcat/mySQL/network
activity. It's basically bringing 3 more 'effectively dormant' web sites
online. This
seems to me that it's got to have something to with the size of the basic
inactive
footprint of each host.
Yes, that's what your observations would tend to suggest. And that's why I was
recommending to set up, and have a look at, the GC logs.
Apart from showing how often GC runs happen (with 7 hosts, and beyond), they will also
show important parameters about the allocated memory.
You have not indicated so far which JVM you are using, nor its startup
parameters.
It could be that some exotic memory-related parameter there introduces a constraint which
hits precisely at the 7-8 Host boundary in your case.
(There are better and more precise ways to find out about memory usage, but they are also
a bit more complicated to set up; that's why I am recommending the GC logging first).
To look at this in another way : I am quite sure that there are a lot of people out there
running tomcat's with a lot more than 7 Hosts (and a lot more traffic than what you say
you have). If there was something within the tomcat code itself which does not scale
properly beyond 7 Hosts or so, this support list would be swamped with reports about it.
It isn't. Ergo, it is unlikely; and thus, it has to be the influence of something outside
tomcat itself. And JVM memory settings and usage would be a convenient first-line suspect.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
