Re: Performance settings for Multiple Hosts

[tomcat]

On 07.10.2017 00:05, Jerry Malcolm wrote:

On 10/6/2017 5:15 AM, André Warnier (tomcat) wrote:

On 06.10.2017 11:57, Suvendu Sekhar Mondal wrote:

On Fri, Oct 6, 2017 at 2:06 AM, Jerry Malcolm  wrote:

I am running TC 8.0 on WinServer8 on a commercially hosted platform with a
WAMP environment.   I am running around 10 virtual hosts.   2 hosts are
dedicated to JSPWiki.  The other 8 are running variations of the same custom
application with around 10-15 individual webapps each.

When I am running 7 of the 10 hosts, performance is great.  I get JSP
response time under a second.  But when I add just a couple of more of the
hosts, my page response time on all of the apps goes from an acceptable
under a second to horrible at around 15-20 seconds per page. It doesn't
seem to be a specific host causing the problem.  Reducing overall hosts in
any order makes the problem go away.

I've looked at the the processor utilization during the good times and slow
times, and don't see a significant difference.  I have 16GB of memory, and
it consistently shows about 35% utilization.  I also checked mySQL response
time, and the per-query SQL response does not vary.  So it doesn't appear to
be a db problem.

I suspect there is some TC configuration parameter such as heap, etc that I
need to tweak. (But I'm not getting OutOfMemory errors).  But I don't know
which one, and I don't know a formula to use to figure out what to set it
to.  So I just need a little education.  What tools can I use to help me
figure out what is going south slowing everything to a crawl when the extra
hosts come online?  And what parameters should I be looking at (and how
should I calculate the proper values based on number of hosts)?

Suggestions?


Interesting problem. Couple of questions:
Is the slowness sporadic or persistent? What happens when you again
shutdown some of the hosts? Does response time comes back to normal?
Is all transactions are slow or some of them which are routed to
specific host(s)?
Are you fronting Tomcat instances with Apache? If yes, then please
post BalancerMember configuration of the Tomcat cluster here.



Taking the OP's observations above at face value, I think that first thing that 
I would
do, is to configure the Java JVM that runs this tomcat, to do some logging of 
the
Garbage Collection events. And then I would look at that logfile, and find out 
if there
is some significant difference (in frequency of GC, or in the mounts of memory 
mentioned
for each event), depending on the number of Hosts enabled.

You would have to look at the appropriate command-line switches to use for the
particular JVM being used, for how to set this up.
And then (since this seems to be on a Windows platform, running tomcat as a 
service),
you should use the "tomcat(version)w.exe" program (a GUI), to set these 
parameters
properly.
See e.g.
https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html#Tomcat_service_application


--> "Tomcat monitor application"


Thanks for the responses. Answers to several questions in responses:

1) The 16GB is physical OS memory, not JVM memory


Ok. And how much /is/ the JVM memory ? (size of the Heap e.g.).
(for this, you may need to examine the command-line switches used for this JVM, using the 
above-mentioned tomcat8w.exe.)



2) Re: 'balancermember config' I am running one instance of Apache with 
mod_jk to one
one instance of Tomcat containing all of the 10 virtual hosts.  I'm not 
familiar with
'balancermember'.  But it sounds like something to do with a cluster config.  
If it still
applies to my config, let me know where I can learn more about it.
3) I'll look into turning on garbage collection logging
4) The problem is constant when I have 'too many' hosts.  And the problem is 
pervasive to
all running hosts. I've found the magic number is 7 hosts.  7 hosts and life is 
good.  8+
hosts  and everything clogs up big-time.

I failed to mention that none of these host sites are anywhere close to 
'high-volume'.
When a user hits the site, there will be flurry of page activity for a few 
minutes.  But
I'd say each of the sites averages one or two user hits per hour on average.  
So bringing
sites 8, 9, and 10 online is not bringing massive additional 
tomcat/mySQL/network
activity.  It's basically bringing 3 more 'effectively dormant' web sites 
online.  This
seems to me that it's got to have something to with the size of the basic 
inactive
footprint of each host.

Yes, that's what your observations would tend to suggest. And that's why I was 
recommending to set up, and have a look at, the GC logs.
Apart from showing how often GC runs happen (with 7 hosts, and beyond), they will also 
show important parameters about the allocated memory.

You have not indicated so far which JVM you are using, nor its startup 
parameters.
It could be that some exotic memory-related parameter there introduces a constraint which 
hits precisely at the 7-8 Host boundary in your 

Re: Performance settings for Multiple Hosts

[Jerry Malcolm]

On 10/6/2017 5:15 AM, André Warnier (tomcat) wrote:

On 06.10.2017 11:57, Suvendu Sekhar Mondal wrote:
On Fri, Oct 6, 2017 at 2:06 AM, Jerry Malcolm 
 wrote:
I am running TC 8.0 on WinServer8 on a commercially hosted platform 
with a

WAMP environment.   I am running around 10 virtual hosts.   2 hosts are
dedicated to JSPWiki.  The other 8 are running variations of the 
same custom

application with around 10-15 individual webapps each.

When I am running 7 of the 10 hosts, performance is great.  I get JSP
response time under a second.  But when I add just a couple of more 
of the

hosts, my page response time on all of the apps goes from an acceptable
under a second to horrible at around 15-20 seconds per page. It doesn't
seem to be a specific host causing the problem.  Reducing overall 
hosts in

any order makes the problem go away.

I've looked at the the processor utilization during the good times 
and slow
times, and don't see a significant difference.  I have 16GB of 
memory, and
it consistently shows about 35% utilization.  I also checked mySQL 
response
time, and the per-query SQL response does not vary.  So it doesn't 
appear to

be a db problem.

I suspect there is some TC configuration parameter such as heap, etc 
that I
need to tweak. (But I'm not getting OutOfMemory errors).  But I 
don't know
which one, and I don't know a formula to use to figure out what to 
set it
to.  So I just need a little education.  What tools can I use to 
help me
figure out what is going south slowing everything to a crawl when 
the extra

hosts come online?  And what parameters should I be looking at (and how
should I calculate the proper values based on number of hosts)?

Suggestions?


Interesting problem. Couple of questions:
Is the slowness sporadic or persistent? What happens when you again
shutdown some of the hosts? Does response time comes back to normal?
Is all transactions are slow or some of them which are routed to
specific host(s)?
Are you fronting Tomcat instances with Apache? If yes, then please
post BalancerMember configuration of the Tomcat cluster here.



Taking the OP's observations above at face value, I think that first 
thing that I would do, is to configure the Java JVM that runs this 
tomcat, to do some logging of the Garbage Collection events. And then 
I would look at that logfile, and find out if there is some 
significant difference (in frequency of GC, or in the mounts of memory 
mentioned for each event), depending on the number of Hosts enabled.


You would have to look at the appropriate command-line switches to use 
for the particular JVM being used, for how to set this up.
And then (since this seems to be on a Windows platform, running tomcat 
as a service), you should use the "tomcat(version)w.exe" program (a 
GUI), to set these parameters properly.

See e.g.
https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html#Tomcat_service_application 



--> "Tomcat monitor application"


Thanks for the responses. Answers to several questions in responses:

1) The 16GB is physical OS memory, not JVM memory
2) Re: 'balancermember config' I am running one instance of Apache 
with mod_jk to one one instance of Tomcat containing all of the 10 
virtual hosts.  I'm not familiar with 'balancermember'.  But it sounds 
like something to do with a cluster config.  If it still applies to my 
config, let me know where I can learn more about it.

3) I'll look into turning on garbage collection logging
4) The problem is constant when I have 'too many' hosts.  And the 
problem is pervasive to all running hosts. I've found the magic number 
is 7 hosts.  7 hosts and life is good.  8+ hosts  and everything clogs 
up big-time.


I failed to mention that none of these host sites are anywhere close to 
'high-volume'.  When a user hits the site, there will be flurry of page 
activity for a few minutes.  But I'd say each of the sites averages one 
or two user hits per hour on average.  So bringing sites 8, 9, and 10 
online is not bringing massive additional tomcat/mySQL/network 
activity.  It's basically bringing 3 more 'effectively dormant' web 
sites online.  This seems to me that it's got to have something to with 
the size of the basic inactive footprint of each host.


Jerry


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance settings for Multiple Hosts

[tomcat]

On 06.10.2017 11:57, Suvendu Sekhar Mondal wrote:

On Fri, Oct 6, 2017 at 2:06 AM, Jerry Malcolm  wrote:

I am running TC 8.0 on WinServer8 on a commercially hosted platform with a
WAMP environment.   I am running around 10 virtual hosts.   2 hosts are
dedicated to JSPWiki.  The other 8 are running variations of the same custom
application with around 10-15 individual webapps each.

When I am running 7 of the 10 hosts, performance is great.  I get JSP
response time under a second.  But when I add just a couple of more of the
hosts, my page response time on all of the apps goes from an acceptable
under a second to horrible at around 15-20 seconds per page.  It doesn't
seem to be a specific host causing the problem.  Reducing overall hosts in
any order makes the problem go away.

I've looked at the the processor utilization during the good times and slow
times, and don't see a significant difference.  I have 16GB of memory, and
it consistently shows about 35% utilization.  I also checked mySQL response
time, and the per-query SQL response does not vary.  So it doesn't appear to
be a db problem.

I suspect there is some TC configuration parameter such as heap, etc that I
need to tweak. (But I'm not getting OutOfMemory errors).  But I don't know
which one, and I don't know a formula to use to figure out what to set it
to.  So I just need a little education.  What tools can I use to help me
figure out what is going south slowing everything to a crawl when the extra
hosts come online?  And what parameters should I be looking at (and how
should I calculate the proper values based on number of hosts)?

Suggestions?


Interesting problem. Couple of questions:
Is the slowness sporadic or persistent? What happens when you again
shutdown some of the hosts? Does response time comes back to normal?
Is all transactions are slow or some of them which are routed to
specific host(s)?
Are you fronting Tomcat instances with Apache? If yes, then please
post BalancerMember configuration of the Tomcat cluster here.



Taking the OP's observations above at face value, I think that first thing that I would 
do, is to configure the Java JVM that runs this tomcat, to do some logging of the Garbage 
Collection events. And then I would look at that logfile, and find out if there is some 
significant difference (in frequency of GC, or in the mounts of memory mentioned for each 
event), depending on the number of Hosts enabled.


You would have to look at the appropriate command-line switches to use for the particular 
JVM being used, for how to set this up.
And then (since this seems to be on a Windows platform, running tomcat as a service), you 
should use the "tomcat(version)w.exe" program (a GUI), to set these parameters properly.

See e.g.
https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html#Tomcat_service_application

--> "Tomcat monitor application"





-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance settings for Multiple Hosts

[Suvendu Sekhar Mondal]

On Fri, Oct 6, 2017 at 2:06 AM, Jerry Malcolm  wrote:
> I am running TC 8.0 on WinServer8 on a commercially hosted platform with a
> WAMP environment.   I am running around 10 virtual hosts.   2 hosts are
> dedicated to JSPWiki.  The other 8 are running variations of the same custom
> application with around 10-15 individual webapps each.
>
> When I am running 7 of the 10 hosts, performance is great.  I get JSP
> response time under a second.  But when I add just a couple of more of the
> hosts, my page response time on all of the apps goes from an acceptable
> under a second to horrible at around 15-20 seconds per page.  It doesn't
> seem to be a specific host causing the problem.  Reducing overall hosts in
> any order makes the problem go away.
>
> I've looked at the the processor utilization during the good times and slow
> times, and don't see a significant difference.  I have 16GB of memory, and
> it consistently shows about 35% utilization.  I also checked mySQL response
> time, and the per-query SQL response does not vary.  So it doesn't appear to
> be a db problem.
>
> I suspect there is some TC configuration parameter such as heap, etc that I
> need to tweak. (But I'm not getting OutOfMemory errors).  But I don't know
> which one, and I don't know a formula to use to figure out what to set it
> to.  So I just need a little education.  What tools can I use to help me
> figure out what is going south slowing everything to a crawl when the extra
> hosts come online?  And what parameters should I be looking at (and how
> should I calculate the proper values based on number of hosts)?
>
> Suggestions?
>
Interesting problem. Couple of questions:
Is the slowness sporadic or persistent? What happens when you again
shutdown some of the hosts? Does response time comes back to normal?
Is all transactions are slow or some of them which are routed to
specific host(s)?
Are you fronting Tomcat instances with Apache? If yes, then please
post BalancerMember configuration of the Tomcat cluster here.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance settings for Multiple Hosts

[Mark Thomas]

On 05/10/17 21:36, Jerry Malcolm wrote:
> I am running TC 8.0 on WinServer8 on a commercially hosted platform with
> a WAMP environment.   I am running around 10 virtual hosts.   2 hosts
> are dedicated to JSPWiki.  The other 8 are running variations of the
> same custom application with around 10-15 individual webapps each.
> 
> When I am running 7 of the 10 hosts, performance is great.  I get JSP
> response time under a second.  But when I add just a couple of more of
> the hosts, my page response time on all of the apps goes from an
> acceptable under a second to horrible at around 15-20 seconds per page. 
> It doesn't seem to be a specific host causing the problem.  Reducing
> overall hosts in any order makes the problem go away.
> 
> I've looked at the the processor utilization during the good times and
> slow times, and don't see a significant difference.  I have 16GB of
> memory, and it consistently shows about 35% utilization.

Is this the OS memory or what you have allocated to Java?

> I also checked
> mySQL response time, and the per-query SQL response does not vary.  So
> it doesn't appear to be a db problem.
> 
> I suspect there is some TC configuration parameter such as heap, etc
> that I need to tweak. (But I'm not getting OutOfMemory errors).  But I
> don't know which one, and I don't know a formula to use to figure out
> what to set it to.  So I just need a little education.  What tools can I
> use to help me figure out what is going south slowing everything to a
> crawl when the extra hosts come online?  And what parameters should I be
> looking at (and how should I calculate the proper values based on number
> of hosts)?
> 
> Suggestions?>
> Thanks.
> 
> Jerry
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Performance settings for Multiple Hosts

[Jerry Malcolm]

I am running TC 8.0 on WinServer8 on a commercially hosted platform with 
a WAMP environment.   I am running around 10 virtual hosts.   2 hosts 
are dedicated to JSPWiki.  The other 8 are running variations of the 
same custom application with around 10-15 individual webapps each.


When I am running 7 of the 10 hosts, performance is great.  I get JSP 
response time under a second.  But when I add just a couple of more of 
the hosts, my page response time on all of the apps goes from an 
acceptable under a second to horrible at around 15-20 seconds per page.  
It doesn't seem to be a specific host causing the problem.  Reducing 
overall hosts in any order makes the problem go away.


I've looked at the the processor utilization during the good times and 
slow times, and don't see a significant difference.  I have 16GB of 
memory, and it consistently shows about 35% utilization.  I also checked 
mySQL response time, and the per-query SQL response does not vary.  So 
it doesn't appear to be a db problem.


I suspect there is some TC configuration parameter such as heap, etc 
that I need to tweak. (But I'm not getting OutOfMemory errors).  But I 
don't know which one, and I don't know a formula to use to figure out 
what to set it to.  So I just need a little education.  What tools can I 
use to help me figure out what is going south slowing everything to a 
crawl when the extra hosts come online?  And what parameters should I be 
looking at (and how should I calculate the proper values based on number 
of hosts)?


Suggestions?

Thanks.

Jerry


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org