Dear All,
I have a strange problem. When I added a new server to my tomcat cluster I have
noticed that the load is getting very high on the server. Tomcat log show a lot
of these lines
18.12.2009 09:07:14 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-120}:4000,{62, 75, -127, -120},4000, alive=65087504,id={-64 -42 103 97 8 -7 69
-88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={}, domain={}, ]
18.12.2009 09:07:14
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-122}:4000,{62, 75, -127, -122},4000, alive=64996684,id={-15 62 -53 -50 -43 81
75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:07:19
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
WARNUNG: Member added, even though we werent
notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 18 -76
79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, domain={}, ]
18.12.2009 09:07:19 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 18 -76
79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, domain={}, ]
18.12.2009 09:08:10 org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75,
-127, -121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83
80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={},
]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
INFO: Suspect member, confirmed
dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83 80 64
76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115
-83 80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115 -83 80 64
76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -122}:4000,{62, 75, -127, -122},4000, alive=65054434,id={-15 62 -53
-50 -43 81 75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -118}:4000,{62, 75, -127, -118},4000, alive=58290426,id={101 61 65 84
-59 -114 65 -57 -106 8 -118 -25 -55 56 -82 111 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -120}:4000,{62, 75, -127, -120},4000, alive=65141440,id={-64 -42 103
97 8 -7 69 -88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={},
domain={}, ]] message. Will verify.
Also the load on the server is getting very high (while no CPU usage). This is
the output of top
top - 08:29:51 up 63 days, 22:53, 1 user, load average: 12.95, 10.61, 8.35
Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.6%sy, 0.0%ni, 98.4%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 33023640k total, 19154120k used, 13869520k free, 373012k buffers
Swap: 999992k total, 676k used, 999316k free, 13818440k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 10312 748 620 S 0 0.0 1:54.74 init
2 root 15 -5 0 0 0 S 0 0.0 3:27.03 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:01.66 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:27.74 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 3365:16 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:08.92 migration/1
This high load start after starting apache2 (with modjk) so users can enter the
server using the hardware balancer. And the load is increasing always to strange
levels like 99 only stopping tomcat fix it.
Please help
