Re: Errors in session replication and very high server load

2009-12-21 Thread Mohamedin 

Thanks for the reply.

I have tried to figure out why the load is high but I couldn't. Any hints?

Thanks,
Mohamed Mohamedin
- Original Message - 
From: Filip Hanik - Dev Lists devli...@hanik.com

To: Tomcat Users List users@tomcat.apache.org
Sent: Sunday, December 20, 2009 3:56 PM
Subject: Re: Errors in session replication and very high server load



Well, the log messages you see, are all based on timeouts.
If your system has a load average of 12, unless you have a 12-way machine, 
that is very high, and could be the cause of your timeouts.

You will need to figure out what is causing the high load average.

Filip

On 12/18/2009 01:30 AM, mohame...@easy-dialog.info wrote:

Dear All,

I have a strange problem. When I added a new server to my tomcat cluster 
I have
noticed that the load is getting very high on the server. Tomcat log show 
a lot

of these lines

18.12.2009 09:07:14 org.apache.catalina.ha.tcp.SimpleTcpCluster 
memberAdded

INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 
75, -127,
-120}:4000,{62, 75, -127, -120},4000, alive=65087504,id={-64 -42 103 97 
8 -7 69
-88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={}, 
domain={}, ]

18.12.2009 09:07:14
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 
75, -127,
-122}:4000,{62, 75, -127, -122},4000, alive=64996684,id={-15 
62 -53 -50 -43 81
75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={}, 
domain={}, ]]

18.12.2009 09:07:19
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
WARNUNG: Member added, even though we werent
notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 
75, -127,
-117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 
18 -76
79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, 
domain={}, ]
18.12.2009 09:07:19 org.apache.catalina.ha.tcp.SimpleTcpCluster 
memberAdded

INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 
75, -127,
-117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 
18 -76
79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, 
domain={}, ]

18.12.2009 09:08:10 org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 
75,
-127, -121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 
115 -83
80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, 
domain={},

]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
INFO: Suspect member, confirmed
dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 
75, -127,
-121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83 
80 64
76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, 
domain={}, ]]

18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -121}:4000,{62, 75, -127, -121},4000, 
alive=65045054,id={-87 -91 115

-83 80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 
75, -127,
-121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115 -83 
80 64
76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, 
domain={}, ]]

18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -122}:4000,{62, 75, -127, -122},4000, alive=65054434,id={-15 
62 -53

-50 -43 81 75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -118}:4000,{62, 75, -127, -118},4000, alive=58290426,id={101 61 
65 84

-59 -114 65 -57 -106 8 -118 -25 -55 56 -82 111 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -120}:4000,{62, 75, -127, -120},4000, 
alive=65141440,id={-64 -42 103
97 8 -7 69 -88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, 
command={},

domain={}, ]] message. Will verify.


Also the load

Re: Errors in session replication and very high server load

2009-12-20 Thread Filip Hanik - Dev Lists 

Well, the log messages you see, are all based on timeouts.
If your system has a load average of 12, unless you have a 12-way 
machine, that is very high, and could be the cause of your timeouts.

You will need to figure out what is causing the high load average.

Filip

On 12/18/2009 01:30 AM, mohame...@easy-dialog.info wrote:

Dear All,

I have a strange problem. When I added a new server to my tomcat cluster I have
noticed that the load is getting very high on the server. Tomcat log show a lot
of these lines

18.12.2009 09:07:14 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-120}:4000,{62, 75, -127, -120},4000, alive=65087504,id={-64 -42 103 97 8 -7 69
-88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={}, domain={}, ]
18.12.2009 09:07:14
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-122}:4000,{62, 75, -127, -122},4000, alive=64996684,id={-15 62 -53 -50 -43 81
75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:07:19
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
WARNUNG: Member added, even though we werent
notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 18 -76
79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, domain={}, ]
18.12.2009 09:07:19 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 18 -76
79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, domain={}, ]
18.12.2009 09:08:10 org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75,
-127, -121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83
80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={},
]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
INFO: Suspect member, confirmed
dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83 80 64
76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115
-83 80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115 -83 80 64
76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -122}:4000,{62, 75, -127, -122},4000, alive=65054434,id={-15 62 -53
-50 -43 81 75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -118}:4000,{62, 75, -127, -118},4000, alive=58290426,id={101 61 65 84
-59 -114 65 -57 -106 8 -118 -25 -55 56 -82 111 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -120}:4000,{62, 75, -127, -120},4000, alive=65141440,id={-64 -42 103
97 8 -7 69 -88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={},
domain={}, ]] message. Will verify.


Also the load on the server is getting very high (while no CPU usage). This is
the output of top

top - 08:29:51 up 63 days, 22:53, 1 user, load average: 12.95, 10.61, 8.35
Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.6%sy, 0.0%ni, 98.4%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 33023640k total, 19154120k used, 13869520k free, 373012k buffers
Swap: 92k total, 676k used, 999316k free,

Errors in session replication and very high server load

2009-12-18 Thread mohame...@easy-dialog.info 
Dear All,

I have a strange problem. When I added a new server to my tomcat cluster I have
noticed that the load is getting very high on the server. Tomcat log show a lot
of these lines

18.12.2009 09:07:14 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-120}:4000,{62, 75, -127, -120},4000, alive=65087504,id={-64 -42 103 97 8 -7 69
-88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={}, domain={}, ]
18.12.2009 09:07:14
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-122}:4000,{62, 75, -127, -122},4000, alive=64996684,id={-15 62 -53 -50 -43 81
75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:07:19
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
WARNUNG: Member added, even though we werent
notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 18 -76
79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, domain={}, ]
18.12.2009 09:07:19 org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-117}:4000,{62, 75, -127, -117},4000, alive=58229968,id={16 -115 -21 -109 18 -76
79 58 -95 -17 57 -32 -69 -111 -20 28 }, payload={}, command={}, domain={}, ]
18.12.2009 09:08:10 org.apache.catalina.ha.tcp.SimpleTcpCluster
memberDisappeared
INFO: Received member
disappeared:org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75,
-127, -121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83
80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={},
]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
performBasicCheck
INFO: Suspect member, confirmed
dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-121}:4000,{62, 75, -127, -121},4000, alive=64986581,id={-87 -91 115 -83 80 64
76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115
-83 80 64 76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Verification complete. Member still
alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62, 75, -127,
-121}:4000,{62, 75, -127, -121},4000, alive=65045054,id={-87 -91 115 -83 80 64
76 -9 -68 -107 -109 52 0 -47 109 98 }, payload={}, command={}, domain={}, ]]
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -122}:4000,{62, 75, -127, -122},4000, alive=65054434,id={-15 62 -53
-50 -43 81 75 18 -112 -43 58 -102 69 72 83 21 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -118}:4000,{62, 75, -127, -118},4000, alive=58290426,id={101 61 65 84
-59 -114 65 -57 -106 8 -118 -25 -55 56 -82 111 }, payload={}, command={},
domain={}, ]] message. Will verify.
18.12.2009 09:08:10
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
memberDisappeared
INFO: Received
memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{62,
75, -127, -120}:4000,{62, 75, -127, -120},4000, alive=65141440,id={-64 -42 103
97 8 -7 69 -88 -113 -106 -32 -64 46 76 -117 -58 }, payload={}, command={},
domain={}, ]] message. Will verify.


Also the load on the server is getting very high (while no CPU usage). This is
the output of top

top - 08:29:51 up 63 days, 22:53, 1 user, load average: 12.95, 10.61, 8.35
Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.6%sy, 0.0%ni, 98.4%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 33023640k total, 19154120k used, 13869520k free, 373012k buffers
Swap: 92k total, 676k used, 999316k free, 13818440k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 10312 748 620 S 0 0.0 1:54.74 init
2 root 15 -5 0 0 0 S 0 0.0 3:27.03 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:01.66 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:27.74 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 3365:16 watchdog/0
6 root RT -5 0 0 0 S 0 0.0