Zhao Yongming created TS-1346:
---------------------------------
Summary: syn cookie may crash config update in cluster
Key: TS-1346
URL: https://issues.apache.org/jira/browse/TS-1346
Project: Traffic Server
Issue Type: Bug
Components: Clustering, Management
Affects Versions: 3.3.1, 3.0.5
Reporter: Zhao Yongming
Fix For: 3.3.1
in clustering, we have about 48 boxes, where the config sync and others may
trigger the 8088 syn cookie or so. which may case the manager killed:
{code}
[179378.192955] possible SYN flooding on port 8088. Sending cookies.
[179472.331827] possible SYN flooding on port 8088. Sending cookies.
[235126.729593] possible SYN flooding on port 8088. Sending cookies.
[235335.369386] possible SYN flooding on port 8088. Sending cookies.
[235483.128214] possible SYN flooding on port 8088. Sending cookies.
[235639.853147] possible SYN flooding on port 8088. Sending cookies.
[235785.611114] possible SYN flooding on port 8088. Sending cookies.
[235852.501364] possible SYN flooding on port 8088. Sending cookies.
[238275.390711] possible SYN flooding on port 8088. Sending cookies.
{code}
{code}
[Jul 12 11:37:15.254] Manager {0x7fe8b33d5700} NOTE: [drainIncomingChannel]
file req: records.config v: 8963 bytes: 29295
[Jul 12 11:37:15.301] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8959
[Jul 12 11:37:15.301] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:37:15.314] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8960
[Jul 12 11:37:15.315] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:37:27.633] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8963 n: 8964
[Jul 12 11:37:27.892] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8964 n: 8965
[Jul 12 11:37:32.374] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8965 n: 8966
[Jul 12 11:37:39.608] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8965
[Jul 12 11:37:39.608] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:37:49.975] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8966 n: 8967
[Jul 12 11:37:50.067] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8967 n: 8968
[Jul 12 11:38:10.051] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8968 n: 8969
[Jul 12 11:38:39.222] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8969 n: 8970
[Jul 12 11:38:40.194] Manager {0x7fe8b33d5700} NOTE: marking server on node
222.186.49.169 as up
[Jul 12 11:38:49.250] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:49.250] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:49.250] Manager {0x7fe8b33d5700} NOTE: marking server on node
222.186.49.165 as up
[Jul 12 11:38:49.252] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:49.252] Manager {0x7fe8b4a6f7e0} NOTE: marking server on node
222.186.49.185 as down
[Jul 12 11:38:49.252] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:49.252] Manager {0x7fe8b33d5700} NOTE: marking server on node
222.186.49.185 as up
[Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.491] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.491] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.492] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.492] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.493] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.493] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.495] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.495] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.496] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.496] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.497] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.497] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.497] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.498] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.498] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.498] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.499] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.499] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR: (last system error 4:
Interrupted system call)
[Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::sendReliableMessageReadTillClose] Unable to connect
[Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:38:54.501] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.501] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:38:54.502] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.502] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:38:54.503] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.503] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:38:54.503] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.504] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:38:54.504] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.504] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:38:54.505] Manager {0x7fe8b33d5700} ERROR:
[ClusterCom::handleMultiCastFilePacket] Update failed
[Jul 12 11:38:54.505] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:38:54.624] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8970 n: 8973
[Jul 12 11:39:02.360] Manager {0x7fe8b4a6f7e0} NOTE: marking manager on node
222.186.49.196 as down
[Jul 12 11:39:02.360] Manager {0x7fe8b4a6f7e0} NOTE: marking server on node
222.186.49.196 as down
[Jul 12 11:39:02.388] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8973 n: 8974
[Jul 12 11:39:02.389] Manager {0x7fe8b33d5700} NOTE: marking manager on node
222.186.49.196 as up
[Jul 12 11:39:03.279] Manager {0x7fe8b33d5700} NOTE: marking server on node
222.186.49.196 as up
[Jul 12 11:39:05.471] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8974 n: 8975
[Jul 12 11:39:18.916] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8975 n: 8976
[Jul 12 11:39:18.918] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8975
[Jul 12 11:39:18.918] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:18.999] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8976 n: 8977
[Jul 12 11:39:22.973] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8977 n: 8978
[Jul 12 11:39:28.455] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8978 n: 8979
[Jul 12 11:39:29.013] Manager {0x7fe8b33d5700} NOTE: [drainIncomingChannel]
file req: records.config v: 8979 bytes: 29295
[Jul 12 11:39:32.995] Manager {0x7fe8b33d5700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8979 n: 8980
[Jul 12 11:39:32.997] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:37.400] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:37.400] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:37.403] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:37.403] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:37.403] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:37.404] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[Jul 12 11:39:37.404] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel]
Error file req: records.config ver: 8979
[Jul 12 11:39:37.404] Manager {0x7fe8b33d5700} ERROR: (last system error 111:
Connection refused)
[TrafficManager] ==> Cleaning up and reissuing signal #11
[Jul 12 11:39:48.329] Manager {0x7fe8b33d5700} ERROR: [TrafficManager] ==>
Cleaning up and reissuing signal #11
[Jul 12 11:39:48.329] Manager {0x7fe8b33d5700} ERROR: (last system error 104:
Connection reset by peer)
[E. Mgmt] log ==> [TrafficManager] using root directory '/usr'
[Jul 12 11:39:49.346] {0x7fc772ec37e0} STATUS: opened
/var/log/trafficserver/manager.log
[Jul 12 11:39:49.346] {0x7fc772ec37e0} NOTE: updated diags config
[Jul 12 11:39:49.353] Manager {0x7fc772ec37e0} NOTE: [ClusterCom::ClusterCom]
Node running on OS: 'Linux' Release: '2.6.32-220.17.1.tb619.el6.x86_6'
[Jul 12 11:39:49.356] Manager {0x7fc772ec37e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 8080
[Jul 12 11:39:49.357] Manager {0x7fc772ec37e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 81
[Jul 12 11:39:49.357] Manager {0x7fc772ec37e0} NOTE: [TrafficManager] Setup
complete
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
