Timur Irmatov created TS-1511:
---------------------------------
Summary: TS crashes when second node joins cluster
Key: TS-1511
URL: https://issues.apache.org/jira/browse/TS-1511
Project: Traffic Server
Issue Type: Bug
Components: Clustering
Affects Versions: 3.2.0
Environment: Ubuntu 12.04.1, ATS 3.2.0
Reporter: Timur Irmatov
I have ATS running in transparent forward proxy configuration on one server. It
was running fine for several weeks already. ATS version is 3.2.0, compiled from
source. I have other server with same OS (Ubuntu 12.04.1) on the same
hardware. Now I am trying to enable clustering on both machines. First, I have
installed ATS on second machine, compiled with same configuration options.
Second, I have copied all configs from first to second machine. Then I have
started second TS server.
Then, I enable clustering on both machines:
traffic_line -s proxy.local.cluster.type -v 1
traffic_line -s proxy.config.cluster.ethernet_interface -v eth1.24
traffic_line -L
Then, looking at the traffic.out, after a minute second machine joins cluster,
first TS server crashes. It restarts, joins cluster and crashes again. I had to
disable clustering to enable first TS server to work. Traffic.out from first
server:
[Oct 3 23:51:06.718] Manager {0x7f87fb6ef700} NOTE: adding node 81.95.224.42
to the cluster
[Oct 3 23:51:07.481] Manager {0x7f87fb6ef700} NOTE: [drainIncomingChannel]
file req: records.config v: 40 bytes: 28974
[Oct 3 23:51:14.551] Manager {0x7f87fb6ef700} NOTE: [drainIncomingChannel]
file req: records.config v: 41 bytes: 28974
[Oct 3 23:51:15.859] Manager {0x7f87fb6ef700} NOTE:
[ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 41 n: 42
[Oct 3 23:51:18.813] Server {0x2b1c59b80700} NOTE: machine up 192.168.10.42:0,
protocol version=3.2
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/local/bin/traffic_server - STACK TRACE:
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x2b1c3d2b5cb0]
/usr/local/bin/traffic_server(_ZN10HttpTunnel16consumer_handlerEiP18HttpTunnelConsumer+0x18d)[0x56dd1d]
/usr/local/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x117)[0x56db67]
/usr/local/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0x801)[0x686a71]
/usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x293)[0x67dc63]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x90)[0x6abbd0]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x52c)[0x6ac6cc]
/usr/local/bin/traffic_server[0x6aa9b2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x2b1c3d2ade9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x2b1c3f1a5dbd]
[Oct 3 23:51:34.288] Manager {0x7f87ff742740} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11:
Segmentation fault
[Oct 3 23:51:34.288] Manager {0x7f87ff742740} ERROR: (last system error 2: No
such file or directory)
[Oct 3 23:51:34.288] Manager {0x7f87ff742740} ERROR: [Alarms::signalAlarm]
Server Process was reset
[Oct 3 23:51:34.288] Manager {0x7f87ff742740} ERROR: (last system error 2: No
such file or directory)
[Oct 3 23:51:35.294] Manager {0x7f87ff742740} NOTE: [LocalManager::startProxy]
Launching ts process
[TrafficServer] using root directory '/usr/local'
[Oct 3 23:51:35.301] Manager {0x7f87ff742740} NOTE:
[LocalManager::pollMgmtProcessServer] New process connecting fd '16'
[Oct 3 23:51:35.301] Manager {0x7f87ff742740} NOTE: [Alarms::signalAlarm]
Server Process born
[Oct 3 23:51:36.317] {0x2addcf865a40} STATUS: opened
/usr/local/var/log/trafficserver/diags.log
[Oct 3 23:51:36.317] {0x2addcf865a40} NOTE: updated diags config
[Oct 3 23:51:36.322] Server {0x2addcf865a40} NOTE: cache clustering enabled
[Oct 3 23:51:36.330] Server {0x2addcf865a40} NOTE: cache clustering enabled
[Oct 3 23:51:36.342] Server {0x2addcf865a40} NOTE: logging initialized[15],
logging_mode = 3
[Oct 3 23:51:36.377] Server {0x2addcf865a40} NOTE: traffic server running
[Oct 3 23:51:52.338] Server {0x2addd4f23700} NOTE: cache enabled
[Oct 3 23:51:53.343] Server {0x2ade3383c700} NOTE: machine up 81.95.224.42:0,
protocol version=3.2
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/local/bin/traffic_server - STACK TRACE:
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x2addcd4a2cb0]
/usr/local/bin/traffic_server(_ZN10HttpTunnel16consumer_handlerEiP18HttpTunnelConsumer+0x18d)[0x56dd1d]
/usr/local/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x117)[0x56db67]
/usr/local/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0x801)[0x686a71]
/usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x293)[0x67dc63]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x90)[0x6abbd0]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x52c)[0x6ac6cc]
/usr/local/bin/traffic_server[0x6aa9b2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x2addcd49ae9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x2addcf392dbd]
[Oct 3 23:52:02.686] Manager {0x7f87ff742740} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11:
Segmentation fault
[Oct 3 23:52:02.686] Manager {0x7f87ff742740} ERROR: (last system error 2: No
such file or directory)
[Oct 3 23:52:02.686] Manager {0x7f87ff742740} ERROR: [Alarms::signalAlarm]
Server Process was reset
[Oct 3 23:52:02.686] Manager {0x7f87ff742740} ERROR: (last system error 2: No
such file or directory)
[Oct 3 23:52:03.689] Manager {0x7f87ff742740} NOTE: [LocalManager::startProxy]
Launching ts process
[TrafficServer] using root directory '/usr/local'
[Oct 3 23:52:03.695] Manager {0x7f87ff742740} NOTE:
[LocalManager::pollMgmtProcessServer] New process connecting fd '16'
[Oct 3 23:52:03.695] Manager {0x7f87ff742740} NOTE: [Alarms::signalAlarm]
Server Process born
[Oct 3 23:52:04.710] {0x2ba4566a3a40} STATUS: opened
/usr/local/var/log/trafficserver/diags.log
[Oct 3 23:52:04.710] {0x2ba4566a3a40} NOTE: updated diags config
[Oct 3 23:52:04.716] Server {0x2ba4566a3a40} NOTE: cache clustering enabled
[Oct 3 23:52:04.727] Server {0x2ba4566a3a40} NOTE: cache clustering enabled
[Oct 3 23:52:04.739] Server {0x2ba4566a3a40} NOTE: logging initialized[15],
logging_mode = 3
[Oct 3 23:52:04.754] Server {0x2ba4566a3a40} NOTE: traffic server running
[Oct 3 23:52:18.867] Server {0x2ba457cf5700} NOTE: cache enabled
[Oct 3 23:52:19.842] Server {0x2ba4b8cd5700} NOTE: machine up 192.168.10.42:0,
protocol version=3.2
NOTE: Traffic Server received Sig 11: Segmentation fault
/usr/local/bin/traffic_server - STACK TRACE:
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x2ba4542e0cb0]
/usr/local/bin/traffic_server(_ZN10HttpTunnel16consumer_handlerEiP18HttpTunnelConsumer+0x18d)[0x56dd1d]
/usr/local/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x117)[0x56db67]
/usr/local/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0x801)[0x686a71]
/usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x293)[0x67dc63]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x90)[0x6abbd0]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x52c)[0x6ac6cc]
/usr/local/bin/traffic_server[0x6aa9b2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x2ba4542d8e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x2ba4561d0dbd]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
