[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316488#comment-14316488
]
James Peach commented on TS-3386:
-
This sounds like something bad is happenning to{{ traffic_manager}}. Is there
anything in {{manager.log}} or {{dags.log}} about that? Is it deadlocked or
crashing?
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 -
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316517#comment-14316517
]
Luca Bruno edited comment on TS-3386 at 2/11/15 4:53 PM:
-
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option. Either ways, it would keep restarting the server, so
fixing the cache issue wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
be killed just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion OR not kill at all yet keep alerting
in the log files for external monitoring.
was (Author: lethalman):
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option. Either ways, it would keep restarting the server, so
fixing the cache issue wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
be killed just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion OR not kill at all, but keep
alerting in the log files for external monitoring.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http
[jira] [Commented] (TS-3164) why the load of trafficserver occurrs a abrupt rise on a occasion ?
[
https://issues.apache.org/jira/browse/TS-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316244#comment-14316244
]
Zhao Yongming commented on TS-3164:
---
I have seen that, but I got some very short lockup like situation, mostly less
than 15s. still don't know why
why the load of trafficserver occurrs a abrupt rise on a occasion ?
---
Key: TS-3164
URL: https://issues.apache.org/jira/browse/TS-3164
Project: Traffic Server
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64bit, 8 cores, 128G mem
Reporter: taoyunxing
Fix For: sometime
I use Tsar to monitor the traffic status of the ATS 4.2.0, and come across
the following problem:
{code}
Time ---cpu-- ---mem-- ---tcp-- -traffic --sda--- --sdb---
--sdc--- ---load-
Time util util retranbytin bytout util util
util load1
03/11/14-18:20 40.6787.19 3.3624.5M 43.9M13.0294.68
0.00 5.34
03/11/14-18:25 40.3087.20 3.2722.5M 42.6M12.3894.87
0.00 5.79
03/11/14-18:30 40.8484.67 3.4421.4M 42.0M13.2995.37
0.00 6.28
03/11/14-18:35 43.6387.36 3.2123.8M 45.0M13.2393.99
0.00 7.37
03/11/14-18:40 42.2587.37 3.0924.2M 44.8M12.8495.77
0.00 7.25
03/11/14-18:45 42.9687.44 3.4623.3M 46.0M12.9695.84
0.00 7.10
03/11/14-18:50 44.0087.42 3.4922.3M 43.0M14.1794.99
0.00 6.57
03/11/14-18:55 42.2087.44 3.4622.3M 43.6M13.1996.05
0.00 6.09
03/11/14-19:00 44.9087.53 3.6023.6M 46.5M13.6196.67
0.00 8.06
03/11/14-19:05 46.2687.73 3.2425.8M 49.1M15.3994.05
0.00 9.98
03/11/14-19:10 43.8587.69 3.1925.4M 50.9M12.8897.80
0.00 7.99
03/11/14-19:15 45.2887.69 3.3625.6M 49.6M13.1096.86
0.00 7.47
03/11/14-19:20 44.1185.20 3.2924.1M 47.8M14.2496.75
0.00 5.82
03/11/14-19:25 45.2687.78 3.5224.4M 47.7M13.2195.44
0.00 7.61
03/11/14-19:30 44.8387.80 3.6425.7M 50.8M13.2798.02
0.00 6.85
03/11/14-19:35 44.8987.78 3.6123.9M 49.0M13.3497.42
0.00 7.04
03/11/14-19:40 69.2188.88 0.5518.3M 33.7M11.3971.23
0.00 65.80
03/11/14-19:45 72.4788.66 0.2715.4M 31.6M11.5172.31
0.00 11.56
03/11/14-19:50 44.8788.72 4.1122.7M 46.3M12.9997.33
0.00 8.29
{code}
in addition, top command show
{code}
hi:0
ni:0
si:45.56
st:0
sy:13.92
us:12.58
wa:14.3
id:15.96
{code}
who help me ? thanks in advance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316517#comment-14316517
]
Luca Bruno edited comment on TS-3386 at 2/11/15 4:53 PM:
-
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option. Either ways, it would keep restarting the server, so
fixing the cache issue wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
be killed just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion.
was (Author: lethalman):
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option. Either ways, it would keep restarting the server, so
fixing the cache issue wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
restarted just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]:
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316517#comment-14316517
]
Luca Bruno edited comment on TS-3386 at 2/11/15 4:52 PM:
-
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option. Either ways, it would keep restarting the server, so
fixing the cache issue wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
restarted just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion.
was (Author: lethalman):
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS, the cache starts from zero. It's a 500gb raw disk
cache. It's not supposed to work like this? Perhaps I'm missing some option.
Either ways, it would keep restarting the server, so fixing the cache issue
wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
restarted just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316517#comment-14316517
]
Luca Bruno edited comment on TS-3386 at 2/11/15 4:53 PM:
-
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option. Either ways, it would keep restarting the server, so
fixing the cache issue wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
be killed just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion OR not kill at all, but keep
alerting in the log files.
was (Author: lethalman):
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option. Either ways, it would keep restarting the server, so
fixing the cache issue wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
be killed just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316517#comment-14316517
]
Luca Bruno edited comment on TS-3386 at 2/11/15 5:26 PM:
-
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
But I haven't found anything in the logs on why it returns 502, perhaps the
connections to the origin server are full and it returns bad gateway for that
reason also to the clients?
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option (EDIT: it re-reads the cache, see comment below).
Either ways, it would keep restarting the server, so fixing the cache issue
wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
be killed just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion OR not kill at all yet keep alerting
in the log files for external monitoring.
was (Author: lethalman):
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
But I haven't found anything in the logs on why it returns 502, perhaps the
connections to the origin server are full and it returns bad gateway for that
reason also to the clients?
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option. Either ways, it would keep restarting the server, so
fixing the cache issue wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
be killed just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion OR not kill at all yet keep alerting
in the log files for external monitoring.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316482#comment-14316482
]
Zhao Yongming commented on TS-3386:
---
well, things get more interesting.
q1: why you will lose the cached content in a restart of the traffic server??
q1.1: is that a cache issue?
q2: you are going to protect the origin server, why you think that limit on the
UA side connection is a better solution to the limit on the origin side?
q2.1 have you seen any occurrence of connection(httpSM) hanghup?
q2.2 what is a better way to handle of the connection issue, for example
timeout?
when you try to handle tons of cache, tons of the traffic, keep it simple, keep
it robust always better than anything intelligent.
yes, we have fixed many cache issue we meet, http SM issues, and connections
timeout issue, connection leaking ... I think most of the important change
already in the official tree. and this is the way we figure out the root issues
in ATS, which may lead to just some very tiny fix that will only affect very
high traffic site with very strict SLA requirement.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316517#comment-14316517
]
Luca Bruno commented on TS-3386:
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS, the cache starts from zero. It's a 500gb raw disk
cache. It's not supposed to work like this? Perhaps I'm missing some option.
Either ways, it would keep restarting the server, so fixing the cache issue
wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
restarted just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316517#comment-14316517
]
Luca Bruno edited comment on TS-3386 at 2/11/15 4:55 PM:
-
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
But I haven't found anything in the logs on why it returns 502, perhaps the
connections to the origin server are full and it returns bad gateway for that
reason also to the clients?
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option. Either ways, it would keep restarting the server, so
fixing the cache issue wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
be killed just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion OR not kill at all yet keep alerting
in the log files for external monitoring.
was (Author: lethalman):
James:
I've pasted the manager.log above, dags had nothing related. And syslog clearly
states that the server returned 502 to cop and decided to kill the server after
2 times that it failed:
{noformat}
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
{noformat}
To reproduce, set connection throttle to 100 and perform 1000 concurrent
requests to ATS with minimal hit ratio (so that it also creates connections to
the origin server).
Zhao:
q1: Whenever I restart ATS with `trafficserver restart`, the cache starts from
zero. It's a 500gb raw disk cache. It's not supposed to work like this? Perhaps
I'm missing some option. Either ways, it would keep restarting the server, so
fixing the cache issue wouldn't solve this problem.
q2: I'm just testing stuff. Of course I won't be limiting the number of
connections like that, but I still find this a bug.
q2.1: not sure sorry, what is httpSM?
q2.2: I think there are 2 options: either don't accept(), or accept and
immediately close the connection with an error (which is apparently what the
server does, except that it does also for the heartbeat). For example it's not
that nginx kills its worker children because it reached the maximum number of
connections.
Sure I keep it simple, but robustness is also the fact that the server must not
be killed just because it's busy. Being busy means it's working and must not be
killed by the heartbeat. In which case it can be solved by using a dedicated
connection for the heartbeat in my opinion OR not kill at all yet keep alerting
in the log files for external monitoring.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316560#comment-14316560
]
Luca Bruno commented on TS-3386:
Sorry, when I restart trafficserver it indeeds re-reads the cache (after adding
the udev rule to the raw disk). But of course when it gets restarted by cop, it
restarts often and thus I see the cache empty.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server
Version: Apache
[jira] [Resolved] (TS-3385) should show scripting loading error when using lua plugin as remap plugin
[ https://issues.apache.org/jira/browse/TS-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kit Chan resolved TS-3385. -- Resolution: Fixed should show scripting loading error when using lua plugin as remap plugin - Key: TS-3385 URL: https://issues.apache.org/jira/browse/TS-3385 Project: Traffic Server Issue Type: Improvement Components: Lua, Plugins Reporter: Kit Chan Assignee: Kit Chan Priority: Minor Fix For: 5.3.0 e.g. if we make a syntax mistake in our lua script the lua plugin will not output the loading error when it is being used as a remap plugin. It works fine and output the syntax mistake though when used as a global plugin in plugin.conf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316683#comment-14316683
]
Leif Hedstrom commented on TS-3386:
---
I think TS-2490 is close / similar in effect, where we can kill traffic_server
because manager can't get a connection to it. Not quite the same, but similar
issue. I can not find the Jira that talked about running out of connections,
maybe it was fixed? I thought there was a commit sometime ago, about excluding
the back door requests from connection throttling?
To the OP, which version of ATS are you running ?
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316671#comment-14316671
]
Leif Hedstrom commented on TS-3386:
---
I'm pretty sure there was a Jira on exactly that suggestion [~jpe...@apache.org]
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:44 test-cache
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316655#comment-14316655
]
James Peach commented on TS-3386:
-
Ok, that's nasty ... we should exempt heartbeats from connection throttling
IMHO.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:44 test-cache
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316783#comment-14316783
]
Luca Bruno edited comment on TS-3386 at 2/11/15 7:07 PM:
-
I'm using latest 5.2.0. Just to say, I've read some of the code of the cop and
it's quite clear: if it fails 2 times, it's killed. Didn't put attention on
whether it's using a separate channel though.
was (Author: lethalman):
I'm using latest 5.2.0. Just to say, I've read some of the code of the cop and
it's quite clear: if it fails 2 times, it's killed. Didn't put attention on
whether it's using a separate connection though.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316783#comment-14316783
]
Luca Bruno commented on TS-3386:
I'm using latest 5.2.0. Just to say, I've read some of the code of the cop and
it's quite clear: if it fails 2 times, it's killed. Didn't put attention on
whether it's using a separate connection though.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server
Version: Apache Traffic
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316648#comment-14316648
]
Leif Hedstrom commented on TS-3386:
---
This is a known problem, if you starve the system on connections, so it starts
throttling, it'll kill traffic_server.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
[jira] [Reopened] (TS-3164) why the load of trafficserver occurrs a abrupt rise on a occasion ?
[
https://issues.apache.org/jira/browse/TS-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leif Hedstrom reopened TS-3164:
---
why the load of trafficserver occurrs a abrupt rise on a occasion ?
---
Key: TS-3164
URL: https://issues.apache.org/jira/browse/TS-3164
Project: Traffic Server
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64bit, 8 cores, 128G mem
Reporter: taoyunxing
Fix For: sometime
I use Tsar to monitor the traffic status of the ATS 4.2.0, and come across
the following problem:
{code}
Time ---cpu-- ---mem-- ---tcp-- -traffic --sda--- --sdb---
--sdc--- ---load-
Time util util retranbytin bytout util util
util load1
03/11/14-18:20 40.6787.19 3.3624.5M 43.9M13.0294.68
0.00 5.34
03/11/14-18:25 40.3087.20 3.2722.5M 42.6M12.3894.87
0.00 5.79
03/11/14-18:30 40.8484.67 3.4421.4M 42.0M13.2995.37
0.00 6.28
03/11/14-18:35 43.6387.36 3.2123.8M 45.0M13.2393.99
0.00 7.37
03/11/14-18:40 42.2587.37 3.0924.2M 44.8M12.8495.77
0.00 7.25
03/11/14-18:45 42.9687.44 3.4623.3M 46.0M12.9695.84
0.00 7.10
03/11/14-18:50 44.0087.42 3.4922.3M 43.0M14.1794.99
0.00 6.57
03/11/14-18:55 42.2087.44 3.4622.3M 43.6M13.1996.05
0.00 6.09
03/11/14-19:00 44.9087.53 3.6023.6M 46.5M13.6196.67
0.00 8.06
03/11/14-19:05 46.2687.73 3.2425.8M 49.1M15.3994.05
0.00 9.98
03/11/14-19:10 43.8587.69 3.1925.4M 50.9M12.8897.80
0.00 7.99
03/11/14-19:15 45.2887.69 3.3625.6M 49.6M13.1096.86
0.00 7.47
03/11/14-19:20 44.1185.20 3.2924.1M 47.8M14.2496.75
0.00 5.82
03/11/14-19:25 45.2687.78 3.5224.4M 47.7M13.2195.44
0.00 7.61
03/11/14-19:30 44.8387.80 3.6425.7M 50.8M13.2798.02
0.00 6.85
03/11/14-19:35 44.8987.78 3.6123.9M 49.0M13.3497.42
0.00 7.04
03/11/14-19:40 69.2188.88 0.5518.3M 33.7M11.3971.23
0.00 65.80
03/11/14-19:45 72.4788.66 0.2715.4M 31.6M11.5172.31
0.00 11.56
03/11/14-19:50 44.8788.72 4.1122.7M 46.3M12.9997.33
0.00 8.29
{code}
in addition, top command show
{code}
hi:0
ni:0
si:45.56
st:0
sy:13.92
us:12.58
wa:14.3
id:15.96
{code}
who help me ? thanks in advance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-3164) why the load of trafficserver occurrs a abrupt rise on a occasion ?
[
https://issues.apache.org/jira/browse/TS-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316230#comment-14316230
]
Leif Hedstrom commented on TS-3164:
---
I don't think it's true that no one is interested, no one else is seeing this
issue. Hard to debug / performance analyze something if you can't see it :).
why the load of trafficserver occurrs a abrupt rise on a occasion ?
---
Key: TS-3164
URL: https://issues.apache.org/jira/browse/TS-3164
Project: Traffic Server
Issue Type: Bug
Components: Core
Environment: CentOS 6.3 64bit, 8 cores, 128G mem
Reporter: taoyunxing
Fix For: sometime
I use Tsar to monitor the traffic status of the ATS 4.2.0, and come across
the following problem:
{code}
Time ---cpu-- ---mem-- ---tcp-- -traffic --sda--- --sdb---
--sdc--- ---load-
Time util util retranbytin bytout util util
util load1
03/11/14-18:20 40.6787.19 3.3624.5M 43.9M13.0294.68
0.00 5.34
03/11/14-18:25 40.3087.20 3.2722.5M 42.6M12.3894.87
0.00 5.79
03/11/14-18:30 40.8484.67 3.4421.4M 42.0M13.2995.37
0.00 6.28
03/11/14-18:35 43.6387.36 3.2123.8M 45.0M13.2393.99
0.00 7.37
03/11/14-18:40 42.2587.37 3.0924.2M 44.8M12.8495.77
0.00 7.25
03/11/14-18:45 42.9687.44 3.4623.3M 46.0M12.9695.84
0.00 7.10
03/11/14-18:50 44.0087.42 3.4922.3M 43.0M14.1794.99
0.00 6.57
03/11/14-18:55 42.2087.44 3.4622.3M 43.6M13.1996.05
0.00 6.09
03/11/14-19:00 44.9087.53 3.6023.6M 46.5M13.6196.67
0.00 8.06
03/11/14-19:05 46.2687.73 3.2425.8M 49.1M15.3994.05
0.00 9.98
03/11/14-19:10 43.8587.69 3.1925.4M 50.9M12.8897.80
0.00 7.99
03/11/14-19:15 45.2887.69 3.3625.6M 49.6M13.1096.86
0.00 7.47
03/11/14-19:20 44.1185.20 3.2924.1M 47.8M14.2496.75
0.00 5.82
03/11/14-19:25 45.2687.78 3.5224.4M 47.7M13.2195.44
0.00 7.61
03/11/14-19:30 44.8387.80 3.6425.7M 50.8M13.2798.02
0.00 6.85
03/11/14-19:35 44.8987.78 3.6123.9M 49.0M13.3497.42
0.00 7.04
03/11/14-19:40 69.2188.88 0.5518.3M 33.7M11.3971.23
0.00 65.80
03/11/14-19:45 72.4788.66 0.2715.4M 31.6M11.5172.31
0.00 11.56
03/11/14-19:50 44.8788.72 4.1122.7M 46.3M12.9997.33
0.00 8.29
{code}
in addition, top command show
{code}
hi:0
ni:0
si:45.56
st:0
sy:13.92
us:12.58
wa:14.3
id:15.96
{code}
who help me ? thanks in advance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316279#comment-14316279
]
Zhao Yongming commented on TS-3386:
---
well, the remap metters, please don't mess up 127.0.0.1 8080 with most of the
services, that is not what ATS working to as a proxy.
use something like map http://mydomain.com:8080/ . and do your testing
using modified /etc/hosts or -x 127.0.0.1:8080 in curl.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316350#comment-14316350
]
Luca Bruno edited comment on TS-3386 at 2/11/15 3:23 PM:
-
Even for 100 connections, it should limit the connections but not restarting
and lose the cache: that's what I call kidding. So if I have 3 like the
default config suggests, and it reaches the maximum, it would still restart.
I think I will disable the kill signal to the child after a failed heartbeat,
would you accept such kind of patch including a config option?
Or find a way to return 200 instead of 502 if a heartbeat request is recognized.
was (Author: lethalman):
Even for 100 connections, it should limit the connections but not restarting
and lose the cache: that's what I call kidding. So if I have 3 like the
default config suggests, and it reaches the maximum, it would still restart.
I think I will disable the kill signal to the child after a failed heartbeat,
would you accept such kind of patch including a config option?
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316316#comment-14316316
]
Luca Bruno edited comment on TS-3386 at 2/11/15 3:06 PM:
-
Ok my remap is now:
{noformat}
map http://test-cache.local:8080/ http://test-cache.local:8081/
{noformat}
Where test-cache.local is a real entry in a dns server of the local network.
Things didn't change much.
Btw I've found that in the error.log there's something valuable (it also
happened before), this is repeated indefinitely in the error.log:
{noformat}
20150211.15h53m56s RESPONSE: sent 192.168.199.31 status 502 (Connect Error
internal error - server connection terminated/-1) for
'http://test-cache.service.farm:8081/storage/test29429?size=11023age=46230sleep=0'
{noformat}
Please note that sending those concurrent requests to the backend server work
just fine without any error. Anyway, I believe an error in the backend should
not result in ATS being restarted.
Debug:
{noformat}
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_seq)
[HttpSM::do_hostdb_lookup] Doing DNS Lookup
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans)
[ink_cluster_time] local: 1423666415, highest_delta: 0, cluster: 1423666415
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans)
[HttpTransact::OSDNSLookup] This was attempt 1
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_seq)
[HttpTransact::OSDNSLookup] DNS Lookup successful
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans) [OSDNSLookup]
DNS lookup for O.S. successful IP: 192.168.x.y
{noformat}
Then I noticed this:
{noformat}
[Feb 11 15:53:35.043] Server {0x2b7ddc565e00} DEBUG: (http) [293] hostdb update
marking IP: 192.168.x.y:8081 as down
{noformat}
Also proxy.config.net.connections_throttle is 1000, and I do 1000 concurrent
requests in my tests. Increasing the throttle to 1 seems to mitigate the
problem.
was (Author: lethalman):
Ok my remap is now:
{noformat}
map http://test-cache.local:8080/ http://test-cache.local:8081/
{noformat}
Where test-cache.local is a real entry in a dns server of the local network.
Things didn't change much.
Btw I've found that in the error.log there's something valuable (it also
happened before), this is repeated indefinitely in the error.log:
{noformat}
20150211.15h53m56s RESPONSE: sent 192.168.199.31 status 502 (Connect Error
internal error - server connection terminated/-1) for
'http://test-cache.service.farm:8081/storage/test29429?size=11023age=46230sleep=0'
{noformat}
Please note that sending those concurrent requests to the backend server work
just fine without any error. Anyway, I believe an error in the backend should
not result in ATS being restarted.
Debug:
{noformat}
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_seq)
[HttpSM::do_hostdb_lookup] Doing DNS Lookup
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans)
[ink_cluster_time] local: 1423666415, highest_delta: 0, cluster: 1423666415
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans)
[HttpTransact::OSDNSLookup] This was attempt 1
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_seq)
[HttpTransact::OSDNSLookup] DNS Lookup successful
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans) [OSDNSLookup]
DNS lookup for O.S. successful IP: 192.168.x.y
{noformat}
Then I noticed this:
{noformat}
[Feb 11 15:53:35.043] Server {0x2b7ddc565e00} DEBUG: (http) [293] hostdb update
marking IP: 192.168.x.y:8081 as down
{noformat}
Also proxy.config.net.connections_throttle is 1000, and I do 1000 concurrent
requests in my tests.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[jira] [Resolved] (TS-3364) Add command line config validation support to traffic_server
[
https://issues.apache.org/jira/browse/TS-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudheer Vinukonda resolved TS-3364.
---
Resolution: Fixed
Add command line config validation support to traffic_server
Key: TS-3364
URL: https://issues.apache.org/jira/browse/TS-3364
Project: Traffic Server
Issue Type: Improvement
Components: Configuration
Affects Versions: 5.3.0
Reporter: Sudheer Vinukonda
Assignee: Sudheer Vinukonda
Fix For: 5.3.0
Currently, traffic_server fails to initialize when it encounters fatal errors
in loading the config files during start up. During dynamic reloading of
config files (e.g. via traffic_line), traffic_server rejects new config and
falls back to existing/old config (however, if there was a traffic_server
crash/restart subsequently, that can again result into failing to initialize).
This jira proposes to make the behavior of traffic_server when it encounters
such fatal errors configurable via a new setting
{{proxy.config.ignore_fatal_errors}} with the below options:
{code}
0 : All errors are fatal, do not load/reload
1 : Ignore a bad config line, continue with the rest
2 : Ignore a bad config line, stop parsing the file further
..
{code}
Based on concerns expressed, it has been agreed to not change the
traffic_server's behavior to loading with fatal config. Instead, this jira
will be used to add a command line option to traffic_server to load and
validate the config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316376#comment-14316376
]
Zhao Yongming commented on TS-3386:
---
if you want to talk about the kill, I'd like to say there should be more work
before taking down the server, but how would you know that the connection full
and all works well?
we have tried to put the heartbeat into a connection that will not be affect in
the connection limit, but sounds not so good too
the heart beat is a fake L7 service health checker, which is design to find out
something abnormal :D
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm]
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316343#comment-14316343
]
Zhao Yongming commented on TS-3386:
---
well, proxy.config.net.connections_throttle = 1000, are you kidding? ATS is not
squid nor httpd-1.x
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:44
[jira] [Commented] (TS-3353) traffic_via memory bug.
[ https://issues.apache.org/jira/browse/TS-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316304#comment-14316304 ] Leif Hedstrom commented on TS-3353: --- How can this possibly be a remote execution problem? traffic_via is a command line tool, that someone with physical access to the box must run explicitly. traffic_via memory bug. --- Key: TS-3353 URL: https://issues.apache.org/jira/browse/TS-3353 Project: Traffic Server Issue Type: Bug Reporter: Bin Assignee: James Peach Fix For: 5.3.0 Attachments: traffic_via_memory_overwrite_bug.diff A memory overwriting bug was found in traffic_via which can potentially lead to remote code execution. In function decodeViaHeader in file traffic_via.cc, a call to strcat can potentially overwrite outside the bounds on the heap. For example, if the via header string is abcde, the strcat concatenates a byte to the end and can corrupt the adjacent byte. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316350#comment-14316350
]
Luca Bruno commented on TS-3386:
Even for 100 connections, it should limit the connections but not restarting
and lose the cache: that's what I call kidding. So if I have 3 like the
default config suggests, and it reaches the maximum, it would still restart.
I think I will disable the kill signal to the child after a failed heartbeat,
would you accept such kind of patch including a config option?
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316388#comment-14316388
]
Luca Bruno commented on TS-3386:
We can find something abnormal by other means in our infrastructure. For
example it suffices to alert: heartbeat failed because of 502, instead of
killing the server. For us, it's better to wait than to lose terabytes of
cache, which would then kill the origin servers afterwards.
Also why isn't putting heartbeat in a dedicated connection not good? What
wasn't good when you tried it? Do you have some related work I can read?
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316238#comment-14316238
]
Zhao Yongming commented on TS-3386:
---
o, please just don't load any traffic and enable debug on http.*|dns.*, and I'd
suspect this is a HostDB reverse lookup on 127.0.0.1 or lookup on localhost
issue. let us dig it out.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server
Version: Apache Traffic Server -
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316316#comment-14316316
]
Luca Bruno commented on TS-3386:
Ok my remap is now:
{noformat}
map http://test-cache.local:8080/ http://test-cache.local:8081/
{noformat}
Where test-cache.local is a real entry in a dns server of the local network.
Things didn't change much.
Btw I've found that in the error.log there's something valuable (it also
happened before), this is repeated indefinitely in the error.log:
{noformat}
20150211.15h53m56s RESPONSE: sent 192.168.199.31 status 502 (Connect Error
internal error - server connection terminated/-1) for
'http://test-cache.service.farm:8081/storage/test29429?size=11023age=46230sleep=0'
{noformat}
Please note that sending those concurrent requests to the backend server work
just fine without any error. Anyway, I believe an error in the backend should
not result in ATS being restarted.
Debug:
{noformat}
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_seq)
[HttpSM::do_hostdb_lookup] Doing DNS Lookup
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans)
[ink_cluster_time] local: 1423666415, highest_delta: 0, cluster: 1423666415
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans)
[HttpTransact::OSDNSLookup] This was attempt 1
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_seq)
[HttpTransact::OSDNSLookup] DNS Lookup successful
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans) [OSDNSLookup]
DNS lookup for O.S. successful IP: 192.168.x.y
{noformat}
Then I noticed this:
{noformat}
[Feb 11 15:53:35.043] Server {0x2b7ddc565e00} DEBUG: (http) [293] hostdb update
marking IP: 192.168.x.y:8081 as down
{noformat}
{noformat}
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager -
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316266#comment-14316266
]
Luca Bruno commented on TS-3386:
Thanks a lot for your answer. I'm seeing this:
{noformat}
[Feb 11 15:31:01.819] Server {0x2b9677700e00} DEBUG: (http_trans) Next action
SM_ACTION_DNS_LOOKUP; OSDNSLookup
[Feb 11 15:31:01.819] Server {0x2b9677700e00} DEBUG: (http) [9] State
Transition: SM_ACTION_API_CACHE_LOOKUP_COMPLETE - SM_ACTION_DNS_LOOKUP
[Feb 11 15:31:01.819] Server {0x2b9677700e00} DEBUG: (http_seq)
[HttpSM::do_hostdb_lookup] Doing DNS Lookup
[Feb 11 15:31:01.819] Server {0x2b9677700e00} DEBUG: (http_trans)
[ink_cluster_time] local: 1423665061, highest_delta: 0, cluster: 1423665061
[Feb 11 15:31:01.819] Server {0x2b9677700e00} DEBUG: (http_trans)
[HttpTransact::OSDNSLookup] This was attempt 1
[Feb 11 15:31:01.819] Server {0x2b9677700e00} DEBUG: (http_seq)
[HttpTransact::OSDNSLookup] DNS Lookup successful
[Feb 11 15:31:01.819] Server {0x2b9677700e00} DEBUG: (http_trans) [OSDNSLookup]
DNS lookup for O.S. successful IP: 127.0.0.1
{noformat}
Should I be looking for something else? This is my remap:
{noformat}
map http://127.0.0.1:8080/ http://127.0.0.1:8081/
{noformat}
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316316#comment-14316316
]
Luca Bruno edited comment on TS-3386 at 2/11/15 3:04 PM:
-
Ok my remap is now:
{noformat}
map http://test-cache.local:8080/ http://test-cache.local:8081/
{noformat}
Where test-cache.local is a real entry in a dns server of the local network.
Things didn't change much.
Btw I've found that in the error.log there's something valuable (it also
happened before), this is repeated indefinitely in the error.log:
{noformat}
20150211.15h53m56s RESPONSE: sent 192.168.199.31 status 502 (Connect Error
internal error - server connection terminated/-1) for
'http://test-cache.service.farm:8081/storage/test29429?size=11023age=46230sleep=0'
{noformat}
Please note that sending those concurrent requests to the backend server work
just fine without any error. Anyway, I believe an error in the backend should
not result in ATS being restarted.
Debug:
{noformat}
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_seq)
[HttpSM::do_hostdb_lookup] Doing DNS Lookup
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans)
[ink_cluster_time] local: 1423666415, highest_delta: 0, cluster: 1423666415
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans)
[HttpTransact::OSDNSLookup] This was attempt 1
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_seq)
[HttpTransact::OSDNSLookup] DNS Lookup successful
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans) [OSDNSLookup]
DNS lookup for O.S. successful IP: 192.168.x.y
{noformat}
Then I noticed this:
{noformat}
[Feb 11 15:53:35.043] Server {0x2b7ddc565e00} DEBUG: (http) [293] hostdb update
marking IP: 192.168.x.y:8081 as down
{noformat}
Also proxy.config.net.connections_throttle is 1000, and I do 1000 concurrent
requests in my tests.
was (Author: lethalman):
Ok my remap is now:
{noformat}
map http://test-cache.local:8080/ http://test-cache.local:8081/
{noformat}
Where test-cache.local is a real entry in a dns server of the local network.
Things didn't change much.
Btw I've found that in the error.log there's something valuable (it also
happened before), this is repeated indefinitely in the error.log:
{noformat}
20150211.15h53m56s RESPONSE: sent 192.168.199.31 status 502 (Connect Error
internal error - server connection terminated/-1) for
'http://test-cache.service.farm:8081/storage/test29429?size=11023age=46230sleep=0'
{noformat}
Please note that sending those concurrent requests to the backend server work
just fine without any error. Anyway, I believe an error in the backend should
not result in ATS being restarted.
Debug:
{noformat}
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_seq)
[HttpSM::do_hostdb_lookup] Doing DNS Lookup
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans)
[ink_cluster_time] local: 1423666415, highest_delta: 0, cluster: 1423666415
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans)
[HttpTransact::OSDNSLookup] This was attempt 1
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_seq)
[HttpTransact::OSDNSLookup] DNS Lookup successful
[Feb 11 15:53:35.041] Server {0x2b7ddc565e00} DEBUG: (http_trans) [OSDNSLookup]
DNS lookup for O.S. successful IP: 192.168.x.y
{noformat}
Then I noticed this:
{noformat}
[Feb 11 15:53:35.043] Server {0x2b7ddc565e00} DEBUG: (http) [293] hostdb update
marking IP: 192.168.x.y:8081 as down
{noformat}
{noformat}
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720}
[jira] [Commented] (TS-2729) Add HTTP/2 support to ATS
[
https://issues.apache.org/jira/browse/TS-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316888#comment-14316888
]
Bryan Call commented on TS-2729:
I looked over the patch and it looks pretty good. I think it is good to add
into the ATS tree in its current state.
Add HTTP/2 support to ATS
-
Key: TS-2729
URL: https://issues.apache.org/jira/browse/TS-2729
Project: Traffic Server
Issue Type: New Feature
Components: HTTP/2
Reporter: Ryo Okubo
Assignee: James Peach
Labels: review
Fix For: 5.3.0
Attachments: 0003-h2-prototype.patch, 0004-h2-prototype.patch,
h2c_upgrade.patch, hpack.patch, http2-0004.patch, improve-mime.patch
h2. Overview
Support HTTP/2 as a client side L7 protocol. This feature is implemented into
ATS core.
Now, it supports the latest HTTP/2 draft version, h2-16.
https://tools.ietf.org/html/draft-ietf-httpbis-http2-16
h2. How to test
# Build ATS codes normally. you need neither any build option nor external
HTTP/2 library.
# Configure settings to use https.
# Add settings to records.config to use http2.
{noformat}
CONFIG proxy.config.http2.enabled INT 1
{noformat}
# Access to ATS by HTTP/2 client.
h2. Descriptions of current attached patches.
* 0003-h2-prototype.patch
** For experiment. Please don't merge it. It enables to interpret HTTP/2
requests and respond for it. But now this code is unsafe and dirty. More
refactoring is required.
h2. DONE
* Fundamental HTTP/2 frame handling
* Flow control
* Some error handlings
h2. TODO
* Refactoring
* More debugging
* Write documents
* Add test tools for HPACK, HTTP/2 frames
h2. No plan
* [Server
Push|https://tools.ietf.org/html/draft-ietf-httpbis-http2-16#section-8.2]
This would probably require support for [Link
preload|http://w3c.github.io/preload/#interoperability-with-http-link-header]?
* [Stream
Priority|https://tools.ietf.org/html/draft-ietf-httpbis-http2-16#section-5.3]
* [Alternative
Services|https://tools.ietf.org/html/draft-ietf-httpbis-alt-svc-06]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TS-3387) pass const arguments to ink_args
James Peach created TS-3387:
---
Summary: pass const arguments to ink_args
Key: TS-3387
URL: https://issues.apache.org/jira/browse/TS-3387
Project: Traffic Server
Issue Type: Bug
Components: Cleanup, Core
Reporter: James Peach
ink_args deals with arrays of {{char *}}, but since it doesn't modify anything,
it is much easier for it to deal with arrays of {{const char *}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (TS-3387) pass const arguments to ink_args
[
https://issues.apache.org/jira/browse/TS-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Peach resolved TS-3387.
-
Resolution: Fixed
Fix Version/s: 5.3.0
Assignee: James Peach
pass const arguments to ink_args
Key: TS-3387
URL: https://issues.apache.org/jira/browse/TS-3387
Project: Traffic Server
Issue Type: Bug
Components: Cleanup, Core
Reporter: James Peach
Assignee: James Peach
Fix For: 5.3.0
ink_args deals with arrays of {{char *}}, but since it doesn't modify
anything, it is much easier for it to deal with arrays of {{const char *}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-3387) pass const arguments to ink_args
[
https://issues.apache.org/jira/browse/TS-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317282#comment-14317282
]
ASF subversion and git services commented on TS-3387:
-
Commit 453e3047b0174ef2f0c90335d0ad18f64678cd25 in trafficserver's branch
refs/heads/master from [~jpe...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=453e304 ]
TS-3387: convert ink_args to accept an array of constant strings
pass const arguments to ink_args
Key: TS-3387
URL: https://issues.apache.org/jira/browse/TS-3387
Project: Traffic Server
Issue Type: Bug
Components: Cleanup, Core
Reporter: James Peach
Assignee: James Peach
Fix For: 5.3.0
ink_args deals with arrays of {{char *}}, but since it doesn't modify
anything, it is much easier for it to deal with arrays of {{const char *}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Build failed in Jenkins: tsqa-master #141
See https://ci.trafficserver.apache.org/job/tsqa-master/141/changes Changes: [James Peach] TS-3387: convert ink_args to accept an array of constant strings -- [...truncated 13159 lines...] 10 670 3353.800 51757260/5175726 2158 2158 2158 84554882 12.90 10 677 3387.100 51556796/5155679 2181 2181 2181 83921115 13.90 10 683 3419.500 52805571/5280557 2209 2209 2210 86195999 14.90 10 694 3475.600 52941045/5294104 2243 2243 2242 86617968 15.90 10 698 3495.300 52538698/5253869 2245 2245 2246 85621293 16.90 10 703 3516.800 52558730/5255873 2256 2256 2255 85879358 17.90 10 708 3542.600 51738208/5173820 2266 2266 2266 84335947 18.90 10 708 3542.100 52408035/5240803 2277 2277 2277 86149455 19.90 con new ops 1B lat bytes/per svrs new ops total time err 10 707 3537.100 51917472/5191747 2266 2266 2266 85049530 20.90 10 715 3576.500 51299637/5129963 2292 2292 2292 83751676 21.90 10 714 3572.400 50599892/5059989 2292 2292 2292 82902964 22.90 10 711 3557.700 51558626/5155862 2277 2277 2276 84497267 23.90 10 708 3544.600 52125917/5212591 2276 2276 2276 85608011 24.90 10 702 3511.200 53600889/5360088 2262 2262 2262 88695258 25.90 10 702 3513.800 53910259/5391025 2255 2255 2256 89354827 26.90 MSG: reloading logging configuration 10 703 3516.800 53430583/5343058 2266 2266 2266 87968527 27.90 10 706 3534.700 53341373/5334137 2271 2271 2271 87523713 28.90 10 710 3551.700 52986696/5298669 2279 2279 2279 87120359 29.90 10 714 3571.000 52791298/5279129 2290 2290 2290 86651965 30.90 10 713 3568.200 53103779/5310377 2278 2278 2278 86433749 31.90 10 712 3561.200 53753396/5375339 2273 2273 2273 87395160 32.90 10 712 3562.100 53715959/5371595 2285 2285 2285 87572669 33.90 10 710 3555.500 53732832/5373283 2272 2272 2272 87175589 34.90 10 708 3543.800 53275375/5327537 2261 2261 2261 86138397 35.90 10 704 3521.200 54185207/5418520 2247 2247 2247 87992239 36.90 10 711 3558.400 55104042/5510404 2279 2279 2279 89994372 37.90 10 713 3567.500 54072861/5407286 2279 2279 2278 88456719 38.90 10 708 3543.400 53376234/5337623 2256 2256 2256 87651748 39.90 con new ops 1B lat bytes/per svrs new ops total time err 10 707 3540.300 53169497/5316949 2254 2254 2255 87139813 40.90 10 710 3553.900 53163238/5316323 2261 2261 2261 87379569 41.90 10 709 3549.300 52936753/5293675 2256 2256 2256 87075544 42.90 10 709 3548.800 53331809/5333180 2259 2259 2259 88107692 43.90 10 704 3524.500 53309815/5330981 2242 2242 2242 88366796 44.90 10 704 3524.000 54887508/5488750 2245 2245 2245 91632634 45.90 10 703 3516.400 55276024/5527602 2239 2239 2239 91953147 46.90 10 696 3483.900 55609184/5560918 2232 2232 2232 92242750 47.90 10 689 3450.100 54714468/5471446 2219 2219 2219 90784762 48.90 10 693 3468.300 53452235/5345223 2231 2231 2231 88858128 49.90 10 692 3466.000 53639994/5363999 2230 2230 2231 89450716 50.90 10 690 3453.800 53652681/5365268 2230 2230 2230 89535997 51.90 10 688 3446.900 53593432/5359343 2231 2231 2231 89184629 52.90 10 684 3422.300 54530019/5453001 2216 2216 2216 90333723 53.90 10 684 3420.400 54925072/5492507 2221 2221 2221 90465147 54.90 10 687 3439.500 54819919/5481991 2225 2225 2225 89896632 55.90 10 690 3452.000 54906487/5490648 2235 2235 2234 90495471 56.90 MSG: reloading logging configuration 10 694 3475.800 54582091/5458209 2251 2251 2251 89917309 57.90 10 696 3484.700 54159612/5415961 2246 2246 2245 88843777 58.90 10 694 3474.500 54219132/5421913 2234 2234 2234 89647757 59.90 con new ops 1B lat bytes/per svrs new ops total time err 10 701 3510.200 54350190/5435019 2269 2269 2269 90135800 60.90 10 561 2808.200 43480152/4348015 1815 1815 1815 72108640 60.90 MSG: checking for crashes ... ./functions: line 177: 26084 Terminated (
Jenkins build is back to normal : tsqa-master #142
See https://ci.trafficserver.apache.org/job/tsqa-master/142/
Jenkins build is back to normal : ubuntu_14_10-master » clang,ubuntu_14_10,release #200
See https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=release/200/
Jenkins build is back to normal : fedora_21-master » clang,fedora_21,release #173
See https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=release/173/
Build failed in Jenkins: freebsd_10-master » clang,freebsd_10,release #738
See https://ci.trafficserver.apache.org/job/freebsd_10-master/compiler=clang,label=freebsd_10,type=release/738/ -- [...truncated 2648 lines...] depbase=`echo test/print_funcs.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/sh ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -I../../../../plugins/experimental/esi/lib -I../../../../plugins/experimental/esi/fetcher -I../../../../plugins/experimental/esi/test -I/usr/local/include -Dfreebsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.6 -Wno-deprecated -Qunused-arguments -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT test/print_funcs.lo -MD -MP -MF $depbase.Tpo -c -o test/print_funcs.lo ../../../../plugins/experimental/esi/test/print_funcs.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -I../../../../plugins/experimental/esi/lib -I../../../../plugins/experimental/esi/fetcher -I../../../../plugins/experimental/esi/test -I/usr/local/include -Dfreebsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.6 -Wno-deprecated -Qunused-arguments -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT lib/gzip.lo -MD -MP -MF lib/.deps/gzip.Tpo -c ../../../../plugins/experimental/esi/lib/gzip.cc -fPIC -DPIC -o lib/.libs/gzip.o depbase=`echo test/HandlerMap.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/sh ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -I../../../../plugins/experimental/esi/lib -I../../../../plugins/experimental/esi/fetcher -I../../../../plugins/experimental/esi/test -I/usr/local/include -Dfreebsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.6 -Wno-deprecated -Qunused-arguments -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT test/HandlerMap.lo -MD -MP -MF $depbase.Tpo -c -o test/HandlerMap.lo ../../../../plugins/experimental/esi/test/HandlerMap.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -I../../../../plugins/experimental/esi/lib -I../../../../plugins/experimental/esi/fetcher -I../../../../plugins/experimental/esi/test -I/usr/local/include -Dfreebsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.6 -Wno-deprecated -Qunused-arguments -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT test/print_funcs.lo -MD -MP -MF test/.deps/print_funcs.Tpo -c ../../../../plugins/experimental/esi/test/print_funcs.cc -fPIC -DPIC -o test/.libs/print_funcs.o depbase=`echo test/StubIncludeHandler.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/sh ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -I../../../../plugins/experimental/esi/lib -I../../../../plugins/experimental/esi/fetcher -I../../../../plugins/experimental/esi/test -I/usr/local/include -Dfreebsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.6 -Wno-deprecated -Qunused-arguments -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT test/StubIncludeHandler.lo -MD -MP -MF $depbase.Tpo -c -o test/StubIncludeHandler.lo ../../../../plugins/experimental/esi/test/StubIncludeHandler.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi
Build failed in Jenkins: fedora_21-master » clang,fedora_21,debug #172
See https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=debug/172/ -- [...truncated 2670 lines...] libtool: link: ( cd .libs rm -f combo_handler.la ln -s ../combo_handler.la combo_handler.la ) libtool: link: ( cd .libs rm -f esi.la ln -s ../esi.la esi.la ) make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=debug/ws/172/build/BUILDS/plugins/experimental/esi' Making all in generator make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=debug/ws/172/build/BUILDS/plugins/experimental/generator' depbase=`echo generator.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/sh ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/generator -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -Qunused-arguments -std=c++11 -ggdb3 -Werror -Wno-invalid-offsetof -mcx16 -MT generator.lo -MD -MP -MF $depbase.Tpo -c -o generator.lo ../../../../plugins/experimental/generator/generator.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/generator -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -Qunused-arguments -std=c++11 -ggdb3 -Werror -Wno-invalid-offsetof -mcx16 -MT generator.lo -MD -MP -MF .deps/generator.Tpo -c ../../../../plugins/experimental/generator/generator.cc -fPIC -DPIC -o .libs/generator.o /bin/sh ../../../libtool --tag=CXX --mode=link clang++ -Qunused-arguments -std=c++11 -ggdb3 -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -o generator.la -rpath https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=debug/ws/172/install/libexec/trafficserver generator.lo -lcap -lpcre -lexpat -llzma -lz -lcrypt -lpthread -ldl libtool: link: /usr/bin/nm -B .libs/generator.o | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /usr/bin/sed 's/.* //' | sort | uniq .libs/generator.exp libtool: link: /usr/bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/generator.exp .libs/generator.expT libtool: link: mv -f .libs/generator.expT .libs/generator.exp libtool: link: clang++ -fPIC -DPIC -shared -nostdlib /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64/crti.o /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/crtbeginS.o .libs/generator.o -lcap -lpcre -lexpat -llzma -lz -lcrypt -lpthread -ldl -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2 -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64 -L/usr/bin/../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../.. -L/usr/bin/../lib -L/lib -L/usr/lib -lstdc++ -lm -lc -lgcc_s /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/crtendS.o /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64/crtn.o -mcx16 -Wl,-soname -Wl,generator.so -Wl,-retain-symbols-file -Wl,.libs/generator.exp -o .libs/generator.so libtool: link: ( cd .libs rm -f generator.la ln -s ../generator.la generator.la ) make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=debug/ws/172/build/BUILDS/plugins/experimental/generator' Making all in geoip_acl make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=debug/ws/172/build/BUILDS/plugins/experimental/geoip_acl' make[3]: Nothing to be done for 'all'. make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=debug/ws/172/build/BUILDS/plugins/experimental/geoip_acl' Making all in header_normalize make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=debug/ws/172/build/BUILDS/plugins/experimental/header_normalize' depbase=`echo header_normalize.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/sh ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I.
Build failed in Jenkins: freebsd_10-master » clang,freebsd_10,debug #738
See https://ci.trafficserver.apache.org/job/freebsd_10-master/compiler=clang,label=freebsd_10,type=debug/738/ -- [...truncated 2652 lines...] libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -I../../../../plugins/experimental/esi/lib -I../../../../plugins/experimental/esi/fetcher -I../../../../plugins/experimental/esi/test -I/usr/local/include -Dfreebsd -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.6 -Wno-deprecated -Qunused-arguments -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT lib/gzip.lo -MD -MP -MF lib/.deps/gzip.Tpo -c ../../../../plugins/experimental/esi/lib/gzip.cc -fPIC -DPIC -o lib/.libs/gzip.o libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -I../../../../plugins/experimental/esi/lib -I../../../../plugins/experimental/esi/fetcher -I../../../../plugins/experimental/esi/test -I/usr/local/include -Dfreebsd -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.6 -Wno-deprecated -Qunused-arguments -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT test/print_funcs.lo -MD -MP -MF test/.deps/print_funcs.Tpo -c ../../../../plugins/experimental/esi/test/print_funcs.cc -fPIC -DPIC -o test/.libs/print_funcs.o depbase=`echo test/HandlerMap.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/sh ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -I../../../../plugins/experimental/esi/lib -I../../../../plugins/experimental/esi/fetcher -I../../../../plugins/experimental/esi/test -I/usr/local/include -Dfreebsd -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.6 -Wno-deprecated -Qunused-arguments -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT test/HandlerMap.lo -MD -MP -MF $depbase.Tpo -c -o test/HandlerMap.lo ../../../../plugins/experimental/esi/test/HandlerMap.cc \ mv -f $depbase.Tpo $depbase.Plo depbase=`echo test/StubIncludeHandler.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/sh ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -I../../../../plugins/experimental/esi/lib -I../../../../plugins/experimental/esi/fetcher -I../../../../plugins/experimental/esi/test -I/usr/local/include -Dfreebsd -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.6 -Wno-deprecated -Qunused-arguments -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT test/StubIncludeHandler.lo -MD -MP -MF $depbase.Tpo -c -o test/StubIncludeHandler.lo ../../../../plugins/experimental/esi/test/StubIncludeHandler.cc \ mv -f $depbase.Tpo $depbase.Plo depbase=`echo test/TestHandlerManager.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/sh ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -I../../../../plugins/experimental/esi/lib -I../../../../plugins/experimental/esi/fetcher -I../../../../plugins/experimental/esi/test -I/usr/local/include -Dfreebsd -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.6 -Wno-deprecated -Qunused-arguments -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT test/TestHandlerManager.lo -MD -MP -MF $depbase.Tpo -c -o test/TestHandlerManager.lo ../../../../plugins/experimental/esi/test/TestHandlerManager.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/esi -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib
Build failed in Jenkins: ubuntu_14_04-master » gcc,ubuntu_14_04,debug #741
See https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=debug/741/ -- [...truncated 2670 lines...] Making all in geoip_acl make[3]: Entering directory `https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=debug/ws/741/build/BUILDS/plugins/experimental/geoip_acl' make[3]: Nothing to be done for `all'. make[3]: Leaving directory `https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=debug/ws/741/build/BUILDS/plugins/experimental/geoip_acl' Making all in header_normalize make[3]: Entering directory `https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=debug/ws/741/build/BUILDS/plugins/experimental/header_normalize' depbase=`echo header_normalize.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CXX --mode=compile ccache c++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Wunused-parameter -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF $depbase.Tpo -c -o header_normalize.lo ../../../../plugins/experimental/header_normalize/header_normalize.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: ccache c++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Wunused-parameter -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF .deps/header_normalize.Tpo -c ../../../../plugins/experimental/header_normalize/header_normalize.cc -fPIC -DPIC -o .libs/header_normalize.o /bin/bash ../../../libtool --tag=CXX --mode=link ccache c++ -Wunused-parameter -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -o header_normalize.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=debug/ws/741/install/libexec/trafficserver header_normalize.lo -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/header_normalize.o | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/header_normalize.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/header_normalize.exp .libs/header_normalize.expT libtool: link: mv -f .libs/header_normalize.expT .libs/header_normalize.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o .libs/header_normalize.o -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o -mcx16 -Wl,-soname -Wl,header_normalize.so -Wl,-retain-symbols-file -Wl,.libs/header_normalize.exp -o .libs/header_normalize.so libtool: link: ( cd .libs rm -f header_normalize.la ln -s ../header_normalize.la header_normalize.la ) make[3]: Leaving directory `https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=debug/ws/741/build/BUILDS/plugins/experimental/header_normalize' Making all in healthchecks make[3]: Entering directory `https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=debug/ws/741/build/BUILDS/plugins/experimental/healthchecks' depbase=`echo healthchecks.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CC --mode=compile ccache cc -DHAVE_CONFIG_H -I.
Build failed in Jenkins: ubuntu_14_10-master » gcc,ubuntu_14_10,debug #199
See https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=debug/199/ -- [...truncated 2678 lines...] /bin/bash ../../../libtool --tag=CXX --mode=compile ccache c++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Wunused-parameter -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF $depbase.Tpo -c -o header_normalize.lo ../../../../plugins/experimental/header_normalize/header_normalize.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: ccache c++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Wunused-parameter -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF .deps/header_normalize.Tpo -c ../../../../plugins/experimental/header_normalize/header_normalize.cc -fPIC -DPIC -o .libs/header_normalize.o /bin/bash ../../../libtool --tag=CXX --mode=link ccache c++ -Wunused-parameter -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -o header_normalize.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=debug/ws/199/install/libexec/trafficserver header_normalize.lo -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/header_normalize.o | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/header_normalize.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/header_normalize.exp .libs/header_normalize.expT libtool: link: mv -f .libs/header_normalize.expT .libs/header_normalize.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.9/crtbeginS.o .libs/header_normalize.o -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.9/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crtn.o -mcx16 -Wl,-soname -Wl,header_normalize.so -Wl,-retain-symbols-file -Wl,.libs/header_normalize.exp -o .libs/header_normalize.so libtool: link: ( cd .libs rm -f header_normalize.la ln -s ../header_normalize.la header_normalize.la ) make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=debug/ws/199/build/BUILDS/plugins/experimental/header_normalize' Making all in healthchecks make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=debug/ws/199/build/BUILDS/plugins/experimental/healthchecks' depbase=`echo healthchecks.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CC --mode=compile ccache cc -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/healthchecks -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Wunused-parameter -ggdb3 -pipe -Wall -Werror -mcx16 -MT healthchecks.lo -MD -MP -MF $depbase.Tpo -c -o healthchecks.lo ../../../../plugins/experimental/healthchecks/healthchecks.c \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: ccache cc -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/healthchecks -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api
Jenkins build is back to normal : ubuntu_14_10-master » clang,ubuntu_14_10,spdy #200
See https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=spdy/200/
Jenkins build is back to normal : ubuntu_14_10-master » gcc,ubuntu_14_10,spdy #200
See https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=spdy/200/
Jenkins build is back to normal : ubuntu_12_04-master » gcc,ubuntu_12_04,release #720
See https://ci.trafficserver.apache.org/job/ubuntu_12_04-master/compiler=gcc,label=ubuntu_12_04,type=release/720/
Jenkins build is back to normal : fedora_21-master » clang,fedora_21,spdy #173
See https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=spdy/173/
Build failed in Jenkins: ubuntu_12_04-master » gcc,ubuntu_12_04,debug #719
See https://ci.trafficserver.apache.org/job/ubuntu_12_04-master/compiler=gcc,label=ubuntu_12_04,type=debug/719/ -- [...truncated 2655 lines...] libtool: link: ranlib .libs/libtest.a libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/combo_handler.exp .libs/combo_handler.expT libtool: link: ( cd .libs rm -f libtest.la ln -s ../libtest.la libtest.la ) libtool: link: mv -f .libs/combo_handler.expT .libs/combo_handler.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.6/crtbeginS.o .libs/combo_handler.o fetcher/.libs/HttpDataFetcherImpl.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -L/usr/lib -lcap -lpcre -lz -lcrypt -lpthread -lrt -ldl /usr/lib/x86_64-linux-gnu/libxml2.so -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.6/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crtn.o -mcx16 -Wl,-soname -Wl,combo_handler.so -Wl,-retain-symbols-file -Wl,.libs/combo_handler.exp -o .libs/combo_handler.so libtool: link: ( cd .libs rm -f combo_handler.la ln -s ../combo_handler.la combo_handler.la ) /bin/bash ../../../libtool --tag=CXX --mode=link ccache c++ -Wno-deprecated -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -L/usr/lib -o esi.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_12_04-master/compiler=gcc,label=ubuntu_12_04,type=debug/ws/719/install/libexec/trafficserver esi.lo fetcher/HttpDataFetcherImpl.lo serverIntercept.lo libesicore.la -lcap -lpcre -lz -lcrypt -lpthread -lrt -lrt -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/esi.o fetcher/.libs/HttpDataFetcherImpl.o .libs/serverIntercept.o ./.libs/libesicore.a | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/esi.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/esi.exp .libs/esi.expT libtool: link: mv -f .libs/esi.expT .libs/esi.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.6/crtbeginS.o .libs/esi.o fetcher/.libs/HttpDataFetcherImpl.o .libs/serverIntercept.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -L/usr/lib -lcap -lpcre -lz -lcrypt -lpthread -lrt -ldl /usr/lib/x86_64-linux-gnu/libxml2.so -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.6/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crtn.o -mcx16 -Wl,-soname -Wl,esi.so -Wl,-retain-symbols-file -Wl,.libs/esi.exp -o .libs/esi.so libtool: link: ( cd .libs rm -f esi.la ln -s ../esi.la esi.la ) make[3]: Leaving directory `https://ci.trafficserver.apache.org/job/ubuntu_12_04-master/compiler=gcc,label=ubuntu_12_04,type=debug/ws/719/build/BUILDS/plugins/experimental/esi' Making all in generator make[3]: Entering directory `https://ci.trafficserver.apache.org/job/ubuntu_12_04-master/compiler=gcc,label=ubuntu_12_04,type=debug/ws/719/build/BUILDS/plugins/experimental/generator' depbase=`echo generator.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CXX --mode=compile ccache c++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/generator -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -I/usr/include -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.5 -I/usr/include/libxml2 -Wunused-parameter -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT generator.lo -MD -MP -MF $depbase.Tpo -c -o generator.lo ../../../../plugins/experimental/generator/generator.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: ccache c++ -DHAVE_CONFIG_H -I.
Build failed in Jenkins: ubuntu_14_10-master » gcc,ubuntu_14_10,spdy #199
See
https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=spdy/199/
--
[...truncated 2694 lines...]
libtool: link: mv -f .libs/header_normalize.expT .libs/header_normalize.exp
libtool: link: c++ -fPIC -DPIC -shared -nostdlib
/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/4.9/crtbeginS.o .libs/header_normalize.o -lcap
-lpcre -lz -lcrypt -lpthread -ldl -lxml2 -L/usr/lib/gcc/x86_64-linux-gnu/4.9
-L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu
-L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib -L/lib/x86_64-linux-gnu
-L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib
-L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../.. -lstdc++ -lm -lc -lgcc_s
/usr/lib/gcc/x86_64-linux-gnu/4.9/crtendS.o
/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crtn.o -O3 -mcx16
-Wl,-soname -Wl,header_normalize.so -Wl,-retain-symbols-file
-Wl,.libs/header_normalize.exp -o .libs/header_normalize.so
libtool: link: ( cd .libs rm -f header_normalize.la ln -s
../header_normalize.la header_normalize.la )
make[3]: Leaving directory
'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=spdy/ws/199/build/BUILDS/plugins/experimental/header_normalize'
Making all in healthchecks
make[3]: Entering directory
'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=spdy/ws/199/build/BUILDS/plugins/experimental/healthchecks'
depbase=`echo healthchecks.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\
/bin/bash ../../../libtool --tag=CC --mode=compile ccache cc -DHAVE_CONFIG_H
-I. -I../../../../plugins/experimental/healthchecks -I../../../lib/ts
-I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts
-I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1
-D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1
-D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6
-I/usr/include/libxml2 -Wunused-parameter -g -pipe -Wall -O3
-feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -mcx16 -MT
healthchecks.lo -MD -MP -MF $depbase.Tpo -c -o healthchecks.lo
../../../../plugins/experimental/healthchecks/healthchecks.c \
mv -f $depbase.Tpo $depbase.Plo
libtool: compile: ccache cc -DHAVE_CONFIG_H -I.
-I../../../../plugins/experimental/healthchecks -I../../../lib/ts
-I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts
-I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1
-D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1
-D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6
-I/usr/include/libxml2 -Wunused-parameter -g -pipe -Wall -O3
-feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -mcx16 -MT
healthchecks.lo -MD -MP -MF .deps/healthchecks.Tpo -c
../../../../plugins/experimental/healthchecks/healthchecks.c -fPIC -DPIC -o
.libs/healthchecks.o
/bin/bash ../../../libtool --tag=CC --mode=link ccache cc -Wunused-parameter
-g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing
-Werror -mcx16 -module -shared -avoid-version -export-symbols-regex
'^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$'
-o healthchecks.la -rpath
https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=spdy/ws/199/install/libexec/trafficserver
healthchecks.lo -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2
libtool: link: /usr/bin/nm -B .libs/healthchecks.o | sed -n -e 's/^.*[
]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][
]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed
's/.* //' | sort | uniq .libs/healthchecks.exp
libtool: link: /bin/grep -E -e
^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$
.libs/healthchecks.exp .libs/healthchecks.expT
libtool: link: mv -f .libs/healthchecks.expT .libs/healthchecks.exp
libtool: link: echo { global: .libs/healthchecks.ver
libtool: link: cat .libs/healthchecks.exp | sed -e s/\(.*\)/\1;/
.libs/healthchecks.ver
libtool: link: echo local: *; }; .libs/healthchecks.ver
libtool: link: cc -shared -fPIC -DPIC .libs/healthchecks.o -lcap -lpcre
-lz -lcrypt -lpthread -ldl -lxml2 -O3 -mcx16 -Wl,-soname -Wl,healthchecks.so
-Wl,-version-script -Wl,.libs/healthchecks.ver -o .libs/healthchecks.so
libtool: link: ( cd .libs rm -f healthchecks.la ln -s
../healthchecks.la healthchecks.la )
make[3]: Leaving directory
'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=spdy/ws/199/build/BUILDS/plugins/experimental/healthchecks'
Making all in hipes
make[3]: Entering directory
Build failed in Jenkins: ubuntu_14_10-master » clang,ubuntu_14_10,spdy #199
See https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=spdy/199/ -- [...truncated 2690 lines...] make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=spdy/ws/199/build/BUILDS/plugins/experimental/geoip_acl' Making all in header_normalize make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=spdy/ws/199/build/BUILDS/plugins/experimental/header_normalize' depbase=`echo header_normalize.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF $depbase.Tpo -c -o header_normalize.lo ../../../../plugins/experimental/header_normalize/header_normalize.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF .deps/header_normalize.Tpo -c ../../../../plugins/experimental/header_normalize/header_normalize.cc -fPIC -DPIC -o .libs/header_normalize.o /bin/bash ../../../libtool --tag=CXX --mode=link clang++ -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -o header_normalize.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=spdy/ws/199/install/libexec/trafficserver header_normalize.lo -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/header_normalize.o | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/header_normalize.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/header_normalize.exp .libs/header_normalize.expT libtool: link: mv -f .libs/header_normalize.expT .libs/header_normalize.exp libtool: link: clang++ -fPIC -DPIC -shared -nostdlib /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtbeginS.o .libs/header_normalize.o -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../.. -L/usr/lib/llvm-3.5/bin/../lib -L/lib -L/usr/lib -lstdc++ -lm -lc -lgcc_s /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtendS.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crtn.o -O3 -mcx16 -Wl,-soname -Wl,header_normalize.so -Wl,-retain-symbols-file -Wl,.libs/header_normalize.exp -o .libs/header_normalize.so libtool: link: ( cd .libs rm -f header_normalize.la ln -s ../header_normalize.la header_normalize.la ) make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=spdy/ws/199/build/BUILDS/plugins/experimental/header_normalize' Making all in healthchecks make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=spdy/ws/199/build/BUILDS/plugins/experimental/healthchecks' depbase=`echo healthchecks.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CC --mode=compile clang -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/healthchecks -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT
Build failed in Jenkins: ubuntu_12_04-master » gcc,ubuntu_12_04,release #719
See https://ci.trafficserver.apache.org/job/ubuntu_12_04-master/compiler=gcc,label=ubuntu_12_04,type=release/719/ -- [...truncated 2654 lines...] /bin/bash ../../../libtool --tag=CXX --mode=link ccache c++ -Wno-deprecated -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -L/usr/lib -o combo_handler.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_12_04-master/compiler=gcc,label=ubuntu_12_04,type=release/ws/719/install/libexec/trafficserver combo_handler.lo fetcher/HttpDataFetcherImpl.lo libesicore.la -lcap -lpcre -lz -lcrypt -lpthread -lrt -lrt -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/combo_handler.o fetcher/.libs/HttpDataFetcherImpl.o ./.libs/libesicore.a | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/combo_handler.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/combo_handler.exp .libs/combo_handler.expT libtool: link: mv -f .libs/combo_handler.expT .libs/combo_handler.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.6/crtbeginS.o .libs/combo_handler.o fetcher/.libs/HttpDataFetcherImpl.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -L/usr/lib -lcap -lpcre -lz -lcrypt -lpthread -lrt -ldl /usr/lib/x86_64-linux-gnu/libxml2.so -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.6/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crtn.o -O3 -mcx16 -Wl,-soname -Wl,combo_handler.so -Wl,-retain-symbols-file -Wl,.libs/combo_handler.exp -o .libs/combo_handler.so libtool: link: ( cd .libs rm -f combo_handler.la ln -s ../combo_handler.la combo_handler.la ) /bin/bash ../../../libtool --tag=CXX --mode=link ccache c++ -Wno-deprecated -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -L/usr/lib -o esi.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_12_04-master/compiler=gcc,label=ubuntu_12_04,type=release/ws/719/install/libexec/trafficserver esi.lo fetcher/HttpDataFetcherImpl.lo serverIntercept.lo libesicore.la -lcap -lpcre -lz -lcrypt -lpthread -lrt -lrt -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/esi.o fetcher/.libs/HttpDataFetcherImpl.o .libs/serverIntercept.o ./.libs/libesicore.a | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/esi.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/esi.exp .libs/esi.expT libtool: link: mv -f .libs/esi.expT .libs/esi.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.6/crtbeginS.o .libs/esi.o fetcher/.libs/HttpDataFetcherImpl.o .libs/serverIntercept.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -L/usr/lib -lcap -lpcre -lz -lcrypt -lpthread -lrt -ldl /usr/lib/x86_64-linux-gnu/libxml2.so -L/usr/lib/gcc/x86_64-linux-gnu/4.6 -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.6/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.6/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/crtn.o -O3 -mcx16 -Wl,-soname -Wl,esi.so -Wl,-retain-symbols-file -Wl,.libs/esi.exp -o .libs/esi.so libtool: link: ( cd .libs rm -f esi.la ln -s ../esi.la esi.la ) make[3]: Leaving directory `https://ci.trafficserver.apache.org/job/ubuntu_12_04-master/compiler=gcc,label=ubuntu_12_04,type=release/ws/719/build/BUILDS/plugins/experimental/esi' Making all in generator make[3]: Entering directory
Build failed in Jenkins: fedora_21-master » clang,fedora_21,spdy #172
See https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=spdy/172/ -- [...truncated 2674 lines...] libtool: link: /usr/bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/esi.exp .libs/esi.expT libtool: link: /usr/bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/combo_handler.exp .libs/combo_handler.expT libtool: link: mv -f .libs/esi.expT .libs/esi.exp libtool: link: clang++ -fPIC -DPIC -shared -nostdlib /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64/crti.o /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/crtbeginS.o .libs/esi.o fetcher/.libs/HttpDataFetcherImpl.o .libs/serverIntercept.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -lcap -lpcre -lexpat -llzma -lz -lcrypt -lpthread -ldl -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2 -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64 -L/usr/bin/../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../.. -L/usr/bin/../lib -L/lib -L/usr/lib -lstdc++ -lm -lc -lgcc_s /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/crtendS.o /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64/crtn.o -O3 -mcx16 -Wl,-soname -Wl,esi.so -Wl,-retain-symbols-file -Wl,.libs/esi.exp -o .libs/esi.so libtool: link: mv -f .libs/combo_handler.expT .libs/combo_handler.exp libtool: link: clang++ -fPIC -DPIC -shared -nostdlib /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64/crti.o /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/crtbeginS.o .libs/combo_handler.o fetcher/.libs/HttpDataFetcherImpl.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -lcap -lpcre -lexpat -llzma -lz -lcrypt -lpthread -ldl -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2 -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64 -L/usr/bin/../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../.. -L/usr/bin/../lib -L/lib -L/usr/lib -lstdc++ -lm -lc -lgcc_s /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/crtendS.o /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64/crtn.o -O3 -mcx16 -Wl,-soname -Wl,combo_handler.so -Wl,-retain-symbols-file -Wl,.libs/combo_handler.exp -o .libs/combo_handler.so libtool: link: ( cd .libs rm -f combo_handler.la ln -s ../combo_handler.la combo_handler.la ) libtool: link: ( cd .libs rm -f esi.la ln -s ../esi.la esi.la ) make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=spdy/ws/172/build/BUILDS/plugins/experimental/esi' Making all in generator make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=spdy/ws/172/build/BUILDS/plugins/experimental/generator' depbase=`echo generator.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/sh ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/generator -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT generator.lo -MD -MP -MF $depbase.Tpo -c -o generator.lo ../../../../plugins/experimental/generator/generator.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/generator -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT generator.lo -MD -MP -MF .deps/generator.Tpo -c ../../../../plugins/experimental/generator/generator.cc -fPIC -DPIC -o .libs/generator.o /bin/sh ../../../libtool --tag=CXX --mode=link clang++ -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -o generator.la -rpath https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=spdy/ws/172/install/libexec/trafficserver generator.lo -lcap -lpcre -lexpat -llzma -lz -lcrypt -lpthread -ldl libtool: link: /usr/bin/nm -B .libs/generator.o | sed
Build failed in Jenkins: ubuntu_14_10-master » gcc,ubuntu_14_10,release #199
See
https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=release/199/
--
[...truncated 2685 lines...]
make[3]: Leaving directory
'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=release/ws/199/build/BUILDS/plugins/experimental/header_normalize'
Making all in healthchecks
make[3]: Entering directory
'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=release/ws/199/build/BUILDS/plugins/experimental/healthchecks'
depbase=`echo healthchecks.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\
/bin/bash ../../../libtool --tag=CC --mode=compile ccache cc -DHAVE_CONFIG_H
-I. -I../../../../plugins/experimental/healthchecks -I../../../lib/ts
-I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts
-I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1
-D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1
-D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6
-I/usr/include/libxml2 -Wunused-parameter -g -pipe -Wall -O3
-feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -mcx16 -MT
healthchecks.lo -MD -MP -MF $depbase.Tpo -c -o healthchecks.lo
../../../../plugins/experimental/healthchecks/healthchecks.c \
mv -f $depbase.Tpo $depbase.Plo
libtool: compile: ccache cc -DHAVE_CONFIG_H -I.
-I../../../../plugins/experimental/healthchecks -I../../../lib/ts
-I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts
-I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1
-D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1
-D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6
-I/usr/include/libxml2 -Wunused-parameter -g -pipe -Wall -O3
-feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -mcx16 -MT
healthchecks.lo -MD -MP -MF .deps/healthchecks.Tpo -c
../../../../plugins/experimental/healthchecks/healthchecks.c -fPIC -DPIC -o
.libs/healthchecks.o
/bin/bash ../../../libtool --tag=CC --mode=link ccache cc -Wunused-parameter
-g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing
-Werror -mcx16 -module -shared -avoid-version -export-symbols-regex
'^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$'
-o healthchecks.la -rpath
https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=release/ws/199/install/libexec/trafficserver
healthchecks.lo -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2
libtool: link: /usr/bin/nm -B .libs/healthchecks.o | sed -n -e 's/^.*[
]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][
]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed
's/.* //' | sort | uniq .libs/healthchecks.exp
libtool: link: /bin/grep -E -e
^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$
.libs/healthchecks.exp .libs/healthchecks.expT
libtool: link: mv -f .libs/healthchecks.expT .libs/healthchecks.exp
libtool: link: echo { global: .libs/healthchecks.ver
libtool: link: cat .libs/healthchecks.exp | sed -e s/\(.*\)/\1;/
.libs/healthchecks.ver
libtool: link: echo local: *; }; .libs/healthchecks.ver
libtool: link: cc -shared -fPIC -DPIC .libs/healthchecks.o -lcap -lpcre
-lz -lcrypt -lpthread -ldl -lxml2 -O3 -mcx16 -Wl,-soname -Wl,healthchecks.so
-Wl,-version-script -Wl,.libs/healthchecks.ver -o .libs/healthchecks.so
libtool: link: ( cd .libs rm -f healthchecks.la ln -s
../healthchecks.la healthchecks.la )
make[3]: Leaving directory
'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=release/ws/199/build/BUILDS/plugins/experimental/healthchecks'
Making all in hipes
make[3]: Entering directory
'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=release/ws/199/build/BUILDS/plugins/experimental/hipes'
depbase=`echo hipes.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\
/bin/bash ../../../libtool --tag=CXX --mode=compile ccache c++
-DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/hipes -I../../../lib/ts
-I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts
-I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1
-D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1
-D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6
-I/usr/include/libxml2 -Wunused-parameter -std=c++11 -g -pipe -Wall -O3
-feliminate-unused-debug-symbols -fno-strict-aliasing -Werror
-Wno-invalid-offsetof -mcx16 -MT hipes.lo -MD -MP -MF $depbase.Tpo -c -o
hipes.lo ../../../../plugins/experimental/hipes/hipes.cc \
mv -f $depbase.Tpo $depbase.Plo
libtool: compile: ccache c++ -DHAVE_CONFIG_H -I.
-I../../../../plugins/experimental/hipes
Build failed in Jenkins: ubuntu_14_10-master » clang,ubuntu_14_10,debug #199
See https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=debug/199/ -- [...truncated 2681 lines...] make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=debug/ws/199/build/BUILDS/plugins/experimental/geoip_acl' Making all in header_normalize make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=debug/ws/199/build/BUILDS/plugins/experimental/header_normalize' depbase=`echo header_normalize.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Qunused-arguments -std=c++11 -ggdb3 -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF $depbase.Tpo -c -o header_normalize.lo ../../../../plugins/experimental/header_normalize/header_normalize.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Qunused-arguments -std=c++11 -ggdb3 -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF .deps/header_normalize.Tpo -c ../../../../plugins/experimental/header_normalize/header_normalize.cc -fPIC -DPIC -o .libs/header_normalize.o /bin/bash ../../../libtool --tag=CXX --mode=link clang++ -Qunused-arguments -std=c++11 -ggdb3 -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -o header_normalize.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=debug/ws/199/install/libexec/trafficserver header_normalize.lo -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/header_normalize.o | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/header_normalize.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/header_normalize.exp .libs/header_normalize.expT libtool: link: mv -f .libs/header_normalize.expT .libs/header_normalize.exp libtool: link: clang++ -fPIC -DPIC -shared -nostdlib /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtbeginS.o .libs/header_normalize.o -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../.. -L/usr/lib/llvm-3.5/bin/../lib -L/lib -L/usr/lib -lstdc++ -lm -lc -lgcc_s /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtendS.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crtn.o -mcx16 -Wl,-soname -Wl,header_normalize.so -Wl,-retain-symbols-file -Wl,.libs/header_normalize.exp -o .libs/header_normalize.so libtool: link: ( cd .libs rm -f header_normalize.la ln -s ../header_normalize.la header_normalize.la ) make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=debug/ws/199/build/BUILDS/plugins/experimental/header_normalize' Making all in healthchecks make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=debug/ws/199/build/BUILDS/plugins/experimental/healthchecks' depbase=`echo healthchecks.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CC --mode=compile clang -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/healthchecks -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1
Build failed in Jenkins: ubuntu_14_10-master » clang,ubuntu_14_10,release #199
See https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=release/199/ -- [...truncated 2679 lines...] make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=release/ws/199/build/BUILDS/plugins/experimental/geoip_acl' Making all in header_normalize make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=release/ws/199/build/BUILDS/plugins/experimental/header_normalize' depbase=`echo header_normalize.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF $depbase.Tpo -c -o header_normalize.lo ../../../../plugins/experimental/header_normalize/header_normalize.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF .deps/header_normalize.Tpo -c ../../../../plugins/experimental/header_normalize/header_normalize.cc -fPIC -DPIC -o .libs/header_normalize.o /bin/bash ../../../libtool --tag=CXX --mode=link clang++ -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -o header_normalize.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=release/ws/199/install/libexec/trafficserver header_normalize.lo -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/header_normalize.o | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/header_normalize.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/header_normalize.exp .libs/header_normalize.expT libtool: link: mv -f .libs/header_normalize.expT .libs/header_normalize.exp libtool: link: clang++ -fPIC -DPIC -shared -nostdlib /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtbeginS.o .libs/header_normalize.o -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../.. -L/usr/lib/llvm-3.5/bin/../lib -L/lib -L/usr/lib -lstdc++ -lm -lc -lgcc_s /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/crtendS.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crtn.o -O3 -mcx16 -Wl,-soname -Wl,header_normalize.so -Wl,-retain-symbols-file -Wl,.libs/header_normalize.exp -o .libs/header_normalize.so libtool: link: ( cd .libs rm -f header_normalize.la ln -s ../header_normalize.la header_normalize.la ) make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=release/ws/199/build/BUILDS/plugins/experimental/header_normalize' Making all in healthchecks make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=release/ws/199/build/BUILDS/plugins/experimental/healthchecks' depbase=`echo healthchecks.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CC --mode=compile clang -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/healthchecks -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT
[jira] [Commented] (TS-3385) should show scripting loading error when using lua plugin as remap plugin
[ https://issues.apache.org/jira/browse/TS-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315828#comment-14315828 ] ASF subversion and git services commented on TS-3385: - Commit 3f8fb92791572cce6b487dbfb815d1a675da01ab in trafficserver's branch refs/heads/master from [~kichan] [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=3f8fb92 ] TS-3385: fix build error should show scripting loading error when using lua plugin as remap plugin - Key: TS-3385 URL: https://issues.apache.org/jira/browse/TS-3385 Project: Traffic Server Issue Type: Improvement Components: Lua, Plugins Reporter: Kit Chan Assignee: Kit Chan Priority: Minor Fix For: 5.3.0 e.g. if we make a syntax mistake in our lua script the lua plugin will not output the loading error when it is being used as a remap plugin. It works fine and output the syntax mistake though when used as a global plugin in plugin.conf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Build failed in Jenkins: fedora_21-master » clang,fedora_21,release #172
See https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=release/172/ -- [...truncated 2663 lines...] libtool: link: /usr/bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/esi.exp .libs/esi.expT libtool: link: mv -f .libs/esi.expT .libs/esi.exp libtool: link: clang++ -fPIC -DPIC -shared -nostdlib /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64/crti.o /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/crtbeginS.o .libs/esi.o fetcher/.libs/HttpDataFetcherImpl.o .libs/serverIntercept.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -lcap -lpcre -lexpat -llzma -lz -lcrypt -lpthread -ldl -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2 -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64 -L/usr/bin/../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../.. -L/usr/bin/../lib -L/lib -L/usr/lib -lstdc++ -lm -lc -lgcc_s /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/crtendS.o /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64/crtn.o -O3 -mcx16 -Wl,-soname -Wl,esi.so -Wl,-retain-symbols-file -Wl,.libs/esi.exp -o .libs/esi.so libtool: link: /usr/bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/combo_handler.exp .libs/combo_handler.expT libtool: link: mv -f .libs/combo_handler.expT .libs/combo_handler.exp libtool: link: clang++ -fPIC -DPIC -shared -nostdlib /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64/crti.o /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/crtbeginS.o .libs/combo_handler.o fetcher/.libs/HttpDataFetcherImpl.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -lcap -lpcre -lexpat -llzma -lz -lcrypt -lpthread -ldl -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2 -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64 -L/usr/bin/../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../.. -L/usr/bin/../lib -L/lib -L/usr/lib -lstdc++ -lm -lc -lgcc_s /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/crtendS.o /usr/bin/../lib/gcc/x86_64-redhat-linux/4.9.2/../../../../lib64/crtn.o -O3 -mcx16 -Wl,-soname -Wl,combo_handler.so -Wl,-retain-symbols-file -Wl,.libs/combo_handler.exp -o .libs/combo_handler.so libtool: link: ( cd .libs rm -f esi.la ln -s ../esi.la esi.la ) libtool: link: ( cd .libs rm -f combo_handler.la ln -s ../combo_handler.la combo_handler.la ) make[3]: Leaving directory 'https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=release/ws/172/build/BUILDS/plugins/experimental/esi' Making all in generator make[3]: Entering directory 'https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=release/ws/172/build/BUILDS/plugins/experimental/generator' depbase=`echo generator.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/sh ../../../libtool --tag=CXX --mode=compile clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/generator -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT generator.lo -MD -MP -MF $depbase.Tpo -c -o generator.lo ../../../../plugins/experimental/generator/generator.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: clang++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/generator -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT generator.lo -MD -MP -MF .deps/generator.Tpo -c ../../../../plugins/experimental/generator/generator.cc -fPIC -DPIC -o .libs/generator.o /bin/sh ../../../libtool --tag=CXX --mode=link clang++ -Qunused-arguments -std=c++11 -g -O3 -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -o generator.la -rpath https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=release/ws/172/install/libexec/trafficserver generator.lo -lcap -lpcre -lexpat -llzma -lz -lcrypt -lpthread -ldl libtool: link: /usr/bin/nm -B
Jenkins build is back to normal : ubuntu_13_10-master » gcc,ubuntu_13_10,debug #721
See https://ci.trafficserver.apache.org/job/ubuntu_13_10-master/compiler=gcc,label=ubuntu_13_10,type=debug/721/
Jenkins build is back to normal : freebsd_10-master » clang,freebsd_10,debug #739
See https://ci.trafficserver.apache.org/job/freebsd_10-master/compiler=clang,label=freebsd_10,type=debug/739/
Jenkins build is back to normal : ubuntu_14_10-master » clang,ubuntu_14_10,debug #200
See https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=clang,label=ubuntu_14_10,type=debug/200/
Jenkins build is back to normal : ubuntu_14_04-master » gcc,ubuntu_14_04,debug #742
See https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=debug/742/
Build failed in Jenkins: ubuntu_14_04-master » gcc,ubuntu_14_04,release #741
See https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=release/741/ -- [...truncated 2669 lines...] Making all in geoip_acl make[3]: Entering directory `https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=release/ws/741/build/BUILDS/plugins/experimental/geoip_acl' make[3]: Nothing to be done for `all'. make[3]: Leaving directory `https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=release/ws/741/build/BUILDS/plugins/experimental/geoip_acl' Making all in header_normalize make[3]: Entering directory `https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=release/ws/741/build/BUILDS/plugins/experimental/header_normalize' depbase=`echo header_normalize.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CXX --mode=compile ccache c++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Wunused-parameter -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF $depbase.Tpo -c -o header_normalize.lo ../../../../plugins/experimental/header_normalize/header_normalize.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: ccache c++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/header_normalize -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.6 -I/usr/include/libxml2 -Wunused-parameter -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT header_normalize.lo -MD -MP -MF .deps/header_normalize.Tpo -c ../../../../plugins/experimental/header_normalize/header_normalize.cc -fPIC -DPIC -o .libs/header_normalize.o /bin/bash ../../../libtool --tag=CXX --mode=link ccache c++ -Wunused-parameter -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -o header_normalize.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=release/ws/741/install/libexec/trafficserver header_normalize.lo -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/header_normalize.o | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/header_normalize.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/header_normalize.exp .libs/header_normalize.expT libtool: link: mv -f .libs/header_normalize.expT .libs/header_normalize.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o .libs/header_normalize.o -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o -O3 -mcx16 -Wl,-soname -Wl,header_normalize.so -Wl,-retain-symbols-file -Wl,.libs/header_normalize.exp -o .libs/header_normalize.so libtool: link: ( cd .libs rm -f header_normalize.la ln -s ../header_normalize.la header_normalize.la ) make[3]: Leaving directory `https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=release/ws/741/build/BUILDS/plugins/experimental/header_normalize' Making all in healthchecks make[3]: Entering directory `https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=release/ws/741/build/BUILDS/plugins/experimental/healthchecks' depbase=`echo healthchecks.lo | sed
Build failed in Jenkins: ubuntu_13_10-master » gcc,ubuntu_13_10,debug #720
See https://ci.trafficserver.apache.org/job/ubuntu_13_10-master/compiler=gcc,label=ubuntu_13_10,type=debug/720/ -- [...truncated 2652 lines...] libtool: link: /usr/bin/nm -B .libs/combo_handler.o fetcher/.libs/HttpDataFetcherImpl.o ./.libs/libesicore.a | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/combo_handler.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/combo_handler.exp .libs/combo_handler.expT libtool: link: mv -f .libs/combo_handler.expT .libs/combo_handler.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o .libs/combo_handler.o fetcher/.libs/HttpDataFetcherImpl.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -L/usr/lib -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o -mcx16 -Wl,-soname -Wl,combo_handler.so -Wl,-retain-symbols-file -Wl,.libs/combo_handler.exp -o .libs/combo_handler.so libtool: link: ( cd .libs rm -f combo_handler.la ln -s ../combo_handler.la combo_handler.la ) /bin/bash ../../../libtool --tag=CXX --mode=link ccache c++ -Wno-deprecated -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -L/usr/lib -o esi.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_13_10-master/compiler=gcc,label=ubuntu_13_10,type=debug/ws/720/install/libexec/trafficserver esi.lo fetcher/HttpDataFetcherImpl.lo serverIntercept.lo libesicore.la -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/esi.o fetcher/.libs/HttpDataFetcherImpl.o .libs/serverIntercept.o ./.libs/libesicore.a | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/esi.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/esi.exp .libs/esi.expT libtool: link: mv -f .libs/esi.expT .libs/esi.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o .libs/esi.o fetcher/.libs/HttpDataFetcherImpl.o .libs/serverIntercept.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -L/usr/lib -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o -mcx16 -Wl,-soname -Wl,esi.so -Wl,-retain-symbols-file -Wl,.libs/esi.exp -o .libs/esi.so libtool: link: ( cd .libs rm -f esi.la ln -s ../esi.la esi.la ) make[3]: Leaving directory `https://ci.trafficserver.apache.org/job/ubuntu_13_10-master/compiler=gcc,label=ubuntu_13_10,type=debug/ws/720/build/BUILDS/plugins/experimental/esi' Making all in generator make[3]: Entering directory `https://ci.trafficserver.apache.org/job/ubuntu_13_10-master/compiler=gcc,label=ubuntu_13_10,type=debug/ws/720/build/BUILDS/plugins/experimental/generator' depbase=`echo generator.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CXX --mode=compile ccache c++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/generator -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -DDEBUG -D_DEBUG -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -I/usr/include -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.5 -I/usr/include/libxml2 -Wunused-parameter -std=c++11 -ggdb3 -pipe -Wall -Werror -Wno-invalid-offsetof -mcx16 -MT generator.lo -MD -MP -MF $depbase.Tpo -c -o generator.lo
Build failed in Jenkins: ubuntu_13_10-master » gcc,ubuntu_13_10,release #720
See https://ci.trafficserver.apache.org/job/ubuntu_13_10-master/compiler=gcc,label=ubuntu_13_10,type=release/720/ -- [...truncated 2657 lines...] libtool: link: /usr/bin/nm -B .libs/esi.o fetcher/.libs/HttpDataFetcherImpl.o .libs/serverIntercept.o ./.libs/libesicore.a | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/esi.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/esi.exp .libs/esi.expT libtool: link: mv -f .libs/esi.expT .libs/esi.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o .libs/esi.o fetcher/.libs/HttpDataFetcherImpl.o .libs/serverIntercept.o -Wl,--whole-archive ./.libs/libesicore.a -Wl,--no-whole-archive -L/usr/lib -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o -O3 -mcx16 -Wl,-soname -Wl,esi.so -Wl,-retain-symbols-file -Wl,.libs/esi.exp -o .libs/esi.so libtool: link: ( cd .libs rm -f esi.la ln -s ../esi.la esi.la ) make[3]: Leaving directory `https://ci.trafficserver.apache.org/job/ubuntu_13_10-master/compiler=gcc,label=ubuntu_13_10,type=release/ws/720/build/BUILDS/plugins/experimental/esi' Making all in generator make[3]: Entering directory `https://ci.trafficserver.apache.org/job/ubuntu_13_10-master/compiler=gcc,label=ubuntu_13_10,type=release/ws/720/build/BUILDS/plugins/experimental/generator' depbase=`echo generator.lo | sed 's|[^/]*$|.deps/|;s|\.lo$||'`;\ /bin/bash ../../../libtool --tag=CXX --mode=compile ccache c++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/generator -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -I/usr/include -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.5 -I/usr/include/libxml2 -Wunused-parameter -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT generator.lo -MD -MP -MF $depbase.Tpo -c -o generator.lo ../../../../plugins/experimental/generator/generator.cc \ mv -f $depbase.Tpo $depbase.Plo libtool: compile: ccache c++ -DHAVE_CONFIG_H -I. -I../../../../plugins/experimental/generator -I../../../lib/ts -I../../../proxy/api -I../../../../proxy/api -I../../../lib/ts -I../../../../lib/ts -I../../../../lib -Dlinux -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -I/usr/include -DOPENSSL_NO_SSL_INTERN -I/usr/include/tcl8.5 -I/usr/include/libxml2 -Wunused-parameter -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -MT generator.lo -MD -MP -MF .deps/generator.Tpo -c ../../../../plugins/experimental/generator/generator.cc -fPIC -DPIC -o .libs/generator.o /bin/bash ../../../libtool --tag=CXX --mode=link ccache c++ -Wunused-parameter -std=c++11 -g -pipe -Wall -O3 -feliminate-unused-debug-symbols -fno-strict-aliasing -Werror -Wno-invalid-offsetof -mcx16 -module -shared -avoid-version -export-symbols-regex '^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$' -L/usr/lib -o generator.la -rpath https://ci.trafficserver.apache.org/job/ubuntu_13_10-master/compiler=gcc,label=ubuntu_13_10,type=release/ws/720/install/libexec/trafficserver generator.lo -lcap -lpcre -lz -lcrypt -lpthread -ldl -lxml2 libtool: link: /usr/bin/nm -B .libs/generator.o | sed -n -e 's/^.*[ ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[ ][ ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq .libs/generator.exp libtool: link: /bin/grep -E -e ^(TSRemapInit|TSRemapDone|TSRemapDoRemap|TSRemapNewInstance|TSRemapDeleteInstance|TSRemapOSResponse|TSPluginInit)$ .libs/generator.exp .libs/generator.expT libtool: link: mv -f .libs/generator.expT .libs/generator.exp libtool: link: c++ -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o .libs/generator.o -L/usr/lib -lcap -lpcre -lz -lcrypt
Jenkins build is back to normal : ubuntu_12_04-master » gcc,ubuntu_12_04,debug #720
See https://ci.trafficserver.apache.org/job/ubuntu_12_04-master/compiler=gcc,label=ubuntu_12_04,type=debug/720/
Jenkins build is back to normal : fedora_21-master » clang,fedora_21,debug #173
See https://ci.trafficserver.apache.org/job/fedora_21-master/compiler=clang,label=fedora_21,type=debug/173/
Jenkins build is back to normal : freebsd_10-master » clang,freebsd_10,release #739
See https://ci.trafficserver.apache.org/job/freebsd_10-master/compiler=clang,label=freebsd_10,type=release/739/
Jenkins build is back to normal : ubuntu_14_04-master » gcc,ubuntu_14_04,release #742
See https://ci.trafficserver.apache.org/job/ubuntu_14_04-master/compiler=gcc,label=ubuntu_14_04,type=release/742/
Jenkins build is back to normal : ubuntu_14_10-master » gcc,ubuntu_14_10,debug #200
See https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=debug/200/
Jenkins build is back to normal : ubuntu_13_10-master » gcc,ubuntu_13_10,release #721
See https://ci.trafficserver.apache.org/job/ubuntu_13_10-master/compiler=gcc,label=ubuntu_13_10,type=release/721/
Jenkins build is back to normal : ubuntu_14_10-master » gcc,ubuntu_14_10,release #200
See https://ci.trafficserver.apache.org/job/ubuntu_14_10-master/compiler=gcc,label=ubuntu_14_10,type=release/200/
[jira] [Created] (TS-3386) Heartbeat failed with high load, trafficserver restarted
Luca Bruno created TS-3386:
--
Summary: Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I have
a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to 1000
concurrent requests, then the ATS worker crashed, restarted and thus lost the
whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server Version:
Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on Feb 10 2015
at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status signal
[32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10 2015
at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server Version:
Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on Feb 10 2015
at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server Version:
Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on Feb 10 2015
at 13:04:42)
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:07:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:07:02 test-cache traffic_cop[32984]: server heartbeat
[jira] [Updated] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luca Bruno updated TS-3386:
---
Description:
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I have
a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to 1000
concurrent requests, then the ATS worker has been restarted and thus lost the
whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server Version:
Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on Feb 10 2015
at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status signal
[32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10 2015
at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server Version:
Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on Feb 10 2015
at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server Version:
Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on Feb 10 2015
at 13:04:42)
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:07:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:07:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:07:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:07:02 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316143#comment-14316143
]
Luca Bruno edited comment on TS-3386 at 2/11/15 12:34 PM:
--
So it's like the server saturated the connections, and it replied 502 even to
the heartbeat server, without analyzing whether the request was a heartbeat
request. Am I right? In this case, I'd either like to disable heartbeat
completely or start a second tcp server just for heartbeat, instead of mixing
heartbeat + client connections within the same server.
was (Author: lethalman):
So it's like the server saturated the connections, and it replied 502 even to
the heartbeat server, without analyzing the response. Am I right? In this case,
I'd either like to disable heartbeat completely or start a second tcp server
just for heartbeat, instead of mixing heartbeat + client connections within the
same server.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]:
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316134#comment-14316134
]
Luca Bruno commented on TS-3386:
Additionally, proxy.config.cop.core_signal is 0, which should not sent a signal
to stop the server. Yet it sends signal 9 it seems, unless I'm misunderstanding
the option description.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: traffic_server
Version: Apache Traffic Server -
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316143#comment-14316143
]
Luca Bruno edited comment on TS-3386 at 2/11/15 12:31 PM:
--
So it's like the server saturated the connections, and it replied 502 even to
the heartbeat server, without analyzing the response. Am I right? In this case,
I'd either like to disable heartbeat completely or start a second tcp server
just for heartbeat, instead of mixing heartbeat + client connections within the
same server.
was (Author: lethalman):
So it's like the server saturated the connections, and it replied 502 even to
the heartbeat server, without analyzing the response. Am I right? In this case,
I'd either like to disable heartbeat completely or start a second tcp server
just for heartbeat, isntead of mixing heartbeat + client connections within the
same server.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11
[jira] [Commented] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316143#comment-14316143
]
Luca Bruno commented on TS-3386:
So it's like the server saturated the connections, and it replied 502 even to
the heartbeat server, without analyzing the response. Am I right? In this case,
I'd either like to disable heartbeat completely or start a second tcp server
just for heartbeat, isntead of mixing heartbeat + client connections within the
same server.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:42 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:42 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:42 test-cache traffic_manager[59057]: {0x7f2c94ded720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:44 test-cache traffic_server[59077]: NOTE: ---
[jira] [Comment Edited] (TS-3386) Heartbeat failed with high load, trafficserver restarted
[
https://issues.apache.org/jira/browse/TS-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316143#comment-14316143
]
Luca Bruno edited comment on TS-3386 at 2/11/15 1:34 PM:
-
So it's like the server saturated the connections, and it replied 502 even to
the heartbeat server, without analyzing whether the request was a heartbeat
request. Am I right? In this case, I'd either like to disable heartbeat
completely or start a second tcp server just for heartbeat, instead of mixing
heartbeat + client connections within the same server, or keep the heartbeat
for monitoring without killing the server.
was (Author: lethalman):
So it's like the server saturated the connections, and it replied 502 even to
the heartbeat server, without analyzing whether the request was a heartbeat
request. Am I right? In this case, I'd either like to disable heartbeat
completely or start a second tcp server just for heartbeat, instead of mixing
heartbeat + client connections within the same server.
Heartbeat failed with high load, trafficserver restarted
Key: TS-3386
URL: https://issues.apache.org/jira/browse/TS-3386
Project: Traffic Server
Issue Type: Bug
Components: Performance
Reporter: Luca Bruno
I've been evaluating ATS for some days. I'm using it with mostly default
settings, except I've lowered the number of connections to the backend, I
have a raw storage of 500gb, and disabled ram cache.
Working fine, then I wanted to stress it more. I've increased the test to
1000 concurrent requests, then the ATS worker has been restarted and thus
lost the whole cache.
/var/log/syslog:
{noformat}
Feb 11 10:05:52 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:05:52 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:02 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:02 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:02 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 9:
Killed
Feb 11 10:06:02 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[Alarms::signalAlarm] Server Process was reset
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:04 test-cache traffic_server[59047]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:12 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:12 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:22 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:22 test-cache traffic_cop[32984]: server heartbeat failed [2]
Feb 11 10:06:22 test-cache traffic_cop[32984]: killing server
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
Feb 11 10:06:22 test-cache traffic_manager[32985]: {0x7f975c537720} ERROR:
(last system error 32: Broken pipe)
Feb 11 10:06:22 test-cache traffic_cop[32984]: cop received child status
signal [32985 256]
Feb 11 10:06:22 test-cache traffic_cop[32984]: traffic_manager not running,
making sure traffic_server is dead
Feb 11 10:06:22 test-cache traffic_cop[32984]: spawning traffic_manager
Feb 11 10:06:22 test-cache traffic_cop[32984]: binpath is bin
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: --- Manager Starting
---
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE: Manager Version:
Apache Traffic Server - traffic_manager - 5.2.0 - (build # 11013 on Feb 10
2015 at 13:05:19)
Feb 11 10:06:22 test-cache traffic_manager[59057]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: --- traffic_server
Starting ---
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE: traffic_server
Version: Apache Traffic Server - traffic_server - 5.2.0 - (build # 11013 on
Feb 10 2015 at 13:04:42)
Feb 11 10:06:24 test-cache traffic_server[59065]: NOTE:
RLIMIT_NOFILE(7):cur(736236),max(736236)
Feb 11 10:06:32 test-cache traffic_cop[32984]: (http test) received non-200
status(502)
Feb 11 10:06:32 test-cache traffic_cop[32984]: server heartbeat failed [1]
Feb 11 10:06:42 test-cache traffic_cop[32984]:
