[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474460#comment-15474460
]
David Brodin commented on TS-4816:
--
True, I did notice that traffic_server and manager changed process-ids. I guess
traffic_cop was looking out for them.
I have traffic_server running through gdb now with args etc. We'll see tomorrow
morning again :)
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, ats_manager-gdb-160906.txt,
> crash-2016-09-05-075309.log, est_socks-ats_6.2.0.png
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
> : Terminated
> traffic_server: Terminatedtraffic_servertraffic_servertraffic_server:
> Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474389#comment-15474389
]
Leif Hedstrom commented on TS-4816:
---
traffic_cop might be killing the process when it's halted inside gdb?
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, ats_manager-gdb-160906.txt,
> crash-2016-09-05-075309.log, est_socks-ats_6.2.0.png
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
> : Terminated
> traffic_server: Terminatedtraffic_servertraffic_servertraffic_server:
> Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 at /lib/libthr.so.3
> 0x80273522c at /lib/libthr.so.3
> getpeereid -> 0 (
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474114#comment-15474114
]
David Brodin commented on TS-4816:
--
I understand, but I installed libunwind and rebuilt ATS with no, to me,
difference. Not that there should have been as James mentioned earlier.
There are two gdb's attached, one of traffic_server and one of traffic_manager,
the manager one has the {{bt}} and {{info local}} which I just realised should
be {{info locals}}. The managers {{bt}} is the same as it was for
traffic_server.
I’ll give running only traffic_server through gdb a try, I thought that
traffic_server needed traffic_manager/cop to run.
I might be misunderstanding, but if the traffic_server is killed/crashes/sig11
whatever, wouldn’t a gdb be the same as I can’t keep the process from “going
away”?
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, ats_manager-gdb-160906.txt,
> crash-2016-09-05-075309.log, est_socks-ats_6.2.0.png
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474085#comment-15474085
]
Leif Hedstrom commented on TS-4816:
---
Without getting a core / stack trace, there's no hope of getting closer to
understanding what is wrong I think :-/.
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, ats_manager-gdb-160906.txt,
> crash-2016-09-05-075309.log, est_socks-ats_6.2.0.png
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
> : Terminated
> traffic_server: Terminatedtraffic_servertraffic_servertraffic_server:
> Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 at /lib/libthr.so.3
> 0x8027
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474076#comment-15474076
]
David Brodin commented on TS-4816:
--
Well, ATS lasted longer this time but still crashed the same. No coredump or
anything else that i haven't already added to this bugreport.
I'll be going back to 5.1.1 tomorrow unless there is anything I should try by
then :/
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, ats_manager-gdb-160906.txt,
> crash-2016-09-05-075309.log, est_socks-ats_6.2.0.png
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
> : Terminated
> traffic_server: Terminatedtraffic_servertraffic_servertraffic_server:
> Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logg
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15471866#comment-15471866
]
David Brodin commented on TS-4816:
--
Yay, it's not only me! :D
Joke aside, I want to make clear that my two servers (5.1.1 & 6.2.0) are not
clustered in any way.
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, ats_manager-gdb-160906.txt,
> crash-2016-09-05-075309.log, est_socks-ats_6.2.0.png
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
> : Terminated
> traffic_server: Terminatedtraffic_servertraffic_servertraffic_server:
> Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 at /lib/libthr.so.3
>
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15471799#comment-15471799
]
Hung-Yi Chen commented on TS-4816:
--
I've got the same issue on FreeBSD 10.2 & 10.3.
Here's traffic.out
{noformat}
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4b1e19 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
/usr/local/bin/traffic_server
0x802953997 at /lib/libthr.so.3
0x8029531a8 at /lib/libthr.so.3
{noformat}
manager.log:
{noformat}
[Sep 8 03:46:22.180] Manager {0x804006400} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
[Sep 8 03:46:22.181] Manager {0x804006400} ERROR: (last system error 32: Broken pipe)
[Sep 8 03:46:22.702] Manager {0x804006400} ERROR:
[LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11:
Segmentation fault
[Sep 8 03:46:22.703] Manager {0x804006400} ERROR: [Alarms::signalAlarm] Server
Process was reset
[Sep 8 03:46:23.771] Manager {0x804006400} WARNING: multicast send timeout exceeded. 21368 seconds since last
send.
[Sep 8 03:46:23.786] Manager {0x804006400} NOTE: [LocalManager::startProxy]
Launching ts process
[Sep 8 03:46:23.921] Manager {0x804006400} NOTE:
[LocalManager::pollMgmtProcessServer] New process connecting fd '16'
[Sep 8 03:46:23.922] Manager {0x804006400} NOTE: [Alarms::signalAlarm] Server
Process born
{noformat}
And I use truss to try to catch it.
{noformat}
clock_gettime(0,{1473277582.156076393 }) = 0 (0x0)
_umtx_op(0x800a492d8,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffde0dfe00) ERR#60
'Operation timed out'
_umtx_op(0x800a49278,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffde4e3e00) ERR#60
'Operation timed out'
kevent(83,{0x109d,EVFILT_WRITE,EV_ADD|EV_CLEAR,0,0x0,0x813fa41b0},1,0x0,0,0x0)
= 0 (0x0)
kevent(66,{0xa66,EVFILT_WRITE,EV_ADD|EV_CLEAR,0,0x0,0x813d00410},1,0x0,0,0x0) =
0 (0x0)
clock_gettime(0,{1473277582.158822671 }) = 0 (0x0)
_umtx_op(0x8050ff880,UMTX_OP_MUTEX_WAIT,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x800a49308,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffddedde00) ERR#60
'Operation timed out'
clock_gettime(0,{1473277582.161559171 }) = 0 (0x0)
sigprocmask(SIG_SETMASK,SIGSEGV,0x0) = 0 (0x0)
write(1305,"HTTP/1.1 404 Not Found\r\nDate: "...,106) = 106 (0x6a)
_umtx_op(0x8050ff880,UMTX_OP_MUTEX_WAKE2,0x0,0x0,0x0) = 0 (0x0)
kevent(85,{0xe32,EVFILT_WRITE,EV_ADD|EV_CLEAR,0,0x0,0x813f10270},1,0x0,0,0x0) =
0 (0x0)
writev(0x10ad,0x7fffdf2f14d0,0x2,0x1a1,0x813fa1040,0x7fffdf2f1674) = 417 (0x1a1)
kevent(66,{0xdf0,EVFILT_WRITE,EV_ADD|EV_CLEAR,0,0x0,0x813eec030},1,0x0,0,0x0) =
0 (0x0)
clock_gettime(0,{1473277582.166437260 }) = 0 (0x0)
clock_gettime(0,{1473277582.164370953 }) = 0 (0x0)
clock_gettime(0,{1473277582.167189089 }) = 0 (0x0)
_umtx_op(0x8050ff880,UMTX_OP_MUTEX_WAKE2,0x0,0x0,0x0) = 0 (0x0)
_umtx_op(0x8050ff880,UMTX_OP_MUTEX_WAIT,0x0,0x0,0x0) = 0 (0x0)
clock_gettime(0,{1473277582.168218088 }) = 0 (0x0)
clock_gettime(13,{1473277582.0 })= 0 (0x0)
_umtx_op(0x800a492d8,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffde0dfe00) ERR#60
'Operation timed out'
kevent(66,{0x61f,EVFILT_WRITE,EV_ADD|EV_CLEAR,0,0x0,0x80d826490},1,0x0,0,0x0) =
0 (0x0)
writev(0xe62,0x7fffde9e84d0,0x2,0x18d,0x813f077e0,0x7fffde9e8674) ERR#32
'Broken pipe'
clock_gettime(0,{1473277582.172662091 }) = 0 (0x0)
read(3358,0x8a6b0305b,4005) ERR#54 'Connection reset by
peer'
close(1757) = 0 (0x0)
close(986) = 0 (0x0)
clock_gettime(0,{1473277582.174860873 }) = 0 (0x0)
close(3358) = 0 (0x0)
_umtx_op(0x800a49278,UMTX_OP_WAIT_UINT_PRIVATE,0x0,0x18,0x7fffde4e3e00) ERR#60
'Operation timed out'
read(4264,0x8a1a5b05b,4005) ERR#54 'Connection reset by
peer'
write(2360,"HTTP/1.1 404 Not Found\r\nDate: "...,106) = 106 (0x6a)
sigreturn(0x7fffddcd92d0,0x7fffddcd92d0,0x300,0x0,0xfbc0,0x8080808080808080)
= 34477128384 (0x806ff3ac0)
SIGNAL 11 (SIGSEGV)
process exit, rval = 0
{noformat}
It seems some kind of issue of threading?
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, ats_manager-gdb-160906.txt,
> crash-2016-09-05-075309.log, est_socks-ats_6.2.0.png
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-b
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15465979#comment-15465979
]
James Peach commented on TS-4816:
-
Ah sorry for misleading you [~durd]. The backtrace stuff grovels around in
{{/proc}} so it is fairly Linux-specific.
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, crash-2016-09-05-075309.log
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
> : Terminated
> traffic_server: Terminatedtraffic_servertraffic_servertraffic_server:
> Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 at /lib/libthr.so.3
> 0x80273522c at /lib/libthr.so.3
> getpeereid -> 0 (54, Connectio
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15465425#comment-15465425
]
David Brodin commented on TS-4816:
--
Ok, for some reason I thought libunwind was a linux-thing, TIL I guess :)
Should it matter if I compiled ATS from ports and not from source downloaded
from apache.org?
I installed libunwind and rebuilt ATS from ports, but got the below when
running {{traffic_ctl server backtrace}}:
{noformat}traffic_ctl: server backtrace failed: [12] Operation not supported on
this platform.{noformat}
I'll reproduce while gdb is attached to the process and remember to do the rest
of the steps.
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, crash-2016-09-05-075309.log
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> C
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15465251#comment-15465251
]
James Peach commented on TS-4816:
-
If you install the libunwind port and rebuild Traffic Server, then it should
automatically be able to generate a backtrace in the crash log. You can test
whether that support works with the {{traffic_ctl server backtrace}} command.
If you are able to reproduce under {{gdb}} again, when the program stops with
SEGV, type {{bt\n}} and then {{info locals\n}} to show additional information
about the crashing context.
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, crash-2016-09-05-075309.log
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
> : Terminated
> traffic_server:
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461708#comment-15461708
]
David Brodin commented on TS-4816:
--
Found the below in /var/log/messages when restarting ATS:
{noformat}
Sep 3 23:04:50 traffic_server[14338]: {0x804006800} FATAL:
[ProcessManager::processSignalQueue] Error writing message!
Sep 3 23:04:50 traffic_server[14338]: {0x804006800} FATAL:
(last system error 32: Broken pipe)
Sep 3 23:04:50 traffic_cop[18119]: --- Cop Starting [Version: Apache
Traffic Server - traffic_cop - 6.2.0 - (build # 083112 on Aug 31 2016 at
12:52:10)] ---
Sep 3 23:04:50 traffic_cop[18119]: traffic_manager not running,
making sure traffic_server is dead
Sep 3 23:04:50 traffic_cop[18119]: spawning traffic_manager
Sep 3 23:04:50 traffic_manager[18120]: NOTE: --- Manager Starting ---
Sep 3 23:04:50 traffic_manager[18120]: NOTE: Manager Version: Apache
Traffic Server - traffic_manager - 6.2.0 - (build # 083112 on Aug 31 2016 at
12:52:18)
Sep 3 23:04:50 traffic_manager[18120]: NOTE:
RLIMIT_NOFILE(8):cur(211442),max(211442)
Sep 3 23:04:54 traffic_server[18121]: NOTE: --- traffic_server
Starting ---
Sep 3 23:04:54 traffic_server[18121]: NOTE: traffic_server Version:
Apache Traffic Server - traffic_server - 6.2.0 - (build # 083112 on Aug 31 2016
at 12:51:58)
Sep 3 23:04:54 traffic_server[18121]: NOTE:
RLIMIT_NOFILE(8):cur(190297),max(190297)
Sep 3 23:04:54 traffic_server[18121]: {0x804007800} ERROR:
[HttpSM::state_raw_http_server_open] event: EVENT_INTERVAL state: 0
server_entry: 0x0
* lots of above *
Sep 3 23:07:00 traffic_cop[18119]: (http test) received malformed
response
Sep 3 23:07:00 traffic_cop[18119]: server heartbeat failed [1]
Sep 3 23:07:03 traffic_server[18121]: {0x804007400} ERROR:
[HttpSM::state_raw_http_server_open] event: EVENT_INTERVAL state: 0
server_entry: 0x0
Sep 3 23:07:10 traffic_cop[18119]: server heartbeat succeeded
Sep 3 23:07:15 traffic_server[18121]: {0x804007c00} ERROR:
[HttpSM::state_raw_http_server_open] event: EVENT_INTERVAL state: 0
server_entry: 0x0
{noformat}
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
>
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461702#comment-15461702
]
David Brodin commented on TS-4816:
--
Nope, no core file or anything except for the logs provided.
Ive tried to find out how I enabled coredumping, but just now found the wiki.
Ive enabled dumping and will attach the dump when that happens.
Might be earliest monday as this seems to be occuring when ATS is under load.
Sorry for spending your time, I thought the logs provided would show something
at least.
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
> : Terminated
> traffic_server: Terminatedtraffic_servertraffic_servertraffic_server:
> Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409
[jira] [Commented] (TS-4816) ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461689#comment-15461689
]
Leif Hedstrom commented on TS-4816:
---
Thanks for the report. Unfortunately, i can't seem to see anything specific to
the crash in any of the information. Maybe I'm missing it. Did you get a core
file? Did you get a crashlog? Any sort of backtrace?
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
>Reporter: David Brodin
> Fix For: 7.0.0
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@ ~]# uname -a
> FreeBSD 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
> [root@ ~]# pkg info | grep traff
> trafficserver-6.2.0Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
> : Terminated
> traffic_server: Terminatedtraffic_servertraffic_servertraffic_server:
> Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 at /lib/libthr.so.3
> 0x80273522c at /lib/libthr.so.3
> getpeereid -> 0 (54, Connection re
