[
https://issues.apache.org/jira/browse/TS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15465425#comment-15465425
]
David Brodin commented on TS-4816:
----------------------------------
Ok, for some reason I thought libunwind was a linux-thing, TIL I guess :)
Should it matter if I compiled ATS from ports and not from source downloaded
from apache.org?
I installed libunwind and rebuilt ATS from ports, but got the below when
running {{traffic_ctl server backtrace}}:
{noformat}traffic_ctl: server backtrace failed: [12] Operation not supported on
this platform.{noformat}
I'll reproduce while gdb is attached to the process and remember to do the rest
of the steps.
> ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
> ---------------------------------------------------------------
>
> Key: TS-4816
> URL: https://issues.apache.org/jira/browse/TS-4816
> Project: Traffic Server
> Issue Type: Bug
> Components: Manager
> Reporter: David Brodin
> Labels: crash
> Fix For: 7.0.0
>
> Attachments: ats_gdb-160905.txt, crash-2016-09-05-075309.log
>
>
> Hi,
> We just upgraded to ATS 6.2.0 via FreeBSD ports:
> {noformat}
> [root@<machine> ~]# uname -a
> FreeBSD <machine> 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
> 18:38:15 UTC 2016
> [email protected]:/usr/obj/usr/src/sys/GENERIC amd64
> [root@<machine> ~]# pkg info | grep traff
> trafficserver-6.2.0 Fast, scalable and extensible HTTP proxy server
> {noformat}
> We are experiencing crashes, usually during the day, hardly any during "low"
> loads, but the mest affecting crashes occur in the early mornings. Along with
> this we can see a memory leak aswell
> We are using ATS as an enterprise proxy to the Internet, and as we have a
> very good Internet-connection we have also disabled caching.
> I'm not sure how I would attach files so here goes :)
> manager.log
> {noformat}
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:48:28.017] Manager {0x804006400} ERROR: <MgmtUtils.cc:289
> (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:48:38.305] {0x804006400} STATUS: opened
> /var/log/trafficserver/manager.log
> [Sep 2 11:48:38.305] {0x804006400} NOTE: <DiagsConfig.cc:141
> (reconfigure_diags)> updated diags config
> [Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom]
> Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
> [Sep 2 11:48:38.312] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE:
> [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
> [Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup
> complete
> [Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
> [Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 2 11:51:32.574] Manager {0x804006400} ERROR: <MgmtUtils.cc:289
> (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 2 11:51:33.689] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Sep 3 04:14:35.380] Manager {0x804006400} ERROR: <MgmtUtils.cc:289
> (mgmt_elog)> (last system error 32: Broken pipe)
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> [Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm]
> Server Process was reset
> [Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy]
> Launching ts process
> [Sep 3 04:14:36.828] Manager {0x804006400} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
> [Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm]
> Server Process born
> {noformat}
> traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of
> the stacktrace out:
> {noformat}
> traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
> : Terminated
> traffic_server: Terminatedtraffic_servertraffic_servertraffic_server:
> Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
> 0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
> getpeereid -> 0 (54, Connection reset by peer)[TrafficManager] ==> Cleaning
> up and reissuing signal #15
> traffic_server: Terminated
> traffic_server: Terminated
> traffic_server: using root directory '/usr/local'
> [TrafficManager] ==> signal #15
> traffic_server: Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
> 0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
> traffic_server: Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
> 0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
> traffic_server: Terminated
> traffic_server: Terminated
> traffic_server: Terminated
> traffic_server: Terminated
> traffic_server: Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
> 0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
> traffic_server: Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
> 0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
> traffic_server: Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
> 0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
> traffic_server: Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
> 0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
> traffic_server: Segmentation fault
> traffic_server - STACK TRACE:
> 0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at
> /usr/local/bin/traffic_server
> 0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
> 0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
> {noformat}
> /var/log/messages
> {noformat}
> Sep 2 06:58:03 <machine> traffic_manager[5604]: {0x804006400} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> Sep 2 06:58:03 <machine> kernel: pid 6680 (traffic_server), uid 80: exited
> on signal 11
> Sep 2 06:58:03 <machine> traffic_manager[5604]: {0x804006400} ERROR:
> <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe)
> Sep 2 06:58:04 <machine> traffic_cop[5603]: cannot find traffic_server [1]
> Sep 2 06:58:04 <machine> traffic_manager[5604]: {0x804006400} ERROR:
> [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig
> 11: Segmentation fault
> Sep 2 06:58:04 <machine> traffic_manager[5604]: {0x804006400} ERROR:
> [Alarms::signalAlarm] Server Process was reset
> Sep 2 06:58:08 <machine> traffic_server[9951]: NOTE: --- traffic_server
> Starting ---
> Sep 2 06:58:08 <machine> traffic_server[9951]: NOTE: traffic_server Version:
> Apache Traffic Server - traffic_server - 6.2.0 - (build # 083112 on Aug 31
> 2016 at 12:51:58)
> Sep 2 06:58:08 <machine> traffic_server[9951]: NOTE:
> RLIMIT_NOFILE(8):cur(190297),max(190297)
> {noformat}
> Every other second /var/log/messages is also getting 1-10 lines of this:
> {noformat}
> Sep 3 17:48:50 <machine> traffic_server[14338]: {0x804008000} ERROR:
> <HttpSM.cc:1159 (state_raw_http_server_open)>
> [HttpSM::state_raw_http_server_open] event: EVENT_INTERVAL state: 0
> server_entry: 0x0
> {noformat}
> And a "ps aux" showing mem usage:
> {noformat}
> [root@<machine> /usr/local/etc/trafficserver]# ps axu | grep "USER\|traff"
> USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
> www 14338 7.9 21.3 1910932 1778200 - S 4:14AM 27:51.59
> /usr/local/bin/traffic_server -M --bind_stdout
> /var/log/trafficserver/traffic.out --bind_stderr /var/log/traff
> root 5602 0.0 0.0 14492 2004 - Is Thu10AM 0:00.00 daemon:
> /usr/local/bin/traffic_cop[5603] (daemon)
> root 5603 0.0 0.1 64360 7516 - Ss Thu10AM 0:07.96
> /usr/local/bin/traffic_cop
> www 10897 0.0 0.2 87544 13492 - S Fri11AM 0:34.78
> /usr/local/bin/traffic_manager --bind_stdout
> /var/log/trafficserver/traffic.out --bind_stderr /var/log/traffic
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
