I've been chasing an issue for a while now. This is 8.1.1 running on Ubuntu Focal. I'm seeing this same crash (Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE`) across different nodes that are handling unrelated traffic, but also ones that don't mention HTTP_SM_MAGIC_ALIVE.
Background: ATS is doing reverse proxy caching in front of web hosting backends, so a completely random assortment of backend domains and a random assortment of file types. Customer domains resolve to a group of 3-node clusters. I'm seeing the same crash across different clusters, so it wouldn't be related to the same backend origin domain(s). Backends are derived via remap+tslua, doing DNS lookups. These boxes constantly run at 200-400 reqs/seq all day, but I'm only seeing this crash anywhere from "once every couple days" to a couple of times a day on a given node (so the other millions of requests per day are processed just fine). ATS sits behind a front-end nginx instance (so nginx->ats is all localhost; origins are remote but on the same LAN). It sounds similar to https://github.com/apache/trafficserver/issues/4921 but that ticket also says that it was fixed for the OP in 8.1, so that could be a red herring. Any patches/settings I can use to mitigate this? Would turning off keepalives help as a bandaid? (Outbound keepalives to the origins would hurt a bit; inbound would probably be fine due to nginx->ats being on localhost). I'm also fine upgrading to 9.x but I ran into the issue where the connection count grows without bound, despite the connections not really being open (https://github.com/apache/trafficserver/issues/7471) I don't have any core files handy and these boxes have anywhere from 64gb-224gb ATS RAM cache, so I imagine a core would be horrifyingly large. The error is always similar to this (most of these are from the same box but that's just out of convenience; I'm seeing the same variety of segfaults across all of my boxes): May 24 07:05:14 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 24 07:05:14 cache0302 traffic_server[30820]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 24 07:05:14 cache0302 traffic_manager[21310]: [May 24 07:05:14.660] traffic_crashlo NOTE: crashlog started, target=30820, debug=false syslog=true, uid=114 euid=0 May 24 07:05:14 cache0302 traffic_crashlog[30822]: traffic_crashlo NOTE: crashlog started, target=30820, debug=false syslog=true, uid=114 euid=0 May 24 07:05:14 cache0302 traffic_crashlog[30822]: traffic_crashlo NOTE: logging to 0x14e10a33ad20 May 24 07:05:14 cache0302 traffic_manager[21310]: [May 24 07:05:14.660] traffic_crashlo NOTE: logging to 0x14e10a33ad20 May 24 07:05:17 cache0302 traffic_manager[21310]: [May 24 07:05:17.094] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-24-070514.log May 24 07:05:17 cache0302 traffic_crashlog[30822]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-24-070514.log May 24 07:05:17 cache0302 traffic_manager[21310]: traffic_server: received signal 6 (Aborted) May 24 07:05:17 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x15ff15ac4ef0] May 24 07:05:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x675351928980] May 24 07:05:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x675350a20fb7] May 24 07:05:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x675350a22921] May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x6753535bdb08] May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x6753535ba7d5] May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x15ff15b3cd0b] May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36cd32)[0x15ff15d70d32] May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x15ff15d72329] May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x15ff15d6324c] May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x15ff15d64d72] May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x15ff15db0f92] May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x15ff15db2233] May 24 07:05:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x15ff15db091a] May 24 07:05:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x67535191d6db] May 24 07:05:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x675350b0371f] Sometimes I get this one: May 22 21:30:01 cache0102 traffic_manager[6292]: Fatal: HttpSM.cc:5854: failed assertion `server_session == nullptr` May 22 21:30:01 cache0102 traffic_server[16246]: Fatal: HttpSM.cc:5854: failed assertion `server_session == nullptr` May 22 21:30:01 cache0102 traffic_manager[6292]: [May 22 21:30:01.658] traffic_crashlo NOTE: crashlog started, target=16246, debug=false syslog=true, uid=114 euid=0 May 22 21:30:01 cache0102 traffic_crashlog[16260]: traffic_crashlo NOTE: crashlog started, target=16246, debug=false syslog=true, uid=114 euid=0 May 22 21:30:01 cache0102 traffic_crashlog[16260]: traffic_crashlo NOTE: logging to 0x18473c4cb6e0 May 22 21:30:01 cache0102 traffic_manager[6292]: [May 22 21:30:01.659] traffic_crashlo NOTE: logging to 0x18473c4cb6e0 May 22 21:30:04 cache0102 traffic_manager[6292]: [May 22 21:30:04.197] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-213001.log May 22 21:30:04 cache0102 traffic_crashlog[16260]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-213001.log May 22 21:30:04 cache0102 traffic_manager[6292]: traffic_server: received signal 6 (Aborted) May 22 21:30:04 cache0102 traffic_manager[6292]: traffic_server - STACK TRACE: May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x10b76b432ef0] May 22 21:30:04 cache0102 traffic_manager[6292]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x62a740d4b980] May 22 21:30:04 cache0102 traffic_manager[6292]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x62a73fe43fb7] May 22 21:30:04 cache0102 traffic_manager[6292]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x62a73fe45921] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x62a7429e0b08] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x62a7429dd7d5] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(_ZN6HttpSM17get_http_scheduleEiPv+0x0)[0x10b76b4a2250] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(_ZN6HttpSM22state_http_server_openEiPv+0x2e2)[0x10b76b4af562] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x10b76b4aade0] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(_ZN12CheckConnect14handle_connectEiP5Event+0x649)[0x10b76b6dc8e9] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(+0x36cd32)[0x10b76b6ded32] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0xd30)[0x10b76b6e77c0] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x27f)[0x10b76b6cec0f] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x10b76b6ceef5] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x10b76b720409] May 22 21:30:04 cache0102 traffic_manager[6292]: /usr/bin/traffic_server(+0x3ac91a)[0x10b76b71e91a] May 22 21:30:04 cache0102 traffic_manager[6292]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x62a740d406db] May 22 21:30:04 cache0102 traffic_manager[6292]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x62a73ff2671f] I also see: May 24 02:05:18 cache0302 traffic_manager[21310]: [May 24 02:05:18.715] traffic_crashlo NOTE: crashlog started, target=28956, debug=false syslog=true, uid=114 euid=0 May 24 02:05:18 cache0302 traffic_manager[21310]: [May 24 02:05:18.715] traffic_crashlo NOTE: logging to 0x1a09c40167a0 May 24 02:05:18 cache0302 traffic_crashlog[28962]: traffic_crashlo NOTE: crashlog started, target=28956, debug=false syslog=true, uid=114 euid=0 May 24 02:05:18 cache0302 traffic_crashlog[28962]: traffic_crashlo NOTE: logging to 0x1a09c40167a0 May 24 02:05:21 cache0302 traffic_manager[21310]: [May 24 02:05:21.582] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-24-020518.log May 24 02:05:21 cache0302 traffic_crashlog[28962]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-24-020518.log May 24 02:05:21 cache0302 traffic_manager[21310]: traffic_server: received signal 11 (Segmentation fault) May 24 02:05:21 cache0302 traffic_manager[21310]: traffic_servertraffic_server - STACK TRACE: May 24 02:05:21 cache0302 traffic_manager[21310]: - STACK TRACE: May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x14d500a4fef0] May 24 02:05:21 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x733afa434980] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM21setup_server_transferEv+0x620)[0x14d500abe110] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7OpenDir11close_writeEP7CacheVC+0x1b8)[0x14d500c20d18] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x225)[0x14d500ac4095] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7CacheVC17openWriteCloseDirEiP5Event+0xab)[0x14d500c4ae9b] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x604)[0x14d500ac4a24] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN11HttpCacheSM8end_bothEv+0xa2)[0x14d500ace812] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xe37)[0x14d500acbf47] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM9kill_thisEv+0x83)[0x14d500ac7463] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x4f)[0x14d500ab4bbf] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x17f)[0x14d500ac7e4f] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xba)[0x14d500ac3f2a] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36cd32)[0x14d500cfbd32] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x604)[0x14d500ac4a24] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x14d500cfd329] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM33state_read_server_response_headerEiPv+0x46d)[0x14d500ac594d] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x14d500cee24c] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x14d500ac7de0] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x14d500cefd72] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x52)[0x14d500d00242] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x14d500d3bf92] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x17b5)[0x14d500cc8ec5] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x14d500d3d233] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x264)[0x14d500cebbf4] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x14d500d3b91a] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x14d500cebef5] May 24 02:05:21 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x733afa4296db] May 24 02:05:21 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x14d500d3d409] May 24 02:05:21 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x733af960f71f] Sometimes it takes a day or two to crash, but sometimes it's much more rapid. Here's a bunch from the span of a couple of hours: May 22 22:34:33 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 22 22:34:33 cache0302 traffic_server[22524]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 22 22:34:33 cache0302 traffic_manager[21310]: [May 22 22:34:33.988] traffic_crashlo NOTE: crashlog started, target=22524, debug=false syslog=true, uid=114 euid=0 May 22 22:34:33 cache0302 traffic_crashlog[22527]: traffic_crashlo NOTE: crashlog started, target=22524, debug=false syslog=true, uid=114 euid=0 May 22 22:34:33 cache0302 traffic_manager[21310]: [May 22 22:34:33.989] traffic_crashlo NOTE: logging to 0x10ba7153b000 May 22 22:34:33 cache0302 traffic_crashlog[22527]: traffic_crashlo NOTE: logging to 0x10ba7153b000 May 22 22:34:36 cache0302 traffic_manager[21310]: [May 22 22:34:36.755] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-223433.log May 22 22:34:36 cache0302 traffic_crashlog[22527]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-223433.log May 22 22:34:36 cache0302 traffic_manager[21310]: traffic_server: received signal 6 (Aborted) May 22 22:34:36 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 22 22:34:36 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1bedf9b81ef0] May 22 22:34:36 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x75b896f0f980] May 22 22:34:36 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x75b896007fb7] May 22 22:34:36 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x75b896009921] May 22 22:34:36 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x75b898ba4b08] May 22 22:34:36 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x75b898ba17d5] May 22 22:34:36 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1bedf9bf9d0b] May 22 22:34:36 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36cd32)[0x1bedf9e2dd32] May 22 22:34:36 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1bedf9e2f329] May 22 22:34:36 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x418)[0x1bedf9e21f18] May 22 22:34:36 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1bedf9e6df92] May 22 22:34:36 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1bedf9e6f233] May 22 22:34:36 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x1bedf9e6d91a] May 22 22:34:36 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x75b896f046db] May 22 22:34:36 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x75b8960ea71f] May 22 22:34:46 cache0302 traffic_server[8819]: NOTE: --- traffic_server Starting --- May 22 22:34:46 cache0302 traffic_server[8819]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 2021 at 15:49:55) May 22 22:41:09 cache0302 traffic_manager[21310]: [May 22 22:41:09.044] traffic_crashlo NOTE: crashlog started, target=8819, debug=false syslog=true, uid=114 euid=0 May 22 22:41:09 cache0302 traffic_manager[21310]: [May 22 22:41:09.044] traffic_crashlo NOTE: logging to 0x13fb55f0e020 May 22 22:41:09 cache0302 traffic_crashlog[8825]: traffic_crashlo NOTE: crashlog started, target=8819, debug=false syslog=true, uid=114 euid=0 May 22 22:41:09 cache0302 traffic_crashlog[8825]: traffic_crashlo NOTE: logging to 0x13fb55f0e020 May 22 22:41:10 cache0302 traffic_manager[21310]: [May 22 22:41:10.463] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-224109.log May 22 22:41:10 cache0302 traffic_crashlog[8825]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-224109.log May 22 22:41:10 cache0302 traffic_manager[21310]: traffic_server: received signal 11 (Segmentation fault) May 22 22:41:10 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x191e98ab9ef0] May 22 22:41:10 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x67dee0e2f980] May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x191e98b88567] May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x191e98b1888c] May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x191e98b84ffd] May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x191e98b85b5a] May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36d222)[0x191e98d66222] May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x191e98d672cf] May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x191e98d59f92] May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x191e98da5f92] May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x191e98da7233] May 22 22:41:10 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x191e98da591a] May 22 22:41:10 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x67dee0e246db] May 22 22:41:10 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x67dee000a71f] May 22 22:41:14 cache0302 traffic_server[11594]: NOTE: --- traffic_server Starting --- May 22 22:41:14 cache0302 traffic_server[11594]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 2021 at 15:49:55) May 22 23:04:37 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 22 23:04:37 cache0302 traffic_server[11594]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 22 23:04:37 cache0302 traffic_manager[21310]: [May 22 23:04:37.520] traffic_crashlo NOTE: crashlog started, target=11594, debug=false syslog=true, uid=114 euid=0 May 22 23:04:37 cache0302 traffic_crashlog[11600]: traffic_crashlo NOTE: crashlog started, target=11594, debug=false syslog=true, uid=114 euid=0 May 22 23:04:37 cache0302 traffic_crashlog[11600]: traffic_crashlo NOTE: logging to 0x1decbe248a90 May 22 23:04:37 cache0302 traffic_manager[21310]: [May 22 23:04:37.520] traffic_crashlo NOTE: logging to 0x1decbe248a90 May 22 23:04:38 cache0302 traffic_manager[21310]: [May 22 23:04:38.880] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-230437.log May 22 23:04:38 cache0302 traffic_crashlog[11600]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-230437.log May 22 23:04:38 cache0302 traffic_manager[21310]: traffic_server: received signal 6 (Aborted) May 22 23:04:38 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1867ffb06ef0] May 22 23:04:38 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x747f2bd26980] May 22 23:04:38 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x747f2ae1efb7] May 22 23:04:38 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x747f2ae20921] May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x747f2d9bbb08] May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x747f2d9b87d5] May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1867ffb7ed0b] May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36cd32)[0x1867ffdb2d32] May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1867ffdb4329] May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1867ffda524c] May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1867ffda6d72] May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1867ffdf2f92] May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1867ffdf4233] May 22 23:04:38 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x1867ffdf291a] May 22 23:04:38 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x747f2bd1b6db] May 22 23:04:38 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x747f2af0171f] May 22 23:04:43 cache0302 traffic_server[23705]: NOTE: --- traffic_server Starting --- May 22 23:04:43 cache0302 traffic_server[23705]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 2021 at 15:49:55) May 22 23:29:41 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:5854: failed assertion `server_session == nullptr` May 22 23:29:41 cache0302 traffic_server[23705]: Fatal: HttpSM.cc:5854: failed assertion `server_session == nullptr` May 22 23:29:41 cache0302 traffic_manager[21310]: [May 22 23:29:41.142] traffic_crashlo NOTE: crashlog started, target=23705, debug=false syslog=true, uid=114 euid=0 May 22 23:29:41 cache0302 traffic_crashlog[23707]: traffic_crashlo NOTE: crashlog started, target=23705, debug=false syslog=true, uid=114 euid=0 May 22 23:29:41 cache0302 traffic_crashlog[23707]: traffic_crashlo NOTE: logging to 0x19db57d47390 May 22 23:29:41 cache0302 traffic_manager[21310]: [May 22 23:29:41.142] traffic_crashlo NOTE: logging to 0x19db57d47390 May 22 23:29:42 cache0302 traffic_manager[21310]: [May 22 23:29:42.544] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-232941.log May 22 23:29:42 cache0302 traffic_crashlog[23707]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-232941.log May 22 23:29:42 cache0302 traffic_manager[21310]: traffic_server: received signal 6 (Aborted) May 22 23:29:42 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x147b5d156ef0] May 22 23:29:42 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x71f05ed35980] May 22 23:29:42 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x71f05de2dfb7] May 22 23:29:42 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x71f05de2f921] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x71f0609cab08] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x71f0609c77d5] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM17get_http_scheduleEiPv+0x0)[0x147b5d1c6250] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM22state_http_server_openEiPv+0x2e2)[0x147b5d1d3562] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x147b5d1cede0] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN12CheckConnect14handle_connectEiP5Event+0x649)[0x147b5d4008e9] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36cd32)[0x147b5d402d32] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0xd30)[0x147b5d40b7c0] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x27f)[0x147b5d3f2c0f] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x147b5d3f2ef5] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x147b5d444409] May 22 23:29:42 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x147b5d44291a] May 22 23:29:42 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x71f05ed2a6db] May 22 23:29:42 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x71f05df1071f] May 22 23:29:46 cache0302 traffic_server[2493]: NOTE: --- traffic_server Starting --- May 22 23:29:46 cache0302 traffic_server[2493]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 2021 at 15:49:55) May 22 23:43:15 cache0302 traffic_manager[21310]: [May 22 23:43:15.997] traffic_crashlo NOTE: crashlog started, target=2493, debug=false syslog=true, uid=114 euid=0 May 22 23:43:15 cache0302 traffic_crashlog[2495]: traffic_crashlo NOTE: crashlog started, target=2493, debug=false syslog=true, uid=114 euid=0 May 22 23:43:15 cache0302 traffic_manager[21310]: [May 22 23:43:15.997] traffic_crashlo NOTE: logging to 0x16dba7520110 May 22 23:43:15 cache0302 traffic_crashlog[2495]: traffic_crashlo NOTE: logging to 0x16dba7520110 May 22 23:43:17 cache0302 traffic_manager[21310]: [May 22 23:43:17.331] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-234315.log May 22 23:43:17 cache0302 traffic_crashlog[2495]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-234315.log May 22 23:43:17 cache0302 traffic_manager[21310]: traffic_server: received signal 11 (Segmentation fault) May 22 23:43:17 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1e354ceb6ef0] May 22 23:43:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6ac17bd5b980] May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x1e354cf85567] May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x1e354cf1588c] May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x1e354cf81ffd] May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x1e354cf82b5a] May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36d222)[0x1e354d163222] May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x1e354d1642cf] May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x1e354d156f92] May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1e354d1a2f92] May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1e354d1a4233] May 22 23:43:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x1e354d1a291a] May 22 23:43:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6ac17bd506db] May 22 23:43:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6ac17af3671f] May 22 23:43:21 cache0302 traffic_server[9064]: NOTE: --- traffic_server Starting --- May 22 23:43:21 cache0302 traffic_server[9064]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 2021 at 15:49:55) May 22 23:51:29 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 22 23:51:29 cache0302 traffic_server[9064]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 22 23:51:29 cache0302 traffic_manager[21310]: [May 22 23:51:29.199] traffic_crashlo NOTE: crashlog started, target=9064, debug=false syslog=true, uid=114 euid=0 May 22 23:51:29 cache0302 traffic_crashlog[9066]: traffic_crashlo NOTE: crashlog started, target=9064, debug=false syslog=true, uid=114 euid=0 May 22 23:51:29 cache0302 traffic_crashlog[9066]: traffic_crashlo NOTE: logging to 0x153045b30140 May 22 23:51:29 cache0302 traffic_manager[21310]: [May 22 23:51:29.199] traffic_crashlo NOTE: logging to 0x153045b30140 May 22 23:51:30 cache0302 traffic_manager[21310]: [May 22 23:51:30.396] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-235129.log May 22 23:51:30 cache0302 traffic_crashlog[9066]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-22-235129.log May 22 23:51:30 cache0302 traffic_manager[21310]: traffic_server: received signal 6 (Aborted) May 22 23:51:30 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1cfb65150ef0] May 22 23:51:30 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x76d5135c1980] May 22 23:51:30 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x76d5126b9fb7] May 22 23:51:30 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x76d5126bb921] May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x76d515256b08] May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x76d5152537d5] May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1cfb651c8d0b] May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36cd32)[0x1cfb653fcd32] May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1cfb653fe329] May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1cfb653ef24c] May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1cfb653f0d72] May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1cfb6543cf92] May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1cfb6543e233] May 22 23:51:30 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x1cfb6543c91a] May 22 23:51:30 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x76d5135b66db] May 22 23:51:30 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x76d51279c71f] May 22 23:51:33 cache0302 traffic_server[12951]: NOTE: --- traffic_server Starting --- May 22 23:51:33 cache0302 traffic_server[12951]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 2021 at 15:49:55) May 23 00:08:16 cache0302 traffic_manager[21310]: [May 23 00:08:16.507] traffic_crashlo NOTE: crashlog started, target=12951, debug=false syslog=true, uid=114 euid=0 May 23 00:08:16 cache0302 traffic_manager[21310]: [May 23 00:08:16.507] traffic_crashlo NOTE: logging to 0x19c404cfdad0 May 23 00:08:16 cache0302 traffic_crashlog[12953]: traffic_crashlo NOTE: crashlog started, target=12951, debug=false syslog=true, uid=114 euid=0 May 23 00:08:16 cache0302 traffic_crashlog[12953]: traffic_crashlo NOTE: logging to 0x19c404cfdad0 May 23 00:08:17 cache0302 traffic_manager[21310]: [May 23 00:08:17.776] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-23-000816.log May 23 00:08:17 cache0302 traffic_crashlog[12953]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-23-000816.log May 23 00:08:17 cache0302 traffic_manager[21310]: traffic_server: received signal 11 (Segmentation fault) May 23 00:08:17 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x19c0a1524ef0] May 23 00:08:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6d52207fd980] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7OpenDir11close_writeEP7CacheVC+0x1b8)[0x19c0a16f5d18] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7CacheVC17openWriteCloseDirEiP5Event+0xab)[0x19c0a171fe9b] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN11HttpCacheSM8end_bothEv+0xa2)[0x19c0a15a3812] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM9kill_thisEv+0x83)[0x19c0a159c463] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x17f)[0x19c0a159ce4f] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36cd32)[0x19c0a17d0d32] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x19c0a17d2329] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x19c0a17c324c] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x19c0a17c4d72] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x19c0a1810f92] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x19c0a1812233] May 23 00:08:17 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x19c0a181091a] May 23 00:08:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6d52207f26db] May 23 00:08:17 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6d521f9d871f] May 23 00:08:21 cache0302 traffic_server[20789]: NOTE: --- traffic_server Starting --- May 23 00:08:21 cache0302 traffic_server[20789]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 2021 at 15:49:55) May 23 00:14:46 cache0302 traffic_manager[21310]: [May 23 00:14:46.669] traffic_crashlo NOTE: crashlog started, target=20789, debug=false syslog=true, uid=114 euid=0 May 23 00:14:46 cache0302 traffic_crashlog[20796]: traffic_crashlo NOTE: crashlog started, target=20789, debug=false syslog=true, uid=114 euid=0 May 23 00:14:46 cache0302 traffic_manager[21310]: [May 23 00:14:46.669] traffic_crashlo NOTE: logging to 0x14f170d7b660 May 23 00:14:46 cache0302 traffic_crashlog[20796]: traffic_crashlo NOTE: logging to 0x14f170d7b660 May 23 00:14:47 cache0302 traffic_manager[21310]: [May 23 00:14:47.825] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-23-001446.log May 23 00:14:47 cache0302 traffic_crashlog[20796]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-23-001446.log May 23 00:14:47 cache0302 traffic_manager[21310]: traffic_server: received signal 11 (Segmentation fault) May 23 00:14:47 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x18c9c6353ef0] May 23 00:14:47 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x69e99706b980] May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x18c9c6422567] May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x18c9c63b288c] May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x18c9c641effd] May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x18c9c641fb5a] May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36d222)[0x18c9c6600222] May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x18c9c66012cf] May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x18c9c65f3f92] May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x18c9c663ff92] May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x18c9c6641233] May 23 00:14:47 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x18c9c663f91a] May 23 00:14:47 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x69e9970606db] May 23 00:14:47 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x69e99624671f] May 23 00:14:51 cache0302 traffic_server[23441]: NOTE: --- traffic_server Starting --- May 23 00:14:51 cache0302 traffic_server[23441]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 2021 at 15:49:55) May 23 00:18:52 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 23 00:18:52 cache0302 traffic_server[23441]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 23 00:18:52 cache0302 traffic_manager[21310]: [May 23 00:18:52.120] traffic_crashlo NOTE: crashlog started, target=23441, debug=false syslog=true, uid=114 euid=0 May 23 00:18:52 cache0302 traffic_manager[21310]: [May 23 00:18:52.120] traffic_crashlo NOTE: logging to 0x12c5d40cced0 May 23 00:18:52 cache0302 traffic_crashlog[23443]: traffic_crashlo NOTE: crashlog started, target=23441, debug=false syslog=true, uid=114 euid=0 May 23 00:18:52 cache0302 traffic_crashlog[23443]: traffic_crashlo NOTE: logging to 0x12c5d40cced0 May 23 00:18:53 cache0302 traffic_manager[21310]: [May 23 00:18:53.113] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-23-001852.log May 23 00:18:53 cache0302 traffic_crashlog[23443]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-23-001852.log May 23 00:18:53 cache0302 traffic_manager[21310]: traffic_server: received signal 6 (Aborted) May 23 00:18:53 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 23 00:18:53 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1b7f44513ef0] May 23 00:18:53 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7244e5c04980] May 23 00:18:53 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7244e4cfcfb7] May 23 00:18:53 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7244e4cfe921] May 23 00:18:53 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x7244e7899b08] May 23 00:18:53 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x7244e78967d5] May 23 00:18:53 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1b7f4458bd0b] May 23 00:18:53 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36cd32)[0x1b7f447bfd32] May 23 00:18:53 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1b7f447c1329] May 23 00:18:53 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x418)[0x1b7f447b3f18] May 23 00:18:53 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1b7f447fff92] May 23 00:18:53 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1b7f44801233] May 23 00:18:53 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x1b7f447ff91a] May 23 00:18:53 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7244e5bf96db] May 23 00:18:53 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7244e4ddf71f] May 23 00:18:56 cache0302 traffic_server[26362]: NOTE: --- traffic_server Starting --- May 23 00:18:56 cache0302 traffic_server[26362]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 2021 at 15:49:55) May 23 00:24:24 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 23 00:24:24 cache0302 traffic_server[26362]: Fatal: HttpSM.cc:2533: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` May 23 00:24:24 cache0302 traffic_manager[21310]: [May 23 00:24:24.239] traffic_crashlo NOTE: crashlog started, target=26362, debug=false syslog=true, uid=114 euid=0 May 23 00:24:24 cache0302 traffic_manager[21310]: [May 23 00:24:24.240] traffic_crashlo NOTE: logging to 0x190a947ba530 May 23 00:24:24 cache0302 traffic_crashlog[26365]: traffic_crashlo NOTE: crashlog started, target=26362, debug=false syslog=true, uid=114 euid=0 May 23 00:24:24 cache0302 traffic_crashlog[26365]: traffic_crashlo NOTE: logging to 0x190a947ba530 May 23 00:24:25 cache0302 traffic_manager[21310]: [May 23 00:24:25.274] traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-23-002424.log May 23 00:24:25 cache0302 traffic_crashlog[26365]: traffic_crashlo ERROR: wrote crash log to /var/log/trafficserver/crash-2021-05-23-002424.log May 23 00:24:25 cache0302 traffic_manager[21310]: traffic_server: received signal 6 (Aborted) May 23 00:24:25 cache0302 traffic_manager[21310]: traffic_server - STACK TRACE: May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1cf361726ef0] May 23 00:24:25 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6d01e9b4d980] May 23 00:24:25 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x6d01e8c45fb7] May 23 00:24:25 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x6d01e8c47921] May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x6d01eb7e2b08] May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x6d01eb7df7d5] May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1cf36179ed0b] May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x36cd32)[0x1cf3619d2d32] May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1cf3619d4329] May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1cf3619c524c] May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1cf3619c6d72] May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1cf361a12f92] May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1cf361a14233] May 23 00:24:25 cache0302 traffic_manager[21310]: /usr/bin/traffic_server(+0x3ac91a)[0x1cf361a1291a] May 23 00:24:25 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6d01e9b426db] May 23 00:24:25 cache0302 traffic_manager[21310]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6d01e8d2871f] May 23 00:24:28 cache0302 traffic_server[28956]: NOTE: --- traffic_server Starting --- May 23 00:24:28 cache0302 traffic_server[28956]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 2021 at 15:49:55)
