Are you using per thread or global session pools? If you are using global, it might be worth a try to switch to per thread (https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html?highlight=records.config#proxy-config-http-server-session-sharing-pool <https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html?highlight=records.config#proxy-config-http-server-session-sharing-pool>).
You can try turning off keep alive as referenced in the issue, but I have a feeling it won’t help. If you can run an ASAN in production with the proxy.config.proxy_binary_opts set to "-M -f -F” that would really help to tracking this issue down. There is some overhead with running ASAN and I don’t know if your servers can handle it (https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html?highlight=records.config#proxy-config-proxy-binary-opts <https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html?highlight=records.config#proxy-config-proxy-binary-opts>). -Bryan > On May 24, 2021, at 4:12 PM, Mark Moseley <[email protected]> wrote: > > I've been chasing an issue for a while now. This is 8.1.1 running on Ubuntu > Focal. I'm seeing this same crash (Fatal: HttpSM.cc:2533: failed assertion > `magic == HTTP_SM_MAGIC_ALIVE`) across different nodes that are handling > unrelated traffic, but also ones that don't mention HTTP_SM_MAGIC_ALIVE. > > Background: > ATS is doing reverse proxy caching in front of web hosting backends, so a > completely random assortment of backend domains and a random assortment of > file types. Customer domains resolve to a group of 3-node clusters. I'm > seeing the same crash across different clusters, so it wouldn't be related to > the same backend origin domain(s). Backends are derived via remap+tslua, > doing DNS lookups. These boxes constantly run at 200-400 reqs/seq all day, > but I'm only seeing this crash anywhere from "once every couple days" to a > couple of times a day on a given node (so the other millions of requests per > day are processed just fine). ATS sits behind a front-end nginx instance (so > nginx->ats is all localhost; origins are remote but on the same LAN). > > It sounds similar to https://github.com/apache/trafficserver/issues/4921 > <https://github.com/apache/trafficserver/issues/4921> but that ticket also > says that it was fixed for the OP in 8.1, so that could be a red herring. > > Any patches/settings I can use to mitigate this? Would turning off keepalives > help as a bandaid? (Outbound keepalives to the origins would hurt a bit; > inbound would probably be fine due to nginx->ats being on localhost). > > I'm also fine upgrading to 9.x but I ran into the issue where the connection > count grows without bound, despite the connections not really being open > (https://github.com/apache/trafficserver/issues/7471 > <https://github.com/apache/trafficserver/issues/7471>) > > I don't have any core files handy and these boxes have anywhere from > 64gb-224gb ATS RAM cache, so I imagine a core would be horrifyingly large. > The error is always similar to this (most of these are from the same box but > that's just out of convenience; I'm seeing the same variety of segfaults > across all of my boxes): > > May 24 07:05:14 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 24 07:05:14 cache0302 traffic_server[30820]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 24 07:05:14 cache0302 traffic_manager[21310]: [May 24 07:05:14.660] > traffic_crashlo NOTE: crashlog started, target=30820, debug=false > syslog=true, uid=114 euid=0 > May 24 07:05:14 cache0302 traffic_crashlog[30822]: traffic_crashlo NOTE: > crashlog started, target=30820, debug=false syslog=true, uid=114 euid=0 > May 24 07:05:14 cache0302 traffic_crashlog[30822]: traffic_crashlo NOTE: > logging to 0x14e10a33ad20 > May 24 07:05:14 cache0302 traffic_manager[21310]: [May 24 07:05:14.660] > traffic_crashlo NOTE: logging to 0x14e10a33ad20 > May 24 07:05:17 cache0302 traffic_manager[21310]: [May 24 07:05:17.094] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-24-070514.log > May 24 07:05:17 cache0302 traffic_crashlog[30822]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-24-070514.log > May 24 07:05:17 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 24 07:05:17 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x15ff15ac4ef0] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x675351928980] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x675350a20fb7] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x675350a22921] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x6753535bdb08] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x6753535ba7d5] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x15ff15b3cd0b] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x15ff15d70d32] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x15ff15d72329] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x15ff15d6324c] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x15ff15d64d72] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x15ff15db0f92] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x15ff15db2233] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x15ff15db091a] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x67535191d6db] > May 24 07:05:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x675350b0371f] > > Sometimes I get this one: > > May 22 21:30:01 cache0102 traffic_manager[6292]: Fatal: HttpSM.cc:5854: > failed assertion `server_session == nullptr` > May 22 21:30:01 cache0102 traffic_server[16246]: Fatal: HttpSM.cc:5854: > failed assertion `server_session == nullptr` > May 22 21:30:01 cache0102 traffic_manager[6292]: [May 22 21:30:01.658] > traffic_crashlo NOTE: crashlog started, target=16246, debug=false > syslog=true, uid=114 euid=0 > May 22 21:30:01 cache0102 traffic_crashlog[16260]: traffic_crashlo NOTE: > crashlog started, target=16246, debug=false syslog=true, uid=114 euid=0 > May 22 21:30:01 cache0102 traffic_crashlog[16260]: traffic_crashlo NOTE: > logging to 0x18473c4cb6e0 > May 22 21:30:01 cache0102 traffic_manager[6292]: [May 22 21:30:01.659] > traffic_crashlo NOTE: logging to 0x18473c4cb6e0 > May 22 21:30:04 cache0102 traffic_manager[6292]: [May 22 21:30:04.197] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-213001.log > May 22 21:30:04 cache0102 traffic_crashlog[16260]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-213001.log > May 22 21:30:04 cache0102 traffic_manager[6292]: traffic_server: received > signal 6 (Aborted) > May 22 21:30:04 cache0102 traffic_manager[6292]: traffic_server - STACK TRACE: > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x10b76b432ef0] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x62a740d4b980] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x62a73fe43fb7] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x62a73fe45921] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x62a7429e0b08] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x62a7429dd7d5] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN6HttpSM17get_http_scheduleEiPv+0x0)[0x10b76b4a2250] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN6HttpSM22state_http_server_openEiPv+0x2e2)[0x10b76b4af562] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x10b76b4aade0] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN12CheckConnect14handle_connectEiP5Event+0x649)[0x10b76b6dc8e9] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(+0x36cd32)[0x10b76b6ded32] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0xd30)[0x10b76b6e77c0] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x27f)[0x10b76b6cec0f] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x10b76b6ceef5] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x10b76b720409] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /usr/bin/traffic_server(+0x3ac91a)[0x10b76b71e91a] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x62a740d406db] > May 22 21:30:04 cache0102 traffic_manager[6292]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x62a73ff2671f] > > > I also see: > > May 24 02:05:18 cache0302 traffic_manager[21310]: [May 24 02:05:18.715] > traffic_crashlo NOTE: crashlog started, target=28956, debug=false > syslog=true, uid=114 euid=0 > May 24 02:05:18 cache0302 traffic_manager[21310]: [May 24 02:05:18.715] > traffic_crashlo NOTE: logging to 0x1a09c40167a0 > May 24 02:05:18 cache0302 traffic_crashlog[28962]: traffic_crashlo NOTE: > crashlog started, target=28956, debug=false syslog=true, uid=114 euid=0 > May 24 02:05:18 cache0302 traffic_crashlog[28962]: traffic_crashlo NOTE: > logging to 0x1a09c40167a0 > May 24 02:05:21 cache0302 traffic_manager[21310]: [May 24 02:05:21.582] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-24-020518.log > May 24 02:05:21 cache0302 traffic_crashlog[28962]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-24-020518.log > May 24 02:05:21 cache0302 traffic_manager[21310]: traffic_server: received > signal 11 (Segmentation fault) > May 24 02:05:21 cache0302 traffic_manager[21310]: > traffic_servertraffic_server - STACK TRACE: > May 24 02:05:21 cache0302 traffic_manager[21310]: - STACK TRACE: > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x14d500a4fef0] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x733afa434980] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM21setup_server_transferEv+0x620)[0x14d500abe110] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7OpenDir11close_writeEP7CacheVC+0x1b8)[0x14d500c20d18] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x225)[0x14d500ac4095] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7CacheVC17openWriteCloseDirEiP5Event+0xab)[0x14d500c4ae9b] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x604)[0x14d500ac4a24] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN11HttpCacheSM8end_bothEv+0xa2)[0x14d500ace812] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xe37)[0x14d500acbf47] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM9kill_thisEv+0x83)[0x14d500ac7463] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x4f)[0x14d500ab4bbf] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x17f)[0x14d500ac7e4f] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xba)[0x14d500ac3f2a] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x14d500cfbd32] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x604)[0x14d500ac4a24] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x14d500cfd329] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM33state_read_server_response_headerEiPv+0x46d)[0x14d500ac594d] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x14d500cee24c] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x14d500ac7de0] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x14d500cefd72] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x52)[0x14d500d00242] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x14d500d3bf92] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x17b5)[0x14d500cc8ec5] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x14d500d3d233] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x264)[0x14d500cebbf4] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x14d500d3b91a] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x14d500cebef5] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x733afa4296db] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x14d500d3d409] > May 24 02:05:21 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x733af960f71f] > > > Sometimes it takes a day or two to crash, but sometimes it's much more rapid. > Here's a bunch from the span of a couple of hours: > > May 22 22:34:33 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 22:34:33 cache0302 traffic_server[22524]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 22:34:33 cache0302 traffic_manager[21310]: [May 22 22:34:33.988] > traffic_crashlo NOTE: crashlog started, target=22524, debug=false > syslog=true, uid=114 euid=0 > May 22 22:34:33 cache0302 traffic_crashlog[22527]: traffic_crashlo NOTE: > crashlog started, target=22524, debug=false syslog=true, uid=114 euid=0 > May 22 22:34:33 cache0302 traffic_manager[21310]: [May 22 22:34:33.989] > traffic_crashlo NOTE: logging to 0x10ba7153b000 > May 22 22:34:33 cache0302 traffic_crashlog[22527]: traffic_crashlo NOTE: > logging to 0x10ba7153b000 > May 22 22:34:36 cache0302 traffic_manager[21310]: [May 22 22:34:36.755] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-223433.log > May 22 22:34:36 cache0302 traffic_crashlog[22527]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-223433.log > May 22 22:34:36 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 22 22:34:36 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1bedf9b81ef0] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x75b896f0f980] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x75b896007fb7] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x75b896009921] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x75b898ba4b08] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x75b898ba17d5] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1bedf9bf9d0b] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x1bedf9e2dd32] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1bedf9e2f329] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x418)[0x1bedf9e21f18] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1bedf9e6df92] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1bedf9e6f233] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1bedf9e6d91a] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x75b896f046db] > May 22 22:34:36 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x75b8960ea71f] > May 22 22:34:46 cache0302 traffic_server[8819]: NOTE: --- traffic_server > Starting --- > May 22 22:34:46 cache0302 traffic_server[8819]: NOTE: traffic_server Version: > Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 > 2021 at 15:49:55) > May 22 22:41:09 cache0302 traffic_manager[21310]: [May 22 22:41:09.044] > traffic_crashlo NOTE: crashlog started, target=8819, debug=false syslog=true, > uid=114 euid=0 > May 22 22:41:09 cache0302 traffic_manager[21310]: [May 22 22:41:09.044] > traffic_crashlo NOTE: logging to 0x13fb55f0e020 > May 22 22:41:09 cache0302 traffic_crashlog[8825]: traffic_crashlo NOTE: > crashlog started, target=8819, debug=false syslog=true, uid=114 euid=0 > May 22 22:41:09 cache0302 traffic_crashlog[8825]: traffic_crashlo NOTE: > logging to 0x13fb55f0e020 > May 22 22:41:10 cache0302 traffic_manager[21310]: [May 22 22:41:10.463] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-224109.log > May 22 22:41:10 cache0302 traffic_crashlog[8825]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-224109.log > May 22 22:41:10 cache0302 traffic_manager[21310]: traffic_server: received > signal 11 (Segmentation fault) > May 22 22:41:10 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x191e98ab9ef0] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x67dee0e2f980] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x191e98b88567] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x191e98b1888c] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x191e98b84ffd] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x191e98b85b5a] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36d222)[0x191e98d66222] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x191e98d672cf] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x191e98d59f92] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x191e98da5f92] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x191e98da7233] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x191e98da591a] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x67dee0e246db] > May 22 22:41:10 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x67dee000a71f] > May 22 22:41:14 cache0302 traffic_server[11594]: NOTE: --- traffic_server > Starting --- > May 22 22:41:14 cache0302 traffic_server[11594]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on > Apr 9 2021 at 15:49:55) > May 22 23:04:37 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 23:04:37 cache0302 traffic_server[11594]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 23:04:37 cache0302 traffic_manager[21310]: [May 22 23:04:37.520] > traffic_crashlo NOTE: crashlog started, target=11594, debug=false > syslog=true, uid=114 euid=0 > May 22 23:04:37 cache0302 traffic_crashlog[11600]: traffic_crashlo NOTE: > crashlog started, target=11594, debug=false syslog=true, uid=114 euid=0 > May 22 23:04:37 cache0302 traffic_crashlog[11600]: traffic_crashlo NOTE: > logging to 0x1decbe248a90 > May 22 23:04:37 cache0302 traffic_manager[21310]: [May 22 23:04:37.520] > traffic_crashlo NOTE: logging to 0x1decbe248a90 > May 22 23:04:38 cache0302 traffic_manager[21310]: [May 22 23:04:38.880] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-230437.log > May 22 23:04:38 cache0302 traffic_crashlog[11600]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-230437.log > May 22 23:04:38 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 22 23:04:38 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1867ffb06ef0] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x747f2bd26980] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x747f2ae1efb7] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x747f2ae20921] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x747f2d9bbb08] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x747f2d9b87d5] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1867ffb7ed0b] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x1867ffdb2d32] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1867ffdb4329] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1867ffda524c] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1867ffda6d72] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1867ffdf2f92] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1867ffdf4233] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1867ffdf291a] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x747f2bd1b6db] > May 22 23:04:38 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x747f2af0171f] > May 22 23:04:43 cache0302 traffic_server[23705]: NOTE: --- traffic_server > Starting --- > May 22 23:04:43 cache0302 traffic_server[23705]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on > Apr 9 2021 at 15:49:55) > May 22 23:29:41 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:5854: > failed assertion `server_session == nullptr` > May 22 23:29:41 cache0302 traffic_server[23705]: Fatal: HttpSM.cc:5854: > failed assertion `server_session == nullptr` > May 22 23:29:41 cache0302 traffic_manager[21310]: [May 22 23:29:41.142] > traffic_crashlo NOTE: crashlog started, target=23705, debug=false > syslog=true, uid=114 euid=0 > May 22 23:29:41 cache0302 traffic_crashlog[23707]: traffic_crashlo NOTE: > crashlog started, target=23705, debug=false syslog=true, uid=114 euid=0 > May 22 23:29:41 cache0302 traffic_crashlog[23707]: traffic_crashlo NOTE: > logging to 0x19db57d47390 > May 22 23:29:41 cache0302 traffic_manager[21310]: [May 22 23:29:41.142] > traffic_crashlo NOTE: logging to 0x19db57d47390 > May 22 23:29:42 cache0302 traffic_manager[21310]: [May 22 23:29:42.544] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-232941.log > May 22 23:29:42 cache0302 traffic_crashlog[23707]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-232941.log > May 22 23:29:42 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 22 23:29:42 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x147b5d156ef0] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x71f05ed35980] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x71f05de2dfb7] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x71f05de2f921] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x71f0609cab08] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x71f0609c77d5] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM17get_http_scheduleEiPv+0x0)[0x147b5d1c6250] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM22state_http_server_openEiPv+0x2e2)[0x147b5d1d3562] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x147b5d1cede0] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN12CheckConnect14handle_connectEiP5Event+0x649)[0x147b5d4008e9] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x147b5d402d32] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0xd30)[0x147b5d40b7c0] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x27f)[0x147b5d3f2c0f] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x147b5d3f2ef5] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x147b5d444409] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x147b5d44291a] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x71f05ed2a6db] > May 22 23:29:42 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x71f05df1071f] > May 22 23:29:46 cache0302 traffic_server[2493]: NOTE: --- traffic_server > Starting --- > May 22 23:29:46 cache0302 traffic_server[2493]: NOTE: traffic_server Version: > Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 > 2021 at 15:49:55) > May 22 23:43:15 cache0302 traffic_manager[21310]: [May 22 23:43:15.997] > traffic_crashlo NOTE: crashlog started, target=2493, debug=false syslog=true, > uid=114 euid=0 > May 22 23:43:15 cache0302 traffic_crashlog[2495]: traffic_crashlo NOTE: > crashlog started, target=2493, debug=false syslog=true, uid=114 euid=0 > May 22 23:43:15 cache0302 traffic_manager[21310]: [May 22 23:43:15.997] > traffic_crashlo NOTE: logging to 0x16dba7520110 > May 22 23:43:15 cache0302 traffic_crashlog[2495]: traffic_crashlo NOTE: > logging to 0x16dba7520110 > May 22 23:43:17 cache0302 traffic_manager[21310]: [May 22 23:43:17.331] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-234315.log > May 22 23:43:17 cache0302 traffic_crashlog[2495]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-234315.log > May 22 23:43:17 cache0302 traffic_manager[21310]: traffic_server: received > signal 11 (Segmentation fault) > May 22 23:43:17 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1e354ceb6ef0] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6ac17bd5b980] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x1e354cf85567] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x1e354cf1588c] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x1e354cf81ffd] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x1e354cf82b5a] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36d222)[0x1e354d163222] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x1e354d1642cf] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x1e354d156f92] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1e354d1a2f92] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1e354d1a4233] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1e354d1a291a] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6ac17bd506db] > May 22 23:43:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6ac17af3671f] > May 22 23:43:21 cache0302 traffic_server[9064]: NOTE: --- traffic_server > Starting --- > May 22 23:43:21 cache0302 traffic_server[9064]: NOTE: traffic_server Version: > Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on Apr 9 > 2021 at 15:49:55) > May 22 23:51:29 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 23:51:29 cache0302 traffic_server[9064]: Fatal: HttpSM.cc:2533: failed > assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 22 23:51:29 cache0302 traffic_manager[21310]: [May 22 23:51:29.199] > traffic_crashlo NOTE: crashlog started, target=9064, debug=false syslog=true, > uid=114 euid=0 > May 22 23:51:29 cache0302 traffic_crashlog[9066]: traffic_crashlo NOTE: > crashlog started, target=9064, debug=false syslog=true, uid=114 euid=0 > May 22 23:51:29 cache0302 traffic_crashlog[9066]: traffic_crashlo NOTE: > logging to 0x153045b30140 > May 22 23:51:29 cache0302 traffic_manager[21310]: [May 22 23:51:29.199] > traffic_crashlo NOTE: logging to 0x153045b30140 > May 22 23:51:30 cache0302 traffic_manager[21310]: [May 22 23:51:30.396] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-22-235129.log > May 22 23:51:30 cache0302 traffic_crashlog[9066]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-22-235129.log > May 22 23:51:30 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 22 23:51:30 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1cfb65150ef0] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x76d5135c1980] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x76d5126b9fb7] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x76d5126bb921] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x76d515256b08] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x76d5152537d5] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1cfb651c8d0b] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x1cfb653fcd32] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1cfb653fe329] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1cfb653ef24c] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1cfb653f0d72] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1cfb6543cf92] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1cfb6543e233] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1cfb6543c91a] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x76d5135b66db] > May 22 23:51:30 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x76d51279c71f] > May 22 23:51:33 cache0302 traffic_server[12951]: NOTE: --- traffic_server > Starting --- > May 22 23:51:33 cache0302 traffic_server[12951]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on > Apr 9 2021 at 15:49:55) > May 23 00:08:16 cache0302 traffic_manager[21310]: [May 23 00:08:16.507] > traffic_crashlo NOTE: crashlog started, target=12951, debug=false > syslog=true, uid=114 euid=0 > May 23 00:08:16 cache0302 traffic_manager[21310]: [May 23 00:08:16.507] > traffic_crashlo NOTE: logging to 0x19c404cfdad0 > May 23 00:08:16 cache0302 traffic_crashlog[12953]: traffic_crashlo NOTE: > crashlog started, target=12951, debug=false syslog=true, uid=114 euid=0 > May 23 00:08:16 cache0302 traffic_crashlog[12953]: traffic_crashlo NOTE: > logging to 0x19c404cfdad0 > May 23 00:08:17 cache0302 traffic_manager[21310]: [May 23 00:08:17.776] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-23-000816.log > May 23 00:08:17 cache0302 traffic_crashlog[12953]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-23-000816.log > May 23 00:08:17 cache0302 traffic_manager[21310]: traffic_server: received > signal 11 (Segmentation fault) > May 23 00:08:17 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x19c0a1524ef0] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6d52207fd980] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7OpenDir11close_writeEP7CacheVC+0x1b8)[0x19c0a16f5d18] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7CacheVC17openWriteCloseDirEiP5Event+0xab)[0x19c0a171fe9b] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN11HttpCacheSM8end_bothEv+0xa2)[0x19c0a15a3812] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM9kill_thisEv+0x83)[0x19c0a159c463] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x17f)[0x19c0a159ce4f] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x19c0a17d0d32] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x19c0a17d2329] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x19c0a17c324c] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x19c0a17c4d72] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x19c0a1810f92] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x19c0a1812233] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x19c0a181091a] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6d52207f26db] > May 23 00:08:17 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6d521f9d871f] > May 23 00:08:21 cache0302 traffic_server[20789]: NOTE: --- traffic_server > Starting --- > May 23 00:08:21 cache0302 traffic_server[20789]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on > Apr 9 2021 at 15:49:55) > May 23 00:14:46 cache0302 traffic_manager[21310]: [May 23 00:14:46.669] > traffic_crashlo NOTE: crashlog started, target=20789, debug=false > syslog=true, uid=114 euid=0 > May 23 00:14:46 cache0302 traffic_crashlog[20796]: traffic_crashlo NOTE: > crashlog started, target=20789, debug=false syslog=true, uid=114 euid=0 > May 23 00:14:46 cache0302 traffic_manager[21310]: [May 23 00:14:46.669] > traffic_crashlo NOTE: logging to 0x14f170d7b660 > May 23 00:14:46 cache0302 traffic_crashlog[20796]: traffic_crashlo NOTE: > logging to 0x14f170d7b660 > May 23 00:14:47 cache0302 traffic_manager[21310]: [May 23 00:14:47.825] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-23-001446.log > May 23 00:14:47 cache0302 traffic_crashlog[20796]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-23-001446.log > May 23 00:14:47 cache0302 traffic_manager[21310]: traffic_server: received > signal 11 (Segmentation fault) > May 23 00:14:47 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x18c9c6353ef0] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x69e99706b980] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x18c9c6422567] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x18c9c63b288c] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x18c9c641effd] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x18c9c641fb5a] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36d222)[0x18c9c6600222] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x18c9c66012cf] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x18c9c65f3f92] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x18c9c663ff92] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x18c9c6641233] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x18c9c663f91a] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x69e9970606db] > May 23 00:14:47 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x69e99624671f] > May 23 00:14:51 cache0302 traffic_server[23441]: NOTE: --- traffic_server > Starting --- > May 23 00:14:51 cache0302 traffic_server[23441]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on > Apr 9 2021 at 15:49:55) > May 23 00:18:52 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 23 00:18:52 cache0302 traffic_server[23441]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 23 00:18:52 cache0302 traffic_manager[21310]: [May 23 00:18:52.120] > traffic_crashlo NOTE: crashlog started, target=23441, debug=false > syslog=true, uid=114 euid=0 > May 23 00:18:52 cache0302 traffic_manager[21310]: [May 23 00:18:52.120] > traffic_crashlo NOTE: logging to 0x12c5d40cced0 > May 23 00:18:52 cache0302 traffic_crashlog[23443]: traffic_crashlo NOTE: > crashlog started, target=23441, debug=false syslog=true, uid=114 euid=0 > May 23 00:18:52 cache0302 traffic_crashlog[23443]: traffic_crashlo NOTE: > logging to 0x12c5d40cced0 > May 23 00:18:53 cache0302 traffic_manager[21310]: [May 23 00:18:53.113] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-23-001852.log > May 23 00:18:53 cache0302 traffic_crashlog[23443]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-23-001852.log > May 23 00:18:53 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 23 00:18:53 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1b7f44513ef0] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7244e5c04980] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7244e4cfcfb7] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7244e4cfe921] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x7244e7899b08] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x7244e78967d5] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1b7f4458bd0b] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x1b7f447bfd32] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1b7f447c1329] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x418)[0x1b7f447b3f18] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1b7f447fff92] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1b7f44801233] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1b7f447ff91a] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7244e5bf96db] > May 23 00:18:53 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7244e4ddf71f] > May 23 00:18:56 cache0302 traffic_server[26362]: NOTE: --- traffic_server > Starting --- > May 23 00:18:56 cache0302 traffic_server[26362]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on > Apr 9 2021 at 15:49:55) > May 23 00:24:24 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 23 00:24:24 cache0302 traffic_server[26362]: Fatal: HttpSM.cc:2533: > failed assertion `magic == HTTP_SM_MAGIC_ALIVE` > May 23 00:24:24 cache0302 traffic_manager[21310]: [May 23 00:24:24.239] > traffic_crashlo NOTE: crashlog started, target=26362, debug=false > syslog=true, uid=114 euid=0 > May 23 00:24:24 cache0302 traffic_manager[21310]: [May 23 00:24:24.240] > traffic_crashlo NOTE: logging to 0x190a947ba530 > May 23 00:24:24 cache0302 traffic_crashlog[26365]: traffic_crashlo NOTE: > crashlog started, target=26362, debug=false syslog=true, uid=114 euid=0 > May 23 00:24:24 cache0302 traffic_crashlog[26365]: traffic_crashlo NOTE: > logging to 0x190a947ba530 > May 23 00:24:25 cache0302 traffic_manager[21310]: [May 23 00:24:25.274] > traffic_crashlo ERROR: wrote crash log to > /var/log/trafficserver/crash-2021-05-23-002424.log > May 23 00:24:25 cache0302 traffic_crashlog[26365]: traffic_crashlo ERROR: > wrote crash log to /var/log/trafficserver/crash-2021-05-23-002424.log > May 23 00:24:25 cache0302 traffic_manager[21310]: traffic_server: received > signal 6 (Aborted) > May 23 00:24:25 cache0302 traffic_manager[21310]: traffic_server - STACK > TRACE: > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1cf361726ef0] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6d01e9b4d980] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x6d01e8c45fb7] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x6d01e8c47921] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x6d01eb7e2b08] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x6d01eb7df7d5] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1cf36179ed0b] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x36cd32)[0x1cf3619d2d32] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1cf3619d4329] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1cf3619c524c] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1cf3619c6d72] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1cf361a12f92] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1cf361a14233] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /usr/bin/traffic_server(+0x3ac91a)[0x1cf361a1291a] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6d01e9b426db] > May 23 00:24:25 cache0302 traffic_manager[21310]: > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6d01e8d2871f] > May 23 00:24:28 cache0302 traffic_server[28956]: NOTE: --- traffic_server > Starting --- > May 23 00:24:28 cache0302 traffic_server[28956]: NOTE: traffic_server > Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on > Apr 9 2021 at 15:49:55)
