I am back from a week of vacation. Reply is inline. -Bryan
> On May 27, 2021, at 5:25 PM, Mark Moseley <[email protected]> wrote: > > According to traffic_ctl (which, btw, I love being able to query the current > configs, as well as metrics), it's already set to 'thread'. Is it worth > switching to 'global' or is it only the "global"->"thread" direction that > would help? I would keep it at thread. > > I've never used ASAN before. Do I just need to use the flags you mentioned > (-M -f -F) or do I need to recompile ATS with --enable-asan? I'm looking at > the PPT presentation about it, but otherwise details about ATS+ASAN are a bit > thin in the search results. You will need to compile with —enable-asan and use -M -f -F in records.config. > > I was able to get ATS to compile with --enable-asan, but I had to remove the > libjansson package from the box where I was building. With libjansson > installed (via the default Bionic package), ATS tries to compile the > url_signing module and blows up with "relocation R_X86_64_PC32 against symbol > `stdin@@GLIBC_2.2.5' can not be used when making a shared object; recompile > with -fPIC" (it's complaining about libjansson needing to be recompiled with > -fPIC, not ATS). > > Are there any ill effects from compiling ATS without libjansson (beyond not > building that experimental module that I don't use anyway)? Shouldn’t be. > > (Incidentally, it'd be great to have a --configure flag to selectively avoid > arbitrary experimental modules. I use tslua, so I need to enable experimental > modules, but I would've liked an easy way to tell the build process "don't > build url_signing") > > I'll have to do some testing after the weekend to see if the overhead of ASAN > is too much. I'm running ATS on very nice boxes, so I'm hoping it'll be fine. > If things work well enough to run the ASAN build in production on a box or > two, what info am I looking for? I'm assuming something in diags.log or > manager.log. > > Thanks! > > > On Thu, May 27, 2021 at 9:44 AM Bryan Call <[email protected] > <mailto:[email protected]>> wrote: > Are you using per thread or global session pools? If you are using global, > it might be worth a try to switch to per thread > (https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html?highlight=records.config#proxy-config-http-server-session-sharing-pool > > <https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html?highlight=records.config#proxy-config-http-server-session-sharing-pool>). > > You can try turning off keep alive as referenced in the issue, but I have a > feeling it won’t help. > > If you can run an ASAN in production with the proxy.config.proxy_binary_opts > set to "-M -f -F” that would really help to tracking this issue down. There > is some overhead with running ASAN and I don’t know if your servers can > handle it > (https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html?highlight=records.config#proxy-config-proxy-binary-opts > > <https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html?highlight=records.config#proxy-config-proxy-binary-opts>). > > -Bryan > > >> On May 24, 2021, at 4:12 PM, Mark Moseley <[email protected] >> <mailto:[email protected]>> wrote: >> >> I've been chasing an issue for a while now. This is 8.1.1 running on Ubuntu >> Focal. I'm seeing this same crash (Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE`) >> across different nodes that are handling unrelated traffic, but also ones >> that don't mention HTTP_SM_MAGIC_ALIVE. >> >> Background: >> ATS is doing reverse proxy caching in front of web hosting backends, so a >> completely random assortment of backend domains and a random assortment of >> file types. Customer domains resolve to a group of 3-node clusters. I'm >> seeing the same crash across different clusters, so it wouldn't be related >> to the same backend origin domain(s). Backends are derived via remap+tslua, >> doing DNS lookups. These boxes constantly run at 200-400 reqs/seq all day, >> but I'm only seeing this crash anywhere from "once every couple days" to a >> couple of times a day on a given node (so the other millions of requests per >> day are processed just fine). ATS sits behind a front-end nginx instance (so >> nginx->ats is all localhost; origins are remote but on the same LAN). >> >> It sounds similar to https://github.com/apache/trafficserver/issues/4921 >> <https://github.com/apache/trafficserver/issues/4921> but that ticket also >> says that it was fixed for the OP in 8.1, so that could be a red herring. >> >> Any patches/settings I can use to mitigate this? Would turning off >> keepalives help as a bandaid? (Outbound keepalives to the origins would hurt >> a bit; inbound would probably be fine due to nginx->ats being on localhost). >> >> I'm also fine upgrading to 9.x but I ran into the issue where the connection >> count grows without bound, despite the connections not really being open >> (https://github.com/apache/trafficserver/issues/7471 >> <https://github.com/apache/trafficserver/issues/7471>) >> >> I don't have any core files handy and these boxes have anywhere from >> 64gb-224gb ATS RAM cache, so I imagine a core would be horrifyingly large. >> The error is always similar to this (most of these are from the same box but >> that's just out of convenience; I'm seeing the same variety of segfaults >> across all of my boxes): >> >> May 24 07:05:14 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 24 07:05:14 cache0302 traffic_server[30820]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 24 07:05:14 cache0302 traffic_manager[21310]: [May 24 07:05:14.660] >> traffic_crashlo NOTE: crashlog started, target=30820, debug=false >> syslog=true, uid=114 euid=0 >> May 24 07:05:14 cache0302 traffic_crashlog[30822]: traffic_crashlo NOTE: >> crashlog started, target=30820, debug=false syslog=true, uid=114 euid=0 >> May 24 07:05:14 cache0302 traffic_crashlog[30822]: traffic_crashlo NOTE: >> logging to 0x14e10a33ad20 >> May 24 07:05:14 cache0302 traffic_manager[21310]: [May 24 07:05:14.660] >> traffic_crashlo NOTE: logging to 0x14e10a33ad20 >> May 24 07:05:17 cache0302 traffic_manager[21310]: [May 24 07:05:17.094] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-24-070514.log >> May 24 07:05:17 cache0302 traffic_crashlog[30822]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-24-070514.log >> May 24 07:05:17 cache0302 traffic_manager[21310]: traffic_server: received >> signal 6 (Aborted) >> May 24 07:05:17 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x15ff15ac4ef0] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x675351928980] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x675350a20fb7] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x675350a22921] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x6753535bdb08] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x6753535ba7d5] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x15ff15b3cd0b] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36cd32)[0x15ff15d70d32] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x15ff15d72329] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x15ff15d6324c] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x15ff15d64d72] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x15ff15db0f92] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x15ff15db2233] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x15ff15db091a] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x67535191d6db] >> May 24 07:05:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x675350b0371f] >> >> Sometimes I get this one: >> >> May 22 21:30:01 cache0102 traffic_manager[6292]: Fatal: HttpSM.cc:5854 >> <http://httpsm.cc:5854/>: failed assertion `server_session == nullptr` >> May 22 21:30:01 cache0102 traffic_server[16246]: Fatal: HttpSM.cc:5854 >> <http://httpsm.cc:5854/>: failed assertion `server_session == nullptr` >> May 22 21:30:01 cache0102 traffic_manager[6292]: [May 22 21:30:01.658] >> traffic_crashlo NOTE: crashlog started, target=16246, debug=false >> syslog=true, uid=114 euid=0 >> May 22 21:30:01 cache0102 traffic_crashlog[16260]: traffic_crashlo NOTE: >> crashlog started, target=16246, debug=false syslog=true, uid=114 euid=0 >> May 22 21:30:01 cache0102 traffic_crashlog[16260]: traffic_crashlo NOTE: >> logging to 0x18473c4cb6e0 >> May 22 21:30:01 cache0102 traffic_manager[6292]: [May 22 21:30:01.659] >> traffic_crashlo NOTE: logging to 0x18473c4cb6e0 >> May 22 21:30:04 cache0102 traffic_manager[6292]: [May 22 21:30:04.197] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-22-213001.log >> May 22 21:30:04 cache0102 traffic_crashlog[16260]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-22-213001.log >> May 22 21:30:04 cache0102 traffic_manager[6292]: traffic_server: received >> signal 6 (Aborted) >> May 22 21:30:04 cache0102 traffic_manager[6292]: traffic_server - STACK >> TRACE: >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x10b76b432ef0] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x62a740d4b980] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x62a73fe43fb7] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x62a73fe45921] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x62a7429e0b08] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x62a7429dd7d5] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(_ZN6HttpSM17get_http_scheduleEiPv+0x0)[0x10b76b4a2250] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(_ZN6HttpSM22state_http_server_openEiPv+0x2e2)[0x10b76b4af562] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x10b76b4aade0] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(_ZN12CheckConnect14handle_connectEiP5Event+0x649)[0x10b76b6dc8e9] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(+0x36cd32)[0x10b76b6ded32] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0xd30)[0x10b76b6e77c0] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x27f)[0x10b76b6cec0f] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x10b76b6ceef5] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x10b76b720409] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /usr/bin/traffic_server(+0x3ac91a)[0x10b76b71e91a] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x62a740d406db] >> May 22 21:30:04 cache0102 traffic_manager[6292]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x62a73ff2671f] >> >> >> I also see: >> >> May 24 02:05:18 cache0302 traffic_manager[21310]: [May 24 02:05:18.715] >> traffic_crashlo NOTE: crashlog started, target=28956, debug=false >> syslog=true, uid=114 euid=0 >> May 24 02:05:18 cache0302 traffic_manager[21310]: [May 24 02:05:18.715] >> traffic_crashlo NOTE: logging to 0x1a09c40167a0 >> May 24 02:05:18 cache0302 traffic_crashlog[28962]: traffic_crashlo NOTE: >> crashlog started, target=28956, debug=false syslog=true, uid=114 euid=0 >> May 24 02:05:18 cache0302 traffic_crashlog[28962]: traffic_crashlo NOTE: >> logging to 0x1a09c40167a0 >> May 24 02:05:21 cache0302 traffic_manager[21310]: [May 24 02:05:21.582] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-24-020518.log >> May 24 02:05:21 cache0302 traffic_crashlog[28962]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-24-020518.log >> May 24 02:05:21 cache0302 traffic_manager[21310]: traffic_server: received >> signal 11 (Segmentation fault) >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> traffic_servertraffic_server - STACK TRACE: >> May 24 02:05:21 cache0302 traffic_manager[21310]: - STACK TRACE: >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x14d500a4fef0] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x733afa434980] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM21setup_server_transferEv+0x620)[0x14d500abe110] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7OpenDir11close_writeEP7CacheVC+0x1b8)[0x14d500c20d18] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x225)[0x14d500ac4095] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7CacheVC17openWriteCloseDirEiP5Event+0xab)[0x14d500c4ae9b] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x604)[0x14d500ac4a24] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN11HttpCacheSM8end_bothEv+0xa2)[0x14d500ace812] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xe37)[0x14d500acbf47] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM9kill_thisEv+0x83)[0x14d500ac7463] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x4f)[0x14d500ab4bbf] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x17f)[0x14d500ac7e4f] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xba)[0x14d500ac3f2a] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36cd32)[0x14d500cfbd32] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x604)[0x14d500ac4a24] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x14d500cfd329] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM33state_read_server_response_headerEiPv+0x46d)[0x14d500ac594d] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x14d500cee24c] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x14d500ac7de0] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x14d500cefd72] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x52)[0x14d500d00242] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x14d500d3bf92] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x17b5)[0x14d500cc8ec5] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x14d500d3d233] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x264)[0x14d500cebbf4] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x14d500d3b91a] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x14d500cebef5] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x733afa4296db] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x14d500d3d409] >> May 24 02:05:21 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x733af960f71f] >> >> >> Sometimes it takes a day or two to crash, but sometimes it's much more >> rapid. Here's a bunch from the span of a couple of hours: >> >> May 22 22:34:33 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 22 22:34:33 cache0302 traffic_server[22524]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 22 22:34:33 cache0302 traffic_manager[21310]: [May 22 22:34:33.988] >> traffic_crashlo NOTE: crashlog started, target=22524, debug=false >> syslog=true, uid=114 euid=0 >> May 22 22:34:33 cache0302 traffic_crashlog[22527]: traffic_crashlo NOTE: >> crashlog started, target=22524, debug=false syslog=true, uid=114 euid=0 >> May 22 22:34:33 cache0302 traffic_manager[21310]: [May 22 22:34:33.989] >> traffic_crashlo NOTE: logging to 0x10ba7153b000 >> May 22 22:34:33 cache0302 traffic_crashlog[22527]: traffic_crashlo NOTE: >> logging to 0x10ba7153b000 >> May 22 22:34:36 cache0302 traffic_manager[21310]: [May 22 22:34:36.755] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-22-223433.log >> May 22 22:34:36 cache0302 traffic_crashlog[22527]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-22-223433.log >> May 22 22:34:36 cache0302 traffic_manager[21310]: traffic_server: received >> signal 6 (Aborted) >> May 22 22:34:36 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1bedf9b81ef0] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x75b896f0f980] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x75b896007fb7] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x75b896009921] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x75b898ba4b08] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x75b898ba17d5] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1bedf9bf9d0b] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36cd32)[0x1bedf9e2dd32] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1bedf9e2f329] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x418)[0x1bedf9e21f18] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1bedf9e6df92] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1bedf9e6f233] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x1bedf9e6d91a] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x75b896f046db] >> May 22 22:34:36 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x75b8960ea71f] >> May 22 22:34:46 cache0302 traffic_server[8819]: NOTE: --- traffic_server >> Starting --- >> May 22 22:34:46 cache0302 traffic_server[8819]: NOTE: traffic_server >> Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on >> Apr 9 2021 at 15:49:55) >> May 22 22:41:09 cache0302 traffic_manager[21310]: [May 22 22:41:09.044] >> traffic_crashlo NOTE: crashlog started, target=8819, debug=false >> syslog=true, uid=114 euid=0 >> May 22 22:41:09 cache0302 traffic_manager[21310]: [May 22 22:41:09.044] >> traffic_crashlo NOTE: logging to 0x13fb55f0e020 >> May 22 22:41:09 cache0302 traffic_crashlog[8825]: traffic_crashlo NOTE: >> crashlog started, target=8819, debug=false syslog=true, uid=114 euid=0 >> May 22 22:41:09 cache0302 traffic_crashlog[8825]: traffic_crashlo NOTE: >> logging to 0x13fb55f0e020 >> May 22 22:41:10 cache0302 traffic_manager[21310]: [May 22 22:41:10.463] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-22-224109.log >> May 22 22:41:10 cache0302 traffic_crashlog[8825]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-22-224109.log >> May 22 22:41:10 cache0302 traffic_manager[21310]: traffic_server: received >> signal 11 (Segmentation fault) >> May 22 22:41:10 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x191e98ab9ef0] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x67dee0e2f980] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x191e98b88567] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x191e98b1888c] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x191e98b84ffd] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x191e98b85b5a] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36d222)[0x191e98d66222] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x191e98d672cf] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x191e98d59f92] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x191e98da5f92] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x191e98da7233] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x191e98da591a] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x67dee0e246db] >> May 22 22:41:10 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x67dee000a71f] >> May 22 22:41:14 cache0302 traffic_server[11594]: NOTE: --- traffic_server >> Starting --- >> May 22 22:41:14 cache0302 traffic_server[11594]: NOTE: traffic_server >> Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on >> Apr 9 2021 at 15:49:55) >> May 22 23:04:37 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 22 23:04:37 cache0302 traffic_server[11594]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 22 23:04:37 cache0302 traffic_manager[21310]: [May 22 23:04:37.520] >> traffic_crashlo NOTE: crashlog started, target=11594, debug=false >> syslog=true, uid=114 euid=0 >> May 22 23:04:37 cache0302 traffic_crashlog[11600]: traffic_crashlo NOTE: >> crashlog started, target=11594, debug=false syslog=true, uid=114 euid=0 >> May 22 23:04:37 cache0302 traffic_crashlog[11600]: traffic_crashlo NOTE: >> logging to 0x1decbe248a90 >> May 22 23:04:37 cache0302 traffic_manager[21310]: [May 22 23:04:37.520] >> traffic_crashlo NOTE: logging to 0x1decbe248a90 >> May 22 23:04:38 cache0302 traffic_manager[21310]: [May 22 23:04:38.880] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-22-230437.log >> May 22 23:04:38 cache0302 traffic_crashlog[11600]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-22-230437.log >> May 22 23:04:38 cache0302 traffic_manager[21310]: traffic_server: received >> signal 6 (Aborted) >> May 22 23:04:38 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1867ffb06ef0] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x747f2bd26980] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x747f2ae1efb7] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x747f2ae20921] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x747f2d9bbb08] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x747f2d9b87d5] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1867ffb7ed0b] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36cd32)[0x1867ffdb2d32] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1867ffdb4329] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1867ffda524c] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1867ffda6d72] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1867ffdf2f92] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1867ffdf4233] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x1867ffdf291a] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x747f2bd1b6db] >> May 22 23:04:38 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x747f2af0171f] >> May 22 23:04:43 cache0302 traffic_server[23705]: NOTE: --- traffic_server >> Starting --- >> May 22 23:04:43 cache0302 traffic_server[23705]: NOTE: traffic_server >> Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on >> Apr 9 2021 at 15:49:55) >> May 22 23:29:41 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:5854 >> <http://httpsm.cc:5854/>: failed assertion `server_session == nullptr` >> May 22 23:29:41 cache0302 traffic_server[23705]: Fatal: HttpSM.cc:5854 >> <http://httpsm.cc:5854/>: failed assertion `server_session == nullptr` >> May 22 23:29:41 cache0302 traffic_manager[21310]: [May 22 23:29:41.142] >> traffic_crashlo NOTE: crashlog started, target=23705, debug=false >> syslog=true, uid=114 euid=0 >> May 22 23:29:41 cache0302 traffic_crashlog[23707]: traffic_crashlo NOTE: >> crashlog started, target=23705, debug=false syslog=true, uid=114 euid=0 >> May 22 23:29:41 cache0302 traffic_crashlog[23707]: traffic_crashlo NOTE: >> logging to 0x19db57d47390 >> May 22 23:29:41 cache0302 traffic_manager[21310]: [May 22 23:29:41.142] >> traffic_crashlo NOTE: logging to 0x19db57d47390 >> May 22 23:29:42 cache0302 traffic_manager[21310]: [May 22 23:29:42.544] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-22-232941.log >> May 22 23:29:42 cache0302 traffic_crashlog[23707]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-22-232941.log >> May 22 23:29:42 cache0302 traffic_manager[21310]: traffic_server: received >> signal 6 (Aborted) >> May 22 23:29:42 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x147b5d156ef0] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x71f05ed35980] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x71f05de2dfb7] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x71f05de2f921] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x71f0609cab08] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x71f0609c77d5] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM17get_http_scheduleEiPv+0x0)[0x147b5d1c6250] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM22state_http_server_openEiPv+0x2e2)[0x147b5d1d3562] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x110)[0x147b5d1cede0] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN12CheckConnect14handle_connectEiP5Event+0x649)[0x147b5d4008e9] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36cd32)[0x147b5d402d32] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0xd30)[0x147b5d40b7c0] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x27f)[0x147b5d3f2c0f] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x295)[0x147b5d3f2ef5] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x729)[0x147b5d444409] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x147b5d44291a] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x71f05ed2a6db] >> May 22 23:29:42 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x71f05df1071f] >> May 22 23:29:46 cache0302 traffic_server[2493]: NOTE: --- traffic_server >> Starting --- >> May 22 23:29:46 cache0302 traffic_server[2493]: NOTE: traffic_server >> Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on >> Apr 9 2021 at 15:49:55) >> May 22 23:43:15 cache0302 traffic_manager[21310]: [May 22 23:43:15.997] >> traffic_crashlo NOTE: crashlog started, target=2493, debug=false >> syslog=true, uid=114 euid=0 >> May 22 23:43:15 cache0302 traffic_crashlog[2495]: traffic_crashlo NOTE: >> crashlog started, target=2493, debug=false syslog=true, uid=114 euid=0 >> May 22 23:43:15 cache0302 traffic_manager[21310]: [May 22 23:43:15.997] >> traffic_crashlo NOTE: logging to 0x16dba7520110 >> May 22 23:43:15 cache0302 traffic_crashlog[2495]: traffic_crashlo NOTE: >> logging to 0x16dba7520110 >> May 22 23:43:17 cache0302 traffic_manager[21310]: [May 22 23:43:17.331] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-22-234315.log >> May 22 23:43:17 cache0302 traffic_crashlog[2495]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-22-234315.log >> May 22 23:43:17 cache0302 traffic_manager[21310]: traffic_server: received >> signal 11 (Segmentation fault) >> May 22 23:43:17 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1e354ceb6ef0] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6ac17bd5b980] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x1e354cf85567] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x1e354cf1588c] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x1e354cf81ffd] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x1e354cf82b5a] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36d222)[0x1e354d163222] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x1e354d1642cf] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x1e354d156f92] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1e354d1a2f92] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1e354d1a4233] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x1e354d1a291a] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6ac17bd506db] >> May 22 23:43:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6ac17af3671f] >> May 22 23:43:21 cache0302 traffic_server[9064]: NOTE: --- traffic_server >> Starting --- >> May 22 23:43:21 cache0302 traffic_server[9064]: NOTE: traffic_server >> Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on >> Apr 9 2021 at 15:49:55) >> May 22 23:51:29 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 22 23:51:29 cache0302 traffic_server[9064]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 22 23:51:29 cache0302 traffic_manager[21310]: [May 22 23:51:29.199] >> traffic_crashlo NOTE: crashlog started, target=9064, debug=false >> syslog=true, uid=114 euid=0 >> May 22 23:51:29 cache0302 traffic_crashlog[9066]: traffic_crashlo NOTE: >> crashlog started, target=9064, debug=false syslog=true, uid=114 euid=0 >> May 22 23:51:29 cache0302 traffic_crashlog[9066]: traffic_crashlo NOTE: >> logging to 0x153045b30140 >> May 22 23:51:29 cache0302 traffic_manager[21310]: [May 22 23:51:29.199] >> traffic_crashlo NOTE: logging to 0x153045b30140 >> May 22 23:51:30 cache0302 traffic_manager[21310]: [May 22 23:51:30.396] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-22-235129.log >> May 22 23:51:30 cache0302 traffic_crashlog[9066]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-22-235129.log >> May 22 23:51:30 cache0302 traffic_manager[21310]: traffic_server: received >> signal 6 (Aborted) >> May 22 23:51:30 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1cfb65150ef0] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x76d5135c1980] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x76d5126b9fb7] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x76d5126bb921] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x76d515256b08] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x76d5152537d5] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1cfb651c8d0b] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36cd32)[0x1cfb653fcd32] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1cfb653fe329] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1cfb653ef24c] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1cfb653f0d72] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1cfb6543cf92] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1cfb6543e233] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x1cfb6543c91a] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x76d5135b66db] >> May 22 23:51:30 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x76d51279c71f] >> May 22 23:51:33 cache0302 traffic_server[12951]: NOTE: --- traffic_server >> Starting --- >> May 22 23:51:33 cache0302 traffic_server[12951]: NOTE: traffic_server >> Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on >> Apr 9 2021 at 15:49:55) >> May 23 00:08:16 cache0302 traffic_manager[21310]: [May 23 00:08:16.507] >> traffic_crashlo NOTE: crashlog started, target=12951, debug=false >> syslog=true, uid=114 euid=0 >> May 23 00:08:16 cache0302 traffic_manager[21310]: [May 23 00:08:16.507] >> traffic_crashlo NOTE: logging to 0x19c404cfdad0 >> May 23 00:08:16 cache0302 traffic_crashlog[12953]: traffic_crashlo NOTE: >> crashlog started, target=12951, debug=false syslog=true, uid=114 euid=0 >> May 23 00:08:16 cache0302 traffic_crashlog[12953]: traffic_crashlo NOTE: >> logging to 0x19c404cfdad0 >> May 23 00:08:17 cache0302 traffic_manager[21310]: [May 23 00:08:17.776] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-23-000816.log >> May 23 00:08:17 cache0302 traffic_crashlog[12953]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-23-000816.log >> May 23 00:08:17 cache0302 traffic_manager[21310]: traffic_server: received >> signal 11 (Segmentation fault) >> May 23 00:08:17 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x19c0a1524ef0] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6d52207fd980] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7OpenDir11close_writeEP7CacheVC+0x1b8)[0x19c0a16f5d18] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7CacheVC17openWriteCloseDirEiP5Event+0xab)[0x19c0a171fe9b] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN11HttpCacheSM8end_bothEv+0xa2)[0x19c0a15a3812] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM9kill_thisEv+0x83)[0x19c0a159c463] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x17f)[0x19c0a159ce4f] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36cd32)[0x19c0a17d0d32] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x19c0a17d2329] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x19c0a17c324c] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x19c0a17c4d72] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x19c0a1810f92] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x19c0a1812233] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x19c0a181091a] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6d52207f26db] >> May 23 00:08:17 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6d521f9d871f] >> May 23 00:08:21 cache0302 traffic_server[20789]: NOTE: --- traffic_server >> Starting --- >> May 23 00:08:21 cache0302 traffic_server[20789]: NOTE: traffic_server >> Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on >> Apr 9 2021 at 15:49:55) >> May 23 00:14:46 cache0302 traffic_manager[21310]: [May 23 00:14:46.669] >> traffic_crashlo NOTE: crashlog started, target=20789, debug=false >> syslog=true, uid=114 euid=0 >> May 23 00:14:46 cache0302 traffic_crashlog[20796]: traffic_crashlo NOTE: >> crashlog started, target=20789, debug=false syslog=true, uid=114 euid=0 >> May 23 00:14:46 cache0302 traffic_manager[21310]: [May 23 00:14:46.669] >> traffic_crashlo NOTE: logging to 0x14f170d7b660 >> May 23 00:14:46 cache0302 traffic_crashlog[20796]: traffic_crashlo NOTE: >> logging to 0x14f170d7b660 >> May 23 00:14:47 cache0302 traffic_manager[21310]: [May 23 00:14:47.825] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-23-001446.log >> May 23 00:14:47 cache0302 traffic_crashlog[20796]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-23-001446.log >> May 23 00:14:47 cache0302 traffic_manager[21310]: traffic_server: received >> signal 11 (Segmentation fault) >> May 23 00:14:47 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x18c9c6353ef0] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x69e99706b980] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10HttpTunnel19finish_all_internalEP18HttpTunnelProducerb+0x177)[0x18c9c6422567] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM21tunnel_handler_serverEiP18HttpTunnelProducer+0x60c)[0x18c9c63b288c] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10HttpTunnel16producer_handlerEiP18HttpTunnelProducer+0x58d)[0x18c9c641effd] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0x14a)[0x18c9c641fb5a] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36d222)[0x18c9c6600222] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x56f)[0x18c9c66012cf] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x492)[0x18c9c65f3f92] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x18c9c663ff92] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x18c9c6641233] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x18c9c663f91a] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x69e9970606db] >> May 23 00:14:47 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x69e99624671f] >> May 23 00:14:51 cache0302 traffic_server[23441]: NOTE: --- traffic_server >> Starting --- >> May 23 00:14:51 cache0302 traffic_server[23441]: NOTE: traffic_server >> Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on >> Apr 9 2021 at 15:49:55) >> May 23 00:18:52 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 23 00:18:52 cache0302 traffic_server[23441]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 23 00:18:52 cache0302 traffic_manager[21310]: [May 23 00:18:52.120] >> traffic_crashlo NOTE: crashlog started, target=23441, debug=false >> syslog=true, uid=114 euid=0 >> May 23 00:18:52 cache0302 traffic_manager[21310]: [May 23 00:18:52.120] >> traffic_crashlo NOTE: logging to 0x12c5d40cced0 >> May 23 00:18:52 cache0302 traffic_crashlog[23443]: traffic_crashlo NOTE: >> crashlog started, target=23441, debug=false syslog=true, uid=114 euid=0 >> May 23 00:18:52 cache0302 traffic_crashlog[23443]: traffic_crashlo NOTE: >> logging to 0x12c5d40cced0 >> May 23 00:18:53 cache0302 traffic_manager[21310]: [May 23 00:18:53.113] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-23-001852.log >> May 23 00:18:53 cache0302 traffic_crashlog[23443]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-23-001852.log >> May 23 00:18:53 cache0302 traffic_manager[21310]: traffic_server: received >> signal 6 (Aborted) >> May 23 00:18:53 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1b7f44513ef0] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7244e5c04980] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7244e4cfcfb7] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7244e4cfe921] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x7244e7899b08] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x7244e78967d5] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1b7f4458bd0b] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36cd32)[0x1b7f447bfd32] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1b7f447c1329] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x418)[0x1b7f447b3f18] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1b7f447fff92] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1b7f44801233] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x1b7f447ff91a] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7244e5bf96db] >> May 23 00:18:53 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7244e4ddf71f] >> May 23 00:18:56 cache0302 traffic_server[26362]: NOTE: --- traffic_server >> Starting --- >> May 23 00:18:56 cache0302 traffic_server[26362]: NOTE: traffic_server >> Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on >> Apr 9 2021 at 15:49:55) >> May 23 00:24:24 cache0302 traffic_manager[21310]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 23 00:24:24 cache0302 traffic_server[26362]: Fatal: HttpSM.cc:2533 >> <http://httpsm.cc:2533/>: failed assertion `magic == HTTP_SM_MAGIC_ALIVE` >> May 23 00:24:24 cache0302 traffic_manager[21310]: [May 23 00:24:24.239] >> traffic_crashlo NOTE: crashlog started, target=26362, debug=false >> syslog=true, uid=114 euid=0 >> May 23 00:24:24 cache0302 traffic_manager[21310]: [May 23 00:24:24.240] >> traffic_crashlo NOTE: logging to 0x190a947ba530 >> May 23 00:24:24 cache0302 traffic_crashlog[26365]: traffic_crashlo NOTE: >> crashlog started, target=26362, debug=false syslog=true, uid=114 euid=0 >> May 23 00:24:24 cache0302 traffic_crashlog[26365]: traffic_crashlo NOTE: >> logging to 0x190a947ba530 >> May 23 00:24:25 cache0302 traffic_manager[21310]: [May 23 00:24:25.274] >> traffic_crashlo ERROR: wrote crash log to >> /var/log/trafficserver/crash-2021-05-23-002424.log >> May 23 00:24:25 cache0302 traffic_crashlog[26365]: traffic_crashlo ERROR: >> wrote crash log to /var/log/trafficserver/crash-2021-05-23-002424.log >> May 23 00:24:25 cache0302 traffic_manager[21310]: traffic_server: received >> signal 6 (Aborted) >> May 23 00:24:25 cache0302 traffic_manager[21310]: traffic_server - STACK >> TRACE: >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xa0)[0x1cf361726ef0] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x6d01e9b4d980] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x6d01e8c45fb7] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x6d01e8c47921] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x52b08)[0x6d01eb7e2b08] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/lib/trafficserver/libtscore.so.8(+0x4f7d5)[0x6d01eb7df7d5] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x3b)[0x1cf36179ed0b] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x36cd32)[0x1cf3619d2d32] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN18UnixNetVConnection9mainEventEiP5Event+0x5c9)[0x1cf3619d4329] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN10NetHandler19manage_active_queueEb+0x4ac)[0x1cf3619c524c] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN13InactivityCop16check_inactivityEiP5Event+0x272)[0x1cf3619c6d72] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x92)[0x1cf361a12f92] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(_ZN7EThread15execute_regularEv+0x553)[0x1cf361a14233] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /usr/bin/traffic_server(+0x3ac91a)[0x1cf361a1291a] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x6d01e9b426db] >> May 23 00:24:25 cache0302 traffic_manager[21310]: >> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x6d01e8d2871f] >> May 23 00:24:28 cache0302 traffic_server[28956]: NOTE: --- traffic_server >> Starting --- >> May 23 00:24:28 cache0302 traffic_server[28956]: NOTE: traffic_server >> Version: Apache Traffic Server - traffic_server - 8.1.1 - (build # 040915 on >> Apr 9 2021 at 15:49:55) >
