[
https://issues.apache.org/jira/browse/TS-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776453#comment-13776453
]
David Carlin commented on TS-1411:
----------------------------------
Our hosts were upgraded to ats-3.3.5 and now I am reliably getting backtraces
along with the stack trace in traffic.out
I have seen the following crash 17 times in just the last day:
{noformat}
#0 LogAccess::marshal_mem (
dest=0x2b1230214b48
"nts-min.css&d/lib/yui/3.4.1/build/cssgrids/cssgrids-min.css&os/mit/media/p/content/grids/v2-master-min-1335630.css&os/mit/media/p/content/grids/v2-smartphone-min-1490385.css&os/mit/media/p/content/oth"...,
source=0x2b12b81e1b9e <Address 0x2b12b81e1b9e out of bounds>, actual_len=1930,
padded_len=1936) at LogAccess.cc:805
#1 0x000000000058a440 in LogAccessHttp::marshal_client_req_url_canon
(this=<value optimized out>, buf=<value optimized out>) at LogAccessHttp.cc:304
#2 0x000000000059b422 in LogFieldList::marshal (this=<value optimized out>,
lad=0x2b1188604a90, buf=0x2b1230214b10 "") at LogField.cc:517
#3 0x00000000005a5ce5 in LogObject::log (this=0x18254b0, lad=0x2b1188604a90,
text_entry=0x0) at LogObject.cc:609
#4 0x000000000058d586 in log (lad=0x2b1188604a90) at LogObject.h:394
#5 Log::access (lad=0x2b1188604a90) at Log.cc:1110
#6 0x00000000005234c8 in HttpSM::update_stats (this=0x2b14df6d8f70) at
HttpSM.cc:6484
#7 0x0000000000530e10 in HttpSM::kill_this (this=0x2b14df6d8f70) at
HttpSM.cc:6445
#8 0x0000000000531198 in HttpSM::main_handler (this=0x2b14df6d8f70,
event=2301, data=0x2b14df6dab30) at HttpSM.cc:2516
#9 0x000000000056b3f7 in handleEvent (this=0x2b14df6dab30, event=<value
optimized out>, data=<value optimized out>)
at ../../iocore/eventsystem/I_Continuation.h:146
#10 HttpTunnel::main_handler (this=0x2b14df6dab30, event=<value optimized out>,
data=<value optimized out>) at HttpTunnel.cc:1506
#11 0x0000000000680f81 in handleEvent (event=<value optimized out>,
nh=0x2b1182d1ebf0, vc=0x2b11d81deba0) at
../../iocore/eventsystem/I_Continuation.h:146
#12 write_signal_and_update (event=<value optimized out>, nh=0x2b1182d1ebf0,
vc=0x2b11d81deba0) at UnixNetVConnection.cc:153
#13 write_signal_done (event=<value optimized out>, nh=0x2b1182d1ebf0,
vc=0x2b11d81deba0) at UnixNetVConnection.cc:180
#14 0x0000000000683db7 in write_to_net_io (nh=0x2b1182d1ebf0,
vc=0x2b11d81deba0, thread=<value optimized out>) at UnixNetVConnection.cc:479
#15 0x000000000067b5e3 in NetHandler::mainNetEvent (this=0x2b1182d1ebf0,
event=<value optimized out>, e=<value optimized out>) at UnixNet.cc:392
#16 0x00000000006a2f2f in handleEvent (this=0x2b1182d1b010, e=0x13210f0,
calling_code=5) at I_Continuation.h:146
#17 EThread::process_event (this=0x2b1182d1b010, e=0x13210f0, calling_code=5)
at UnixEThread.cc:141
#18 0x00000000006a3913 in EThread::execute (this=0x2b1182d1b010) at
UnixEThread.cc:265
#19 0x00000000006a1dca in spawn_thread_internal (a=0x1252660) at Thread.cc:88
#20 0x0000003d378077f1 in start_thread () from /lib64/libpthread.so.0
#21 0x0000003d374e5ccd in clone () from /lib64/libc.so.6
NOTE: Traffic Server received Sig 11: Segmentation fault
/home/y/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0[0x3d3780f4a0]
/home/y/bin/traffic_server(_ZN9LogAccess11marshal_memEPcPKcii+0x48)[0x588228]
/home/y/bin/traffic_server(_ZN13LogAccessHttp28marshal_client_req_url_canonEPc+0x20)[0x58a440]
/home/y/bin/traffic_server(_ZN12LogFieldList7marshalEP9LogAccessPc+0x32)[0x59b422]
/home/y/bin/traffic_server(_ZN9LogObject3logEP9LogAccessPc+0x375)[0x5a5ce5]
/home/y/bin/traffic_server(_ZN3Log6accessEP9LogAccess+0x146)[0x58d586]
/home/y/bin/traffic_server(_ZN6HttpSM12update_statsEv+0x5e8)[0x5234c8]
/home/y/bin/traffic_server(_ZN6HttpSM9kill_thisEv+0x7a0)[0x530e10]
/home/y/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x188)[0x531198]
/home/y/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0xd7)[0x56b3f7]
/home/y/bin/traffic_server[0x680f81]
/home/y/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0x917)[0x683db7]
/home/y/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x283)[0x67b5e3]
/home/y/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6a2f2f]
/home/y/bin/traffic_server(_ZN7EThread7executeEv+0x4a3)[0x6a3913]
/home/y/bin/traffic_server[0x6a1dca]
/lib64/libpthread.so.0[0x3d378077f1]
/lib64/libc.so.6(clone+0x6d)[0x3d374e5ccd]
{noformat}
> Seg faulting during log rotation
> --------------------------------
>
> Key: TS-1411
> URL: https://issues.apache.org/jira/browse/TS-1411
> Project: Traffic Server
> Issue Type: Bug
> Components: Logging
> Affects Versions: 3.2.0
> Environment: RHEL 6.2 x86_64
> Reporter: David Carlin
> Assignee: Yunkai Zhang
> Fix For: 4.2.0
>
> Attachments: Log rotation segaults.txt, TS-1411 backtraces.txt
>
>
> I've been experiencing some segfaults during log rotation. The sequence of
> events is this.. log rotation occurs, then I get hundreds of dropping log
> buffer error msgs, then the segfault.
> This started occurring when I lengthened the default log format to include
> the unmapped URL and the user agent string:
> %<cqtq> %<ttms> %<chi> %<crc>/%<pssc> %<psql> %<cqhm> %<cquc> %<caun>
> %<phr>/%<pqsn> %<psct> %<xid> %<cquuc> \"%<{User-Agent}cqh>\"
> In terms of frequency, we have a number of boxes and I probably see one of
> these crashed per day since the above change. Logs are rotated every 2 hours.
> I've had other log related segfaults, reported in TS-1330 - these new ones
> seem to have a different cause.
> [Aug 14 21:07:20.002] Server {0x2ae3a8887700} STATUS: The rolled logfile,
> /home/y/logs/trafficserver/error.log_l30.ycs.a4e.yahoo.com.20120814.17h59m50s-20120814.20h00m00s.old,
> was auto-deleted; 3148252 bytes were reclaimed.
> [Aug 14 21:07:42.859] Server {0x2ae3a8887700} STATUS: The rolled logfile,
> /home/y/logs/trafficserver/squid.blog_l30.ycs.a4e.yahoo.com.20120814.18h00m00s-20120814.20h00m00s.old,
> was auto-deleted; 14735520048 bytes were reclaimed.
> [Aug 14 21:07:42.865] Server {0x2ae3a8887700} WARNING: Dropping log buffer,
> can't keep up.
> [Aug 14 21:07:42.865] Server {0x2ae3a8887700} WARNING: Dropping log buffer,
> can't keep up.
> [Aug 14 21:07:42.865] Server {0x2ae3a8887700} WARNING: Dropping log buffer,
> can't keep up.
> [Aug 14 21:07:42.865] Server {0x2ae3a8887700} WARNING: Dropping log buffer,
> can't keep up.
> [Aug 14 21:07:42.865] Server {0x2ae3a8887700} WARNING: Dropping log buffer,
> can't keep up.
> [...]
> [Aug 14 21:07:42.876] Server {0x2ae3a8887700} WARNING: Dropping log buffer,
> can't keep up.
> [Aug 14 21:07:42.876] Server {0x2ae3a8887700} WARNING: Dropping log buffer,
> can't keep up.
> [Aug 14 21:07:42.876] Server {0x2ae3a8887700} WARNING: Dropping log buffer,
> can't keep up.
> [Aug 14 21:07:42.876] Server {0x2ae3a8887700} WARNING: Dropping log buffer,
> can't keep up.
> NOTE: Traffic Server received Sig 11: Segmentation fault
> /home/y/bin/traffic_server - STACK TRACE:
> /lib64/libpthread.so.0[0x383f00f500]
> /home/y/bin/traffic_server(_ZN9LogAccess11marshal_memEPcPKcii+0x48)[0x58a118]
> /home/y/bin/traffic_server(_ZN13LogAccessHttp28marshal_client_req_url_canonEPc+0x20)[0x58c3f0]
> /home/y/bin/traffic_server(_ZN12LogFieldList7marshalEP9LogAccessPc+0x32)[0x59d5a2]
> /home/y/bin/traffic_server(_ZN9LogObject3logEP9LogAccessPc+0x399)[0x5a7ed9]
> /home/y/bin/traffic_server(_ZN3Log6accessEP9LogAccess+0x146)[0x58f506]
> /home/y/bin/traffic_server(_ZN6HttpSM12update_statsEv+0x630)[0x526c50]
> /home/y/bin/traffic_server(_ZN6HttpSM9kill_thisEv+0x928)[0x52b548]
> /home/y/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x198)[0x52b868]
> /home/y/bin/traffic_server(_ZN10HttpTunnel12main_handlerEiPv+0xde)[0x56c3ee]
> /home/y/bin/traffic_server[0x673871]
> /home/y/bin/traffic_server(_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread+0x847)[0x6756e7]
> /home/y/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x286)[0x66e076]
> /home/y/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0xb4)[0x696ce4]
> /home/y/bin/traffic_server(_ZN7EThread7executeEv+0x4c3)[0x697673]
> /home/y/bin/traffic_server[0x695cb2]
> /lib64/libpthread.so.0[0x383f007851]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
