[jira] [Commented] (TS-4206) Stalled connections show a client request but no HTTP response
[
https://issues.apache.org/jira/browse/TS-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424921#comment-15424921
]
Eric Sproul commented on TS-4206:
-
The client was curl.
> Stalled connections show a client request but no HTTP response
> --
>
> Key: TS-4206
> URL: https://issues.apache.org/jira/browse/TS-4206
> Project: Traffic Server
> Issue Type: Bug
> Components: Core, HTTP
>Reporter: Eric Sproul
>Assignee: Bryan Call
>Priority: Blocker
> Labels: regression
> Fix For: 7.0.0
>
>
> Have been discussing this one on IRC but wanted to capture everything here
> since it seems like a fairly serious issue. Since upgrading from 5.3.2 to
> 6.1.1 we have witnessed connections that, from the client perspective, seem
> to stall after the client sends the request. TS logs the connection only
> after it seemingly hits {{proxy.config.net.default_inactivity_timeout}} (5
> minutes), but logs a response code of 000, despite the presence of a request
> (e.g. GET with a request URL logged).
> For the time being we have failed back to 5.3.2 but I was able to capture a
> sample of this situation in the slow log. [This
> paste|http://apaste.info/ds1] shows the slow log as well as the corresponding
> squid.blog entry (default format).
> This issue feels similar to TS-3456 but we are not using tunneling, though we
> are using SSL/TLS in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-4206) Stalled connections show a client request but no HTTP response
[
https://issues.apache.org/jira/browse/TS-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241909#comment-15241909
]
Eric Sproul commented on TS-4206:
-
[~bcall] I think I have reproduced the issue in a dev environment. Here is the
squid log and the slow log for this request (lines wrapped for clarity):
{code}
1460667286.245 900263 x.x.x.x ERROR_UNKNOWN(90)/000 0 GET - EMPTY/- -
{code}
{code}
[Apr 14 20:54:46.245] Server {0x16} ERROR: [49]
Slow Request:
client_ip: x.x.x.x:60916 url:
status: 0 unique id: redirection_tries: 0 bytes: 0 fd: 0
client state: 0 server state: 0 ua_begin: 0.000 ua_first_read: 0.000
ua_read_header_done: -1.000
cache_open_read_begin: -1.000 cache_open_read_end: -1.000
dns_lookup_begin: -1.000 dns_lookup_end: -1.000
server_connect: -1.000 server_first_read: -1.000 server_read_header_done:
-1.000
server_close: -1.000 ua_close: -1.000 sm_finish: 900.264 plugin_active: -1.000
plugin_total: -1.000
{code}
> Stalled connections show a client request but no HTTP response
> --
>
> Key: TS-4206
> URL: https://issues.apache.org/jira/browse/TS-4206
> Project: Traffic Server
> Issue Type: Bug
> Components: Core, HTTP
>Reporter: Eric Sproul
>Assignee: Bryan Call
>Priority: Blocker
> Labels: regression
> Fix For: 6.2.0
>
>
> Have been discussing this one on IRC but wanted to capture everything here
> since it seems like a fairly serious issue. Since upgrading from 5.3.2 to
> 6.1.1 we have witnessed connections that, from the client perspective, seem
> to stall after the client sends the request. TS logs the connection only
> after it seemingly hits {{proxy.config.net.default_inactivity_timeout}} (5
> minutes), but logs a response code of 000, despite the presence of a request
> (e.g. GET with a request URL logged).
> For the time being we have failed back to 5.3.2 but I was able to capture a
> sample of this situation in the slow log. [This
> paste|http://apaste.info/ds1] shows the slow log as well as the corresponding
> squid.blog entry (default format).
> This issue feels similar to TS-3456 but we are not using tunneling, though we
> are using SSL/TLS in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-4206) Stalled connections show a client request but no HTTP response
[
https://issues.apache.org/jira/browse/TS-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239339#comment-15239339
]
Eric Sproul commented on TS-4206:
-
[~zwoop] I confirmed from the package build log that it did not find epoll:
{code}
$ grep epoll build.log
checking for epoll_ctl... no
checking sys/epoll.h usability... no
checking sys/epoll.h presence... no
checking for sys/epoll.h... no
{code}
This was on r151014, which I think pre-dates the epoll commit.
> Stalled connections show a client request but no HTTP response
> --
>
> Key: TS-4206
> URL: https://issues.apache.org/jira/browse/TS-4206
> Project: Traffic Server
> Issue Type: Bug
> Components: Core, HTTP
>Reporter: Eric Sproul
>Assignee: Bryan Call
>Priority: Blocker
> Labels: regression
> Fix For: 6.2.0
>
>
> Have been discussing this one on IRC but wanted to capture everything here
> since it seems like a fairly serious issue. Since upgrading from 5.3.2 to
> 6.1.1 we have witnessed connections that, from the client perspective, seem
> to stall after the client sends the request. TS logs the connection only
> after it seemingly hits {{proxy.config.net.default_inactivity_timeout}} (5
> minutes), but logs a response code of 000, despite the presence of a request
> (e.g. GET with a request URL logged).
> For the time being we have failed back to 5.3.2 but I was able to capture a
> sample of this situation in the slow log. [This
> paste|http://apaste.info/ds1] shows the slow log as well as the corresponding
> squid.blog entry (default format).
> This issue feels similar to TS-3456 but we are not using tunneling, though we
> are using SSL/TLS in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-4206) Stalled connections show a client request but no HTTP response
[
https://issues.apache.org/jira/browse/TS-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229174#comment-15229174
]
Eric Sproul commented on TS-4206:
-
If I had to hazard a guess at what the issue might be, I'd say it was something
specific to an eventer change in 6.x that interacts badly with Solaris/illumos
ports. It seems like we're just losing track of the session and never handling
it. This might explain why we've seen it and (apparently) no one else has, as
there probably aren't many Solaris/illumos ATS users out there.
> Stalled connections show a client request but no HTTP response
> --
>
> Key: TS-4206
> URL: https://issues.apache.org/jira/browse/TS-4206
> Project: Traffic Server
> Issue Type: Bug
> Components: Core, HTTP
>Reporter: Eric Sproul
>Assignee: Bryan Call
>Priority: Blocker
> Labels: regression
> Fix For: 6.2.0
>
>
> Have been discussing this one on IRC but wanted to capture everything here
> since it seems like a fairly serious issue. Since upgrading from 5.3.2 to
> 6.1.1 we have witnessed connections that, from the client perspective, seem
> to stall after the client sends the request. TS logs the connection only
> after it seemingly hits {{proxy.config.net.default_inactivity_timeout}} (5
> minutes), but logs a response code of 000, despite the presence of a request
> (e.g. GET with a request URL logged).
> For the time being we have failed back to 5.3.2 but I was able to capture a
> sample of this situation in the slow log. [This
> paste|http://apaste.info/ds1] shows the slow log as well as the corresponding
> squid.blog entry (default format).
> This issue feels similar to TS-3456 but we are not using tunneling, though we
> are using SSL/TLS in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-4206) Stalled connections show a client request but no HTTP response
[
https://issues.apache.org/jira/browse/TS-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229121#comment-15229121
]
Eric Sproul commented on TS-4206:
-
[~bcall] argh, I thought Pastes Are Forever at apaste.info... :( It will take
me some time to try to recreate this, as we've reverted to 5.3.2 in production,
and I haven't yet been able to reproduce this in dev. Will probably be able to
get to this sometime next week.
> Stalled connections show a client request but no HTTP response
> --
>
> Key: TS-4206
> URL: https://issues.apache.org/jira/browse/TS-4206
> Project: Traffic Server
> Issue Type: Bug
> Components: Core, HTTP
>Reporter: Eric Sproul
>Assignee: Bryan Call
>Priority: Blocker
> Labels: regression
> Fix For: 6.2.0
>
>
> Have been discussing this one on IRC but wanted to capture everything here
> since it seems like a fairly serious issue. Since upgrading from 5.3.2 to
> 6.1.1 we have witnessed connections that, from the client perspective, seem
> to stall after the client sends the request. TS logs the connection only
> after it seemingly hits {{proxy.config.net.default_inactivity_timeout}} (5
> minutes), but logs a response code of 000, despite the presence of a request
> (e.g. GET with a request URL logged).
> For the time being we have failed back to 5.3.2 but I was able to capture a
> sample of this situation in the slow log. [This
> paste|http://apaste.info/ds1] shows the slow log as well as the corresponding
> squid.blog entry (default format).
> This issue feels similar to TS-3456 but we are not using tunneling, though we
> are using SSL/TLS in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TS-4206) Stalled connections show a client request but no HTTP response
Eric Sproul created TS-4206:
---
Summary: Stalled connections show a client request but no HTTP
response
Key: TS-4206
URL: https://issues.apache.org/jira/browse/TS-4206
Project: Traffic Server
Issue Type: Bug
Components: Core, HTTP
Reporter: Eric Sproul
Have been discussing this one on IRC but wanted to capture everything here
since it seems like a fairly serious issue. Since upgrading from 5.3.2 to
6.1.1 we have witnessed connections that, from the client perspective, seem to
stall after the client sends the request. TS logs the connection only after it
seemingly hits {{proxy.config.net.default_inactivity_timeout}} (5 minutes), but
logs a response code of 000, despite the presence of a request (e.g. GET with a
request URL logged).
For the time being we have failed back to 5.3.2 but I was able to capture a
sample of this situation in the slow log. [This paste|http://apaste.info/ds1]
shows the slow log as well as the corresponding squid.blog entry (default
format).
This issue feels similar to TS-3456 but we are not using tunneling, though we
are using SSL/TLS in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TS-4191) Typo of debug tags setting name in Developer Guide
[
https://issues.apache.org/jira/browse/TS-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Sproul updated TS-4191:
Priority: Trivial (was: Major)
> Typo of debug tags setting name in Developer Guide
> --
>
> Key: TS-4191
> URL: https://issues.apache.org/jira/browse/TS-4191
> Project: Traffic Server
> Issue Type: Bug
> Components: Documentation
>Affects Versions: Docs
>Reporter: Eric Sproul
>Priority: Trivial
>
> On the page about Debug Tags, the configuration setting is listed as
> {{proxy.config.diag.debug.tags}} where it should be "diags" (plural).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TS-4191) Typo of debug tags setting name in Developer Guide
Eric Sproul created TS-4191:
---
Summary: Typo of debug tags setting name in Developer Guide
Key: TS-4191
URL: https://issues.apache.org/jira/browse/TS-4191
Project: Traffic Server
Issue Type: Bug
Components: Documentation
Reporter: Eric Sproul
On the page about Debug Tags, the configuration setting is listed as
{{proxy.config.diag.debug.tags}} where it should be "diags" (plural).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TS-4191) Typo of debug tags setting name in Developer Guide
[
https://issues.apache.org/jira/browse/TS-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Sproul updated TS-4191:
Affects Version/s: Docs
> Typo of debug tags setting name in Developer Guide
> --
>
> Key: TS-4191
> URL: https://issues.apache.org/jira/browse/TS-4191
> Project: Traffic Server
> Issue Type: Bug
> Components: Documentation
>Affects Versions: Docs
>Reporter: Eric Sproul
>Priority: Trivial
>
> On the page about Debug Tags, the configuration setting is listed as
> {{proxy.config.diag.debug.tags}} where it should be "diags" (plural).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-4165) Logging breaks if changing log format
[
https://issues.apache.org/jira/browse/TS-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124142#comment-15124142
]
Eric Sproul commented on TS-4165:
-
To reproduce, I established a custom log with some format, ran for a while to
generate some logs, then shut down and changed the format in
{{logs_xml.config}} and upon restart, hit the issue.
> Logging breaks if changing log format
> -
>
> Key: TS-4165
> URL: https://issues.apache.org/jira/browse/TS-4165
> Project: Traffic Server
> Issue Type: Bug
> Components: Logging
>Reporter: Eric Sproul
>Assignee: Daniel Xu
> Labels: regression
> Fix For: 6.2.0
>
>
> When upgrading to 6.1.0 from 5.3.2, I was also changing the format of a
> custom log, and that apparently caused all request logging to stop. In
> syslog I see this error:
> {code}
> {0x1} ERROR: Cannot roll log
> file /var/log/circonus/trafficserver/request.blog to fix log filename
> conflicts
> {code}
> Worked with folks on IRC, and ran under gdb, setting a breakpoint at
> {{BaseLogFile::roll}} and stepping through 3 times subsequently, before
> taking a backtrace:
> {code}
> (gdb) bt
> #0 BaseLogFile::roll (this=0x57f7550, interval_start=,
> interval_end=) at BaseLogFile.cc:119
> #1 0x006ebee0 in LogObjectManager::_solve_filename_conflicts
> (this=this@entry=0x36b9df8, log_object=log_object@entry=0xf17a90,
> maxConflicts=maxConflicts@entry=99) at LogObject.cc:1029
> #2 0x006ec619 in LogObjectManager::_manage_object (this=0x36b9df8,
> log_object=0xf17a90, is_api_object=false, maxConflicts=99)
> at LogObject.cc:894
> #3 0x006db8db in manage_object (maxConflicts=99, logObject=0xf17a90,
> this=0x36b9df8) at LogObject.h:394
> #4 LogConfig::read_xml_log_config (this=this@entry=0x36b9dd0) at
> LogConfig.cc:1468
> #5 0x006dda70 in setup_log_objects (this=0x36b9dd0) at
> LogConfig.cc:503
> #6 LogConfig::init (this=0x36b9dd0, prev_config=0x0) at LogConfig.cc:388
> #7 0x00815bcd in main ()
> (gdb) print *this
> $1 = {m_fp = 0x0, m_start_time = 1454090917, m_end_time = 0, m_bytes_written
> = 0, m_signature = 0, m_has_signature = true,
> m_name = {> = {
> _r = 0x5809950 "/var/log/circonus/trafficserver/request.blog"}, data fields>},
> m_hostname = {> =
> {_r = 0x10fa238 "lbva1"}, }, m_is_regfile = false,
> m_is_init = false, m_meta_info = 0x0}
> {code}
> [~amc] pointed out that {{m_is_regfile}} is false and that {{m_fp}} is null,
> and speculates that ATS might be assuming the file is open when it is not,
> and failing to roll the file. The one case where we would try to roll the
> files without them being open is when the log format is changing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608502#comment-14608502
]
Eric Sproul commented on TS-3486:
-
Our system with the ts-3266-complete diff has been running without a crash for
almost 24 hours. This looks good.
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Susan Hinrichs
Labels: crash
Fix For: 6.1.0
Attachments: ts-3266-2.diff, ts-3266-complete.diff,
ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at HttpSM.cc:1455
#25 0x005ba712 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6876
#26 0x005ba702 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6919
#27 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#28
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14601199#comment-14601199
]
Eric Sproul commented on TS-3486:
-
[~shinrich] Have seen a couple of crashes since deploying. Below is the first.
{noformat}
(gdb) thread 8
[Switching to thread 8 (LWP8)]
#0 0x00800687 in operator IOBufferBlock* (this=0x12058d00) at
../../lib/ts/Ptr.h:300
300 operator T *() const { return (m_ptr); }
(gdb) bt full
#0 0x00800687 in operator IOBufferBlock* (this=0x12058d00) at
../../lib/ts/Ptr.h:300
No locals.
#1 first_write_block (this=0x12058cf0) at
../../iocore/eventsystem/I_IOBuffer.h:926
No locals.
#2 read_from_net (nh=0x165e760, vc=0x1fe56a10, thread=optimized out) at
UnixNetVConnection.cc:275
b = optimized out
buf = @0x1fe56b50: {mbuf = 0x12058cf0, entry = 0x0}
toread = optimized out
total_read = optimized out
r = 0
lock = {m = {m_ptr = 0x1d7b80d0}, lock_acquired = true}
rattempted = optimized out
__FUNCTION__ = read_from_net
s = 0x1fe56b28
mutex = 0xfc95d0
ntodo = 9223372036854775807
niov = optimized out
tiovec = {{iov_base = 0x187ca0d6, iov_len = 3882}, {iov_base =
0x31876000, iov_len = 32768}, {iov_base = 0x165e760, iov_len = 16763984},
{iov_base = 0x11, iov_len = 644902944}, {
iov_base = 0xdd7ffe6bae40, iov_len = 18446706140541555064},
{iov_base = 0x0, iov_len = 325825}, {iov_base = 0x1, iov_len = 3}, {iov_base =
0x0, iov_len = 8323579}, {
iov_base = 0xdd7ffe6bae40, iov_len = 8372075}, {iov_base = 0x0,
iov_len = 463521516}, {iov_base = 0xdd7ffe6bae00, iov_len =
18446706140542042283}, {iov_base = 0x0,
iov_len = 57252368}, {iov_base = 0x0, iov_len = 463521504},
{iov_base = 0x165e760, iov_len = 1435219220224819912}, {iov_base = 0x10357f0,
iov_len = 18446706140541555235}, {
iov_base = 0xdd7ffe6bae40, iov_len = 16763984}}
#3 0x007f09b7 in NetHandler::mainNetEvent (this=0x165e760,
event=optimized out, e=optimized out) at UnixNet.cc:546
epd = optimized out
pd = 0x3b48010
retval = optimized out
nget = 1
__FUNCTION__ = mainNetEvent
poll_timeout = 747651072
vc = optimized out
ptimeout = {tv_sec = 0, tv_nsec = 1000}
#4 0x00821af0 in handleEvent (data=0x3699a70, event=5, this=optimized
out) at I_Continuation.h:145
No locals.
#5 process_event (e=0x3699a70, this=0x165b010, calling_code=optimized out)
at UnixEThread.cc:128
c_temp = optimized out
lock = {m = {m_ptr = 0xfc8910}, lock_acquired = true}
calling_code = 5
#6 EThread::execute (this=0x165b010) at UnixEThread.cc:252
done_one = optimized out
e = optimized out
NegativeQueue = optimized out
next_time = 1435219134815957937
#7 0x008202aa in spawn_thread_internal (a=0x3693f10) at Thread.cc:85
p = 0x3693f10
#8 0xdd7fff070dba in _thrp_setup () from /lib/64/libc.so.1
No symbol table info available.
#9 0xdd7fff0710d0 in ?? () from /lib/64/libc.so.1
No symbol table info available.
#10 0x in ?? ()
No symbol table info available.
{noformat}
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts-3266-2.diff, ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599604#comment-14599604
]
Eric Sproul commented on TS-3486:
-
I deployed the ts-3266-2 diff about 2 hours ago. So far so good.
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts-3266-2.diff, ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at HttpSM.cc:1455
#25 0x005ba712 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6876
#26 0x005ba702 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6919
#27 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#28 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6,
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593433#comment-14593433
]
Eric Sproul commented on TS-3486:
-
We saw some crashes overnight as well, but they are different from the previous
ones. All the stacks look like this:
{noformat}
read_from_net(NetHandler*, UnixNetVConnection*, EThread*)+0x2d7()
NetHandler::mainNetEvent+0x1f7()
EThread::execute+0x8d0()
spawn_thread_internal(void*)+0x4a()
libc.so.1`_thrp_setup+0x8a(dd7ffecdc280)
libc.so.1`_lwp_start()
{noformat}
So, possibly something else. Didn't see any open issues that looked related.
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at HttpSM.cc:1455
#25
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593501#comment-14593501
]
Eric Sproul commented on TS-3486:
-
[~shinrich] We had to disable our Backtrace crash logger because the way it is
implemented it breaks service management (meaning the service won't
stop/restart properly). I do have core files. Once it started crashing, there
were 5 crashes in about 8 hours, but we've gone about 9 hours now with no
crashes.
I can't share the core file but I can try to extract whatever you need; I'm not
all that good with a debugger though so no promises. :)
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at HttpSM.cc:1455
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593717#comment-14593717
]
Eric Sproul commented on TS-3486:
-
[~shinrich] I see; gdb is sometimes wonky on illumos, but appears to have
worked well enough here:
{noformat}
(gdb) bt
#0 0xdd7fff07758a in _portfs () from /lib/64/libc.so.1
#1 0xdd7ffeff1223 in port_getn () from /lib/64/libc.so.1
#2 0x007f0bad in NetHandler::mainNetEvent (this=0x1137760,
event=optimized out, e=optimized out) at UnixNet.cc:472
#3 0x00821e40 in handleEvent (data=0x36a8e30, event=5, this=optimized
out) at I_Continuation.h:145
#4 process_event (e=0x36a8e30, this=0x1134010, calling_code=optimized out)
at UnixEThread.cc:128
#5 EThread::execute (this=0x1134010) at UnixEThread.cc:252
#6 0x00824d0b in main ()
{noformat}
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0)
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593733#comment-14593733
]
Eric Sproul commented on TS-3486:
-
I realize now that the previous post was erroneous; I don't think that's the
faulting thread. I'm working on getting the right one.
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at HttpSM.cc:1455
#25 0x005ba712 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6876
#26 0x005ba702 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6919
#27 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#28 0x005b45f8 in
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593671#comment-14593671
]
Eric Sproul commented on TS-3486:
-
[~shinrich] well this is odd, but possibly a C++ism...
My instruction pointer is 0x008009d7, but addr2line puts me in a
header?!?
{noformat}
# gaddr2line -e /opt/circonus/bin/traffic_server 0x008009d7
/tmp/build_esproul/trafficserver-5.3.0/iocore/net/../../lib/ts/Ptr.h:300
{noformat}
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at HttpSM.cc:1455
#25 0x005ba712 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6876
#26 0x005ba702 in HttpSM::set_next_state
[jira] [Comment Edited] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593733#comment-14593733
]
Eric Sproul edited comment on TS-3486 at 6/19/15 6:20 PM:
--
I realize now that the previous post was erroneous; I don't think that's the
faulting thread. I'm working on getting the right one.
[Edited]
Maybe it's correct after all. [~shinrich] if you think I've screwed something
up, let me know and I'll dig deeper.
was (Author: esproul):
I realize now that the previous post was erroneous; I don't think that's the
faulting thread. I'm working on getting the right one.
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593784#comment-14593784
]
Eric Sproul commented on TS-3486:
-
I guess I'm used to debuggers like mdb and dbx that put you in the faulted
thread from the start...
From dbx I determined it was thread/LWP 26 that segfaulted, AND I grabbed the
source while I was at it, so...
{noformat}
(gdb) thread 26
[Switching to thread 26 (LWP26)]
#0 0x008009d7 in operator IOBufferBlock* (this=0x16218ed0) at
../../lib/ts/Ptr.h:300
300 operator T *() const { return (m_ptr); }
(gdb) bt
#0 0x008009d7 in operator IOBufferBlock* (this=0x16218ed0) at
../../lib/ts/Ptr.h:300
#1 first_write_block (this=0x16218ec0) at
../../iocore/eventsystem/I_IOBuffer.h:926
#2 read_from_net (nh=0x2921760, vc=0x19445740, thread=optimized out) at
UnixNetVConnection.cc:275
#3 0x007f0d17 in NetHandler::mainNetEvent (this=0x2921760,
event=optimized out, e=optimized out) at UnixNet.cc:546
#4 0x00821e40 in handleEvent (data=0x36a7cf0, event=5, this=optimized
out) at I_Continuation.h:145
#5 process_event (e=0x36a7cf0, this=0x291e010, calling_code=optimized out)
at UnixEThread.cc:128
#6 EThread::execute (this=0x291e010) at UnixEThread.cc:252
#7 0x008205fa in spawn_thread_internal (a=0x36a2a50) at Thread.cc:85
#8 0xdd7fff070dba in _thrp_setup () from /lib/64/libc.so.1
#9 0xdd7fff0710d0 in ?? () from /lib/64/libc.so.1
#10 0x in ?? ()
{noformat}
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593806#comment-14593806
]
Eric Sproul commented on TS-3486:
-
Additionally, here's a `bt full` output (thanks [~psudaemon]!)
{noformat}
(gdb) bt full
#0 0x008009d7 in operator IOBufferBlock* (this=0x16218ed0) at
../../lib/ts/Ptr.h:300
No locals.
#1 first_write_block (this=0x16218ec0) at
../../iocore/eventsystem/I_IOBuffer.h:926
No locals.
#2 read_from_net (nh=0x2921760, vc=0x19445740, thread=optimized out) at
UnixNetVConnection.cc:275
b = optimized out
buf = @0x19445880: {mbuf = 0x16218ec0, entry = 0x0}
toread = optimized out
total_read = optimized out
r = 0
lock = {m = {m_ptr = 0x13cbe790}, lock_acquired = true}
rattempted = optimized out
__FUNCTION__ = read_from_net
s = 0x19445858
mutex = 0xfc9150
ntodo = 9223372036854775807
niov = optimized out
tiovec = {{iov_base = 0x159bc0a7, iov_len = 3929}, {iov_base =
0xbc7a000, iov_len = 32768}, {iov_base = 0xdd7ffd4a8e40, iov_len =
8270580}, {iov_base = 0x32, iov_len = 428876640}, {
iov_base = 0xdd7ffd4a8e40, iov_len = 18446706140541555064},
{iov_base = 0x0, iov_len = 18883}, {iov_base = 0x1, iov_len = 2487}, {iov_base
= 0x0, iov_len = 8324443}, {
iov_base = 0xdd7ffd4a8e40, iov_len = 8372923}, {iov_base = 0x0,
iov_len = 386230828}, {iov_base = 0xdd7ffd4a8e00, iov_len =
18446706140542042283}, {iov_base = 0x0,
iov_len = 57310352}, {iov_base = 0x0, iov_len = 386230816},
{iov_base = 0x2921760, iov_len = 1434694902351034537}, {iov_base =
0xdd7ffd4a8e40, iov_len = 18446706140541555235}, {
iov_base = 0xdd7ffd4a8e40, iov_len = 16594640}}
#3 0x007f0d17 in NetHandler::mainNetEvent (this=0x2921760,
event=optimized out, e=optimized out) at UnixNet.cc:546
epd = optimized out
pd = 0x4bc1010
retval = optimized out
nget = 1
__FUNCTION__ = mainNetEvent
poll_timeout = 318363216
vc = optimized out
ptimeout = {tv_sec = 0, tv_nsec = 1000}
#4 0x00821e40 in handleEvent (data=0x36a7cf0, event=5, this=optimized
out) at I_Continuation.h:145
No locals.
#5 process_event (e=0x36a7cf0, this=0x291e010, calling_code=optimized out)
at UnixEThread.cc:128
c_temp = optimized out
lock = {m = {m_ptr = 0xfc8010}, lock_acquired = true}
calling_code = 5
#6 EThread::execute (this=0x291e010) at UnixEThread.cc:252
done_one = optimized out
e = optimized out
NegativeQueue = optimized out
next_time = 1434694895853026566
#7 0x008205fa in spawn_thread_internal (a=0x36a2a50) at Thread.cc:85
p = 0x36a2a50
#8 0xdd7fff070dba in _thrp_setup () from /lib/64/libc.so.1
No symbol table info available.
#9 0xdd7fff0710d0 in ?? () from /lib/64/libc.so.1
No symbol table info available.
#10 0x in ?? ()
No symbol table info available.
(gdb)
{noformat}
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591911#comment-14591911
]
Eric Sproul commented on TS-3486:
-
This morning I've deployed a build that includes this patch. Should know by
COB if it's stable.
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at HttpSM.cc:1455
#25 0x005ba712 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6876
#26 0x005ba702 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6919
#27 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#28 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592598#comment-14592598
]
Eric Sproul commented on TS-3486:
-
So far so good, it hasn't crashed since I deployed. Since deploying 5.3.0 28
days ago, we've averaged 11 crashes/day, or a crash about every 2 hours. If we
go 24 hours without a crash, I'll call this one fixed. :)
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at HttpSM.cc:1455
#25 0x005ba712 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6876
#26 0x005ba702 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6919
#27 0x005b3f5f in
[jira] [Updated] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Sproul updated TS-3486:
Attachment: ts3486-ptrace.txt.gz
[~shinrich] This is (sanitized) ptrace output from one of the Backtrace
objects. I don't see a reference to server_vc but I will shortly be deploying
an updated build that uses libumem, so a future crash dump may be more useful.
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
Attachments: ts3486-ptrace.txt.gz
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at HttpSM.cc:1455
#25 0x005ba712 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6876
#26 0x005ba702 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6919
#27 0x005b3f5f in HttpSM::handle_api_return
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590070#comment-14590070
]
Eric Sproul commented on TS-3486:
-
[~shinrich] we have some Backtrace.io data from crashes like this that we could
share. I can't give you direct access to our BT database, but if [~psudaemon]
or [~zwoop] can guide me, I can pull whatever info you need.
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Assignee: Phil Sorber
Labels: crash
Fix For: sometime
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=value optimized out, data=value optimized out) at HttpSM.cc:1381
#8 0x005b627d in HttpSM::do_http_server_open (this=0x2aaadccc4bf0,
raw=value optimized out) at HttpSM.cc:4639
#9 0x005baa04 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:7021
#10 0x005b25a3 in HttpSM::state_cache_open_write
(this=0x2aaadccc4bf0, event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2442
#11 0x005b5b28 in HttpSM::main_handler (this=0x2aaadccc4bf0,
event=1108, data=0x2aab1c3b6800) at HttpSM.cc:2554
#12 0x0059338a in handleEvent (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at
../../iocore/eventsystem/I_Continuation.h:145
#13 HttpCacheSM::state_cache_open_write (this=0x2aaadccc6618, event=value
optimized out, data=0x2aab1c3b6800) at HttpCacheSM.cc:167
#14 0x00697223 in handleEvent (this=0x2aab1c3b6800, event=value
optimized out) at ../../iocore/eventsystem/I_Continuation.h:145
#15 CacheVC::callcont (this=0x2aab1c3b6800, event=value optimized out) at
../../iocore/cache/P_CacheInternal.h:662
#16 0x00715940 in Cache::open_write (this=value optimized out,
cont=value optimized out, key=0x2ba0ff762d70, info=value optimized out,
apin_in_cache=46914401429576, type=CACHE_FRAG_TYPE_HTTP,
hostname=0x2aaadd281078
www.mifangba.comhttpapi.phpwww.mifangba.comhttp://www.mifangba.com/api.php?op=countid=4modelid=12;,
host_len=16) at CacheWrite.cc:1788
#17 0x006e5765 in open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at
P_CacheInternal.h:1093
#18 CacheProcessor::open_write (this=value optimized out,
cont=0x2aaadccc6618, expected_size=value optimized out, url=0x2aaadccc5310,
cluster_cache_local=value optimized out, request=value optimized out,
old_info=0x0, pin_in_cache=0, type=CACHE_FRAG_TYPE_HTTP) at Cache.cc:3622
#19 0x005936f0 in HttpCacheSM::open_write (this=0x2aaadccc6618,
url=value optimized out, request=value optimized out, old_info=value
optimized out,
pin_in_cache=value optimized out, retry=value optimized out,
allow_multiple=false) at HttpCacheSM.cc:298
#20 0x005a022e in HttpSM::do_cache_prepare_action
(this=0x2aaadccc4bf0, c_sm=0x2aaadccc6618, object_read_info=0x0, retry=true,
allow_multiple=false) at HttpSM.cc:4511
#21 0x005babd9 in do_cache_prepare_write (this=0x2aaadccc4bf0) at
HttpSM.cc:4436
#22 HttpSM::set_next_state (this=0x2aaadccc4bf0) at HttpSM.cc:7098
#23 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1517
#24 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=0, data=0x0) at HttpSM.cc:1455
#25 0x005ba712 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6876
#26 0x005ba702 in HttpSM::set_next_state (this=0x2aaadccc4bf0) at
HttpSM.cc:6919
#27 0x005b3f5f in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
[jira] [Commented] (TS-3642) proxy.config.http.share_server_sessions not working
[ https://issues.apache.org/jira/browse/TS-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14590111#comment-14590111 ] Eric Sproul commented on TS-3642: - +1 for a 5.3 backport. proxy.config.http.share_server_sessions not working --- Key: TS-3642 URL: https://issues.apache.org/jira/browse/TS-3642 Project: Traffic Server Issue Type: Bug Components: Configuration Affects Versions: 5.3.0 Reporter: David Carlin Assignee: Phil Sorber Fix For: 6.0.0 Testing 5.3.0 and I noticed proxy.config.http.share_server_sessions = 1 no longer works. Saw a 10-15x increase in origin connections; there appears to be some reuse, I am seeing approximately 1.2-1.3 requests per origin connection. Setting proxy.config.http.server_session_sharing.pool = global restored expected behavior (Thanks [~sudheerv]!) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3642) proxy.config.http.share_server_sessions not working
[ https://issues.apache.org/jira/browse/TS-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14588337#comment-14588337 ] Eric Sproul commented on TS-3642: - We are still seeing higher-than-expected origin connection counts with 5.3.0. We currently have explicitly set: * CONFIG proxy.config.http.share_server_sessions INT 1 * CONFIG proxy.config.http.server_session_sharing.pool STRING global proxy.config.http.share_server_sessions not working --- Key: TS-3642 URL: https://issues.apache.org/jira/browse/TS-3642 Project: Traffic Server Issue Type: Bug Components: Configuration Affects Versions: 5.3.0 Reporter: David Carlin Assignee: Alan M. Carroll Fix For: 6.0.0 Testing 5.3.0 and I noticed proxy.config.http.share_server_sessions = 1 no longer works. Saw a 10-15x increase in origin connections; there appears to be some reuse, I am seeing approximately 1.2-1.3 requests per origin connection. Setting proxy.config.http.server_session_sharing.pool = global restored expected behavior (Thanks [~sudheerv]!) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-1153) On Solaris, link with libumem by default
[ https://issues.apache.org/jira/browse/TS-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582490#comment-14582490 ] Eric Sproul commented on TS-1153: - Ah ok, a fun foray into Autotools. ;) So do we need to follow the jemalloc/tcmalloc example as stated above, or just do a basic AC_CHECK_LIB or some such? On Solaris, link with libumem by default Key: TS-1153 URL: https://issues.apache.org/jira/browse/TS-1153 Project: Traffic Server Issue Type: Improvement Components: Performance Environment: Solaris Reporter: Igor Galić Assignee: Eric Sproul Labels: memory Fix For: sometime http://www.oracle.com/technetwork/server-storage/solaris10/solaris-memory-135224.html We should make use of this and link to libumem by default on Solaris and Illumos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-1153) On Solaris, link with libumem by default
[ https://issues.apache.org/jira/browse/TS-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582522#comment-14582522 ] Eric Sproul commented on TS-1153: - Maybe [~i.galic] would like to weigh in again on the by-default question. The ability to just turn on the libumem debug features without a rebuild would seem a win, in any case. On Solaris, link with libumem by default Key: TS-1153 URL: https://issues.apache.org/jira/browse/TS-1153 Project: Traffic Server Issue Type: Improvement Components: Performance Environment: Solaris Reporter: Igor Galić Assignee: Eric Sproul Labels: memory Fix For: sometime http://www.oracle.com/technetwork/server-storage/solaris10/solaris-memory-135224.html We should make use of this and link to libumem by default on Solaris and Illumos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-1153) On Solaris, link with libumem by default
[ https://issues.apache.org/jira/browse/TS-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582559#comment-14582559 ] Eric Sproul commented on TS-1153: - libumem's malloc/free should be more performant on modern multi-core systems than traditional libc malloc, and you get all the other nice features along for the ride. I'll see if I can test LD_PRELOADing libumem on our instances just as a quick test. On Solaris, link with libumem by default Key: TS-1153 URL: https://issues.apache.org/jira/browse/TS-1153 Project: Traffic Server Issue Type: Improvement Components: Performance Environment: Solaris Reporter: Igor Galić Assignee: Eric Sproul Labels: memory Fix For: sometime http://www.oracle.com/technetwork/server-storage/solaris10/solaris-memory-135224.html We should make use of this and link to libumem by default on Solaris and Illumos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-1153) On Solaris, link with libumem by default
[ https://issues.apache.org/jira/browse/TS-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582307#comment-14582307 ] Eric Sproul commented on TS-1153: - [~psudaemon] yes I would. What would you need from us? On Solaris, link with libumem by default Key: TS-1153 URL: https://issues.apache.org/jira/browse/TS-1153 Project: Traffic Server Issue Type: Improvement Components: Performance Environment: Solaris Reporter: Igor Galić Labels: memory Fix For: sometime http://www.oracle.com/technetwork/server-storage/solaris10/solaris-memory-135224.html We should make use of this and link to libumem by default on Solaris and Illumos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3667) SSL Handhake read does not correctly handle EOF and error cases
[ https://issues.apache.org/jira/browse/TS-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573663#comment-14573663 ] Eric Sproul commented on TS-3667: - Initial indications are that this fixes our high CPU issue. Previous to this patch we were at around 50% CPU usage on a dual hex-core system (24 hardware threads). After the patch we are back to where we were with 4.2.x, about 2.5%. SSL Handhake read does not correctly handle EOF and error cases --- Key: TS-3667 URL: https://issues.apache.org/jira/browse/TS-3667 Project: Traffic Server Issue Type: Bug Components: SSL Reporter: Susan Hinrichs Assignee: Susan Hinrichs Attachments: ts-3667.diff Reported by [~esproul] and postwait. The return value of SSLNetVConnection::read_raw_data() is being ignored. So EOF and errors are not terminated, but rather spin until the inactivity timeout is reached. EAGAIN is not being descheduled until more data is available. This results in higher CPU utilization and hitting the SSL_error() function much more than it needs to be hit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TS-3648) Desire support for client TLS cipher in custom log format
Eric Sproul created TS-3648: --- Summary: Desire support for client TLS cipher in custom log format Key: TS-3648 URL: https://issues.apache.org/jira/browse/TS-3648 Project: Traffic Server Issue Type: New Feature Components: Logging Reporter: Eric Sproul While the TLS cipher stats from TS-2169 are awesome for seeing what ciphers are in active use, I would also like to be able to create a custom log to track cipher by client request. That would enable me, for example, to determine who among my users might be affected by a cipher list change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3486) Segfault in do_io_write with plugin (??)
[
https://issues.apache.org/jira/browse/TS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14560925#comment-14560925
]
Eric Sproul commented on TS-3486:
-
I believe I am also seeing this with 5.3.0 and stats_over_http (using a custom
path, if that matters.)
This is from OmniOS r151006 (illumos). I can make core files available upon
request.
Many stacks look similar to this:
{noformat}
HttpServerSession::do_io_write(Continuation*, long, IOBufferReader*, bool)+0xb()
HttpSM::state_api_callout(int, void*)+0x2d0()
HttpSM::do_http_server_open(bool)+0x268()
HttpSM::set_next_state()+0x510()
HttpSM::state_api_callout(int, void*)+0x2d0()
HttpSM::set_next_state()+0x248()
HttpSM::do_hostdb_lookup()+0x272()
HttpSM::set_next_state()+0xd9f()
HttpSM::state_api_callout(int, void*)+0x2d0()
HttpSM::state_cache_open_read(int, void*)+0x188()
HttpSM::main_handler(int, void*)+0xc9()
HttpCacheSM::state_cache_open_read(int, void*)+0x163()
_ZN5Cache9open_readEP12ContinuationPN3ats10CryptoHashEP7HTTPHdrP21CacheLookupHtt
pConfig13CacheFragTypePci+0x6e1()
_ZN14CacheProcessor9open_readEP12ContinuationP3URLbP7HTTPHdrP21CacheLookupHttpCo
nfigl13CacheFragType+0x8e()
HttpCacheSM::open_read(URL*, HTTPHdr*, CacheLookupHttpConfig*, long)+0x85()
HttpSM::do_cache_lookup_and_read()+0xfd()
HttpSM::set_next_state()+0x811()
HttpSM::state_api_callout(int, void*)+0x2d0()
HttpSM::set_next_state()+0x248()
HttpSM::set_next_state()+0x862()
HttpSM::state_api_callout(int, void*)+0x2d0()
HttpSM::set_next_state()+0x248()
HttpSM::state_api_callout(int, void*)+0x2d0()
HttpSM::state_api_callback(int, void*)+0x8f()
TSHttpTxnReenable+0x1ed()
stats_over_http.so`stats_origin+0x86()
HttpSM::state_api_callout(int, void*)+0xde()
HttpSM::set_next_state()+0x248()
HttpSM::state_read_client_request_header(int, void*)+0x207()
HttpSM::main_handler(int, void*)+0xc9()
HttpSM::state_api_callout(int, void*)+0x2d0()
HttpSM::state_add_to_list(int, void*)+0x198()
HttpSM::attach_client_session(HttpClientSession*, IOBufferReader*)+0x3b5()
HttpClientSession::new_transaction()+0xc4()
ProxyClientSession::state_api_callout(int, void*)+0x13d()
_ZN17HttpClientSession14new_connectionEP14NetVConnectionP9MIOBufferP14IOBufferRe
aderb+0x224()
HttpSessionAccept::accept(NetVConnection*, MIOBuffer*, IOBufferReader*)+0x22
4()
ProtocolProbeTrampoline::ioCompletionEvent(int, void*)+0x3ca()
UnixNetVConnection::readSignalAndUpdate(int)+0x48()
SSLNetVConnection::net_read_io(NetHandler*, EThread*)+0xfad()
NetHandler::mainNetEvent(int, Event*)+0x1f7()
EThread::execute()+0x8d0()
spawn_thread_internal(void*)+0x4a()
libc.so.1`_thrp_setup+0x8a(dd7ffed71a80)
libc.so.1`_lwp_start()
{noformat}
Crash status and registers:
{noformat}
SIGSEGV: Segmentation Fault
%rax = 0x00838950 %r8 = 0x
%rbx = 0x08993340 %r9 = 0x0a0d
%rcx = 0x065cbcf8 %r10 = 0x0658
%rdx = 0x017b %r11 = 0xdd7fff087348
%rsi = 0x08993340 %r12 = 0x065cbcf8
%rdi = 0x %r13 = 0x017b
%r14 = 0x08994c68
%r15 = 0x08993a02
%cs = 0x0053%fs = 0x%gs = 0x
%ds = 0x004b%es = 0x004b%ss = 0x004b
%rip = 0x0067f66b
_ZN17HttpServerSession11do_io_writeEP12ContinuationlP14IOBufferReaderb+0xb
%rbp = 0xdd7ffeb5ba40
%rsp = 0xdd7ffeb5a9e8
%rflags = 0x00010246
id=0 vip=0 vif=0 ac=0 vm=0 rf=1 nt=0 iopl=0x0
status=of,df,IF,tf,sf,ZF,af,PF,cf
%gsbase = 0x
%fsbase = 0xdd7ffed71a80
%trapno = 0xe
%err = 0x4
{noformat}
Segfault in do_io_write with plugin (??)
Key: TS-3486
URL: https://issues.apache.org/jira/browse/TS-3486
Project: Traffic Server
Issue Type: Bug
Affects Versions: 5.2.0, 5.3.0
Reporter: Qiang Li
Labels: crash
Fix For: sometime
{code}
(gdb) bt
#0 0x005bdb8b in HttpServerSession::do_io_write (this=value
optimized out, c=0x2aaadccc4bf0, nbytes=576, buf=0x2aaafc2ffee8, owner=false)
at HttpServerSession.cc:104
#1 0x005acc1d in HttpSM::setup_server_send_request
(this=0x2aaadccc4bf0) at HttpSM.cc:5686
#2 0x005b3f85 in HttpSM::handle_api_return (this=0x2aaadccc4bf0) at
HttpSM.cc:1520
#3 0x005b45f8 in HttpSM::state_api_callout (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1455
#4 0x005b980b in HttpSM::state_api_callback (this=0x2aaadccc4bf0,
event=6, data=0x0) at HttpSM.cc:1275
#5 0x004d7a1b in TSHttpTxnReenable (txnp=0x2aaadccc4bf0,
event=TS_EVENT_HTTP_CONTINUE) at InkAPI.cc:5614
#6 0x2ba118441c89 in cachefun (contp=value optimized out, event=value
optimized out, edata=0x2aaadccc4bf0) at main.cpp:1876
#7 0x005b4466
[jira] [Created] (TS-3553) Fix illumos build
Eric Sproul created TS-3553: --- Summary: Fix illumos build Key: TS-3553 URL: https://issues.apache.org/jira/browse/TS-3553 Project: Traffic Server Issue Type: Bug Components: Core Reporter: Eric Sproul Have hit numerous issues building 5.3-rc2 on an illumos platform (OmniOS r151014). This issue should reference efforts to correct these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3553) Fix illumos build
[ https://issues.apache.org/jira/browse/TS-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509274#comment-14509274 ] Eric Sproul commented on TS-3553: - Linking previous issues into this catch-all. Fix illumos build - Key: TS-3553 URL: https://issues.apache.org/jira/browse/TS-3553 Project: Traffic Server Issue Type: Bug Components: Core Reporter: Eric Sproul Have hit numerous issues building 5.3-rc2 on an illumos platform (OmniOS r151014). This issue should reference efforts to correct these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TS-3548) psiginfo differs on illumos vs. Linux
Eric Sproul created TS-3548:
---
Summary: psiginfo differs on illumos vs. Linux
Key: TS-3548
URL: https://issues.apache.org/jira/browse/TS-3548
Project: Traffic Server
Issue Type: Bug
Components: Core
Reporter: Eric Sproul
The arguments to psiginfo() are different on illumos platforms such as OmniOS.
This results in a build error:
{noformat}
CXXLDCompileParseRules
./CompileParseRules
CXX ParseRules.lo
CCLD mkdfa
signals.cc: In function 'void signal_format_siginfo(int, siginfo_t*, const
char*)':
signals.cc:162:21: error: invalid conversion from 'const char*' to 'char*'
[-fpermissive]
psiginfo(info, msg);
^
In file included from ink_platform.h:104:0,
from libts.h:45,
from signals.cc:29:
/usr/include/siginfo.h:53:13: error: initializing argument 2 of 'void
psiginfo(siginfo_t*, char*)' [-fpermissive]
extern void psiginfo(siginfo_t *, char *);
^
gmake[3]: *** [signals.lo] Error 1
{noformat}
This happens with 5.2.1 as well as 5.3.0-rc2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
