[
https://issues.apache.org/jira/browse/TS-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229378#comment-15229378
]
Thomas Jackson commented on TS-4207:
------------------------------------
[~zwoop] I have spent a decent amount of time on this while I was OOO on
vacation the last couple of weeks. It seems that the root cause of this issue
has always existed, and that the addition of always doing hostname storing
(https://github.com/apache/trafficserver/commit/0e703e1e) we are just causing
the issue to happen all the time.
To understand the issue I'll give a little background in how hostdb is
currently working. Basically hostdb is just a wrapper around this templated
struct called MultiCache. MultiCache is "multi" not because it is templated,
but because it has two types of storage (static-- blocks and dynamic-- alloc).
The static side of the cache can hold N HostDBInfo structs (the results of DNS
queries). The dynamic side is used to store the round robin records and various
strings associated with the record. The size of this dynamic space is defined
as (N x
[[estimated_heap_bytes_per_entry|https://github.com/apache/trafficserver/blob/master/iocore/hostdb/P_HostDBProcessor.h#L218].
The basic problem we are running into is that we are putting too much
preassure on the dynamic heap-- such that the heap is getting re-used while
people still have references to items in that space.
So, I've actually been working on re-writing MultiCache to allocate the entire
required block at once (so we don't have this problem where the parent exists
but not the children), but I'm not certain if we want such a change to go into
the 6.x branch (I'm willing to discuss if we want). If we aren't comfortable
with such a large change I suggest just accounting for the hostname size in the
estimated_heap_bytes_per_entry as a stopgap solution. The maximum allowable
size is 253 (so 254 with null terminator), but we could pick a smaller number
(~120 or so seems to be more reasonable). Alternatively you can increase the
number of records in hostdb (and the size accordingly) to increase the dynamic
heap size.
TLDR; almost done with the long term solution, but I'm not sure if we want to
merge that into 6.x-- alternatively we can do a simple workaround in 6.x
(https://github.com/apache/trafficserver/pull/553)
> Crash in HostDB, likely a regression from 5.x
> ---------------------------------------------
>
> Key: TS-4207
> URL: https://issues.apache.org/jira/browse/TS-4207
> Project: Traffic Server
> Issue Type: Bug
> Components: HostDB
> Reporter: Leif Hedstrom
> Assignee: Leif Hedstrom
> Priority: Blocker
> Labels: crash
> Fix For: 6.2.0
>
>
> We're seeing a new crash in HostDB, which did not occur in 5.3.x:
> {code}
> (gdb) bt
> #0 0x00002aaaaac7b2bb in HttpSM::process_hostdb_info(HostDBInfo*) () at
> ../../iocore/hostdb/P_HostDBProcessor.h:295
> #1 0x00002aaaaac88b16 in HttpSM::state_hostdb_lookup(int, void*) () at
> HttpSM.cc:2126
> #2 0x00002aaaaac9713d in HttpSM::main_handler(int, void*) () at
> HttpSM.cc:2561
> #3 0x00002aaaaad7803e in reply_to_cont(Continuation*, HostDBInfo*, bool) ()
> at ../../iocore/eventsystem/I_Continuation.h:153
> #4 0x00002aaaaad7eca5 in HostDBContinuation::dnsEvent(int, HostEnt*) () at
> HostDB.cc:1685
> #5 0x00002aaaaad98faf in DNSEntry::postEvent(int, Event*) () at
> ../../iocore/eventsystem/I_Continuation.h:153
> #6 0x00002aaaaae7e420 in EThread::process_event(Event*, int) () at
> I_Continuation.h:153
> #7 0x00002aaaaae7f2ab in EThread::execute() () at UnixEThread.cc:179
> #8 0x00002aaaaae7de06 in spawn_thread_internal(void*) () at Thread.cc:86
> #9 0x00002aaaad6ac9d1 in start_thread () from /lib64/libpthread.so.0
> #10 0x00002aaaae8b58fd in clone () from /lib64/libc.so.6
> {code}
> I think some inlining here complicates things, what it looks like the "r" is
> NULL, but it somehow still ends up using r->rr ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
