I am seeing an intermittent (2-3 times a week) segfault inside libcurl
code, and it is frequent enough that it is becoming a real problem.
I'm having problems finding the root cause so any help would be
appreciated.
A bit of context: This is an application that is responsible for
fetching data from multiple sources via HTTP and uploading the results
to a central server via HTTPS. It is multi-threaded, but to rule out
any threading problems a global mutex is locked around use of libcurl.
This at least reduced the number of different stack traces I am seeing
to this one:
(gdb) where
<<snip>>
#6 <signal handler called>
#7 0x080a8e12 in Curl_expire_clear (data=0xb3c1acf4) at multi.c:3022
#8 0x080be7f7 in Curl_close (data=0xb3c1acf4) at url.c:396
#9 0x080a7986 in curl_multi_cleanup (multi=0xb3c19b8c) at multi.c:2211
#10 0x080be833 in Curl_close (data=0xb3c03204) at url.c:408
#11 0x080a126a in curl_easy_cleanup (data=0xb3c03204) at easy.c:833
#12 0x080876da in HTTPClient::HTTPRequest (this=0x98ad74c,
aURL=0xb3c13604 "http://localhost:50080/<<redacted>>",
aPostData=0x0, aPostSize=0, aPage=0xb6dd32d8, aTimeoutMS=10000,
aConnectTimeoutMS=10000, aContentType=0x0, aHeaderFields=0x0)
at <<redacted>>/HTTPClient.cpp:115
<<snip>>
The crash happens in the cleanup after what appears to be an otherwise
successful HTTP GET request.
It looks like something in Curl's state has become corrupted somewhere:
(gdb) list
3021 /* flush the timeout list too */
3022 while(list->size > 0)
<--- crash is here
3023 Curl_llist_remove(list, list->tail, NULL);
(gdb) print list
$9 = (struct curl_llist *) 0x0
(gdb) print data->state
$11 = {conn_cache = 0xb3c19bf4, multi_owned_by_easy = false, keeps_speed = {
tv_sec = 0, tv_usec = 0}, lastconnect = 0x0,
headerbuff = 0xb3c19e4c
"�\002��@\236��@\236��20375\t66700\t2\t2\t2\t0\t0\t0\t0\t0\t0\t4\t14\t3\t0\t0\t0\t0\n1481544716015001\t1\t2\t4\t194\t9717\t20373\t59900\t4\t4\t4\t0\t0\t0\t0\t0\t0\t5\t12\t2\t0\t0\t0\t0\n1481544716015002\t1\t3\t7\t194\t9717\t20374\t57900\t4\t4\t4\t0\t0\t0\t0\t0\t0\t5\t2\t"...,
headersize = 256, buffer = '\0' <repeats 16384 times>,
uploadbuffer = '\0' <repeats 16384 times>, current_speed = -1,
this_is_a_follow = false, first_host = 0x0, first_remote_port = 0,
session = 0x0, sessionage = 0, tempwrite = 0x0, tempwritesize = 0,
tempwritetype = 0, scratch = 0x0, errorbuf = false, os_errno = 0,
prev_signal = 0, allow_port = false, digest = {nonce = 0x0, cnonce = 0x0,
realm = 0x0, algo = 0, stale = false, opaque = 0x0, qop = 0x0,
algorithm = 0x0, nc = 0}, proxydigest = {nonce = 0x0, cnonce = 0x0,
realm = 0x0, algo = 0, stale = false, opaque = 0x0, qop = 0x0,
algorithm = 0x0, nc = 0}, authhost = {want = 0, picked = 0, avail = 0,
done = false, multi = false, iestyle = false}, authproxy = {want = 0,
picked = 0, avail = 0, done = false, multi = false, iestyle = false},
authproblem = false, resolver = 0x0, expiretime = {tv_sec = 0,
tv_usec = -1}, timenode = {smaller = 0xffffffff, larger = 0x0, same = 0x0,
key = {tv_sec = 0, tv_usec = 0}, payload = 0x0}, timeoutlist = 0x0,
most_recent_ftp_entrypath = 0x0, ftp_trying_alternative = false,
httpversion = 0, expect100header = false, pipe_broke = false,
---Type <return> to continue, or q <return> to quit---
prev_block_had_trailing_cr = false, crlf_conversions = 0, pathbuffer = 0x0,
path = 0x0, slash_removed = false, use_range = false,
rangestringalloc = false, range = 0x0, resume_from = 0,
rtsp_next_client_CSeq = 0, rtsp_next_server_CSeq = 0, rtsp_CSeq_recv = 0,
infilesize = 0, drain = 0, done = false, fread_func = 0, in = 0x0,
stream_depends_on = 0x0, stream_depends_e = false, stream_weight = 0}
From my reading of the code I think tv_usec and timenode.smaller are
not supposed to be 0xffffffff. The list of timeouts is allowed to be
NULL, but only if expiretime.tv_sec and expiretime.tv_usec are both 0.
That is the immediate cause of the segfault. But I can't figure out
why expiretime.tv_usec and timenode.smaller are set to invalid values.
Of course adding a NULL guard around that list dereference on line
3022 would fix the immediate crash but that doesn't fix the real
issue.
The code that calls libcurl is pretty much a copy-paste of libcurl
example code, but I can post a (slightly redacted) version if it's
needed.
