Re: libcurl 8.16.0 spawning large number of getaddrinfo threads?

Mon, 20 Oct 2025 09:49:19 -0700 

On 10/17/25 4:27 AM, Stefan Eissing wrote:

!-------------------------------------------------------------------|
   This Message Is From an Untrusted Sender
   You have not previously corresponded with this sender.
|-------------------------------------------------------------------!




Am 17.10.2025 um 01:45 schrieb Vadim Grinshpun via curl-library 
<[email protected]>:

Hi folks,
I'm working on a project that makes use of libcurl.
After upgrading libcurl from 8.14.1 to 8.16.0, I've noticed an unexpected 
behavior change.
Specifically, I'm observing a large number of curl_thread_create_thunk threads, 
all of which
appear to have a backtrace like the one shown below.

Since the only thing that changed is the libcurl version,
I want to double-check a few things:
- are there any known issues in 8.16.0 that might have symptoms like this?
- has anything changed between 8.14.1 and 8.16.1 that might lead to this 
behavior?
- is it possible to misuse libcurl in a way that can lead to this behavior?

curl 8.16.0 uses pthread_cancel() to make threads terminate early when the 
results are no longer needed. This was reverted in 
de3fc1d7adb78c078e4cc7ccc48e550758094ad3, as it proved to be unreliable due to 
unsoundness of libc's implementations. See 
https://urldefense.com/v3/__https://eissing.org/icing/posts/rip_pthread_cancel/__;!!GjvTz_vk!UJmqfctaSfqRaMoKcJNtzsV1Ybl8bHlQronZU_c2gC1kcbFeQUMHGKqUCiOvcUyHAZG6WzhL5zaaLFc$
  for my analysis of this.

What you see in the stacktraces seem to be exactly that. libc holds a lock and 
does not release it on a pthread_cancel, leading to all other started resolver 
threads to wait on this.

The workarounds are either to apply the mentioned patch or to build curl using 
c-ares for resolving. Or try our recent release candidate 
https://urldefense.com/v3/__https://curl.se/rc/__;!!GjvTz_vk!UJmqfctaSfqRaMoKcJNtzsV1Ybl8bHlQronZU_c2gC1kcbFeQUMHGKqUCiOvcUyHAZG6WzhL8F18nQc$
  for curl 8.17 due on November 5.

Hi Stefan,
Thanks for confirming this, and for an interesting write-up of the problem :) The write-up talks about the memory leak as being the observed symptom; in our case, I think it was more severe than that--each thread also had an associated open file descriptor (eventfd, I believe), causing our long-running process to run out of fds once enough threads have been spawned. Anyway, glad this was found and reverted.
I'll look into enabling c-ares. In another part of this thread, Daniel mentioned that c-ares can behave differently from getaddrinfo; are known differences documented somewhere? (I'd like to ensure we won't get affected). 

Thanks very much for your help!
-Vadim


- Stefan


Thanks for any suggestions!
-Vadim


Stack traces:
Hundreds of threads look like this, providing a couple of examples below:

Thread 953 (Thread 0x7f620bbf6640 (LWP 4084852) "ddc_beacon_rece"):
#0  futex_wait (private=0, expected=2, futex_word=0x7f640ac23040 <lock>) at 
../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait_private (futex=futex@entry=0x7f640ac23040 <lock>) at 
./nptl/lowlevellock.c:34
#2  0x00007f640ab43dc4 in __check_pf (seen_ipv4=seen_ipv4@entry=0x7f620bbf1516, 
seen_ipv6=seen_ipv6@entry=0x7f620bbf1517, in6ai=in6ai@entry=0x7f620bbf1528, 
in6ailen=in6ailen@entry=0x7f620bbf1530) at 
../sysdeps/unix/sysv/linux/check_pf.c:307
#3  0x00007f640ab0bd61 in __GI_getaddrinfo (name=<optimized out>, service=<optimized out>, 
service@entry=0x7f620bbf1cfc "8482", hints=<optimized out>, hints@entry=0x7f638ebb1008, 
pai=pai@entry=0x7f620bbf1c10) at ../sysdeps/posix/getaddrinfo.c:2446
#4  0x0000561dbba888b8 in Curl_getaddrinfo_ex (nodename=<optimized out>, 
servname=servname@entry=0x7f620bbf1cfc "8482", hints=hints@entry=0x7f638ebb1008, 
result=result@entry=0x7f638ebb1000) at curl_addrinfo.c:122
#5  0x0000561dbba7a808 in getaddrinfo_thread (arg=arg@entry=0x7f638ebb0fc0) at 
asyn-thrdd.c:245
#6  0x0000561dbba88fff in curl_thread_create_thunk (arg=<optimized out>) at 
curl_threads.c:57
#7  0x00007f640aa94ac3 in start_thread (arg=<optimized out>) at 
./nptl/pthread_create.c:442
#8  0x00007f640ab269d0 in clone3 () at 
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

...
Thread 11 (Thread 0x7f63f4572640 (LWP 2016127) "ddc_beacon_rece"):
#0  futex_wait (private=0, expected=2, futex_word=0x7f640ac23040 <lock>) at 
../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait_private (futex=futex@entry=0x7f640ac23040 <lock>) at 
./nptl/lowlevellock.c:34
#2  0x00007f640ab43dc4 in __check_pf (seen_ipv4=seen_ipv4@entry=0x7f63f456d516, 
seen_ipv6=seen_ipv6@entry=0x7f63f456d517, in6ai=in6ai@entry=0x7f63f456d528, 
in6ailen=in6ailen@entry=0x7f63f456d530) at 
../sysdeps/unix/sysv/linux/check_pf.c:307
#3  0x00007f640ab0bd61 in __GI_getaddrinfo (name=<optimized out>, service=<optimized out>, 
service@entry=0x7f63f456dcfc "8482", hints=<optimized out>, hints@entry=0x7f6406422348, 
pai=pai@entry=0x7f63f456dc10) at ../sysdeps/posix/getaddrinfo.c:2446
#4  0x0000561dbba888b8 in Curl_getaddrinfo_ex (nodename=<optimized out>, 
servname=servname@entry=0x7f63f456dcfc "8482", hints=hints@entry=0x7f6406422348, 
result=result@entry=0x7f6406422340) at curl_addrinfo.c:122
#5  0x0000561dbba7a808 in getaddrinfo_thread (arg=arg@entry=0x7f6406422300) at 
asyn-thrdd.c:245
#6  0x0000561dbba88fff in curl_thread_create_thunk (arg=<optimized out>) at 
curl_threads.c:57
#7  0x00007f640aa94ac3 in start_thread (arg=<optimized out>) at 
./nptl/pthread_create.c:442
#8  0x00007f640ab269d0 in clone3 () at 
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Possibly relevant: there's another thread that's in the process of destroying 
an object,
which leads to this (abbreviated) stack trace:
Thread 3 (Thread 0x7f64093ff640 (LWP 1898221) "ddc_beacon_rece"):
#0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, 
op=265, expected=3821886, futex_word=0x7f622fb54910) at 
./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, 
clockid=0, expected=3821886, futex_word=0x7f622fb54910) at 
./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 
(futex_word=futex_word@entry=0x7f622fb54910, expected=3821886, 
clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) 
at ./nptl/futex-internal.c:139
#3  0x00007f640aa96624 in __pthread_clockjoin_ex (threadid=140059683931712, 
thread_return=thread_return@entry=0x0, clockid=clockid@entry=0, 
abstime=abstime@entry=0x0, block=block@entry=true) at 
./nptl/pthread_join_common.c:105
#4  0x00007f640aa964c3 in ___pthread_join (threadid=<optimized out>, 
thread_return=thread_return@entry=0x0) at ./nptl/pthread_join.c:24
#5  0x0000561dbba8910a in Curl_thread_join (hnd=hnd@entry=0x7f638eb9ac80) at 
curl_threads.c:95
#6  0x0000561dbba7acbf in asyn_thrdd_await (data=data@entry=0x7f6407663800, 
addr_ctx=0x7f638eb9ac80, entry=entry@entry=0x0) at asyn-thrdd.c:557
#7  0x0000561dbba7ad00 in Curl_async_thrdd_destroy 
(data=data@entry=0x7f6407663800) at asyn-thrdd.c:586
#8  0x0000561dbba7a41d in Curl_async_destroy (data=data@entry=0x7f6407663800) 
at asyn-base.c:214
#9  0x0000561dbba6678e in Curl_close (datap=datap@entry=0x7f64093fa8b8) at 
url.c:303
#10 0x0000561dbba4610c in curl_easy_cleanup (ptr=<optimized out>) at easy.c:870
(additional application-specific stack levels not shown).


--
Unsubscribe: 
https://urldefense.com/v3/__https://lists.haxx.se/mailman/listinfo/curl-library__;!!GjvTz_vk!UJmqfctaSfqRaMoKcJNtzsV1Ybl8bHlQronZU_c2gC1kcbFeQUMHGKqUCiOvcUyHAZG6WzhL-w766fY$
Etiquette:   
https://urldefense.com/v3/__https://curl.se/mail/etiquette.html__;!!GjvTz_vk!UJmqfctaSfqRaMoKcJNtzsV1Ybl8bHlQronZU_c2gC1kcbFeQUMHGKqUCiOvcUyHAZG6WzhLl2iWLaY$


--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Reply via email to