On Wed, 22 Mar 2017, mandar choure wrote:
- 21 Easy handles added to the multi-handle
We are observing CPU utilization to 100% under load testing when libcurl
started to spawn multiple threads to resolve DNS. In our case we are using
same HTTP url and port to post the curl request.
I don't even think starting 21 new threads should use 100% CPU for very long.
Name resolving should be mostly waiting for the first response and the
subsequent calls should be awfully quick - when the OS resolver has the
entries cached for the same names.
Are you absolutely sure it is related to name resolving?
So it should not spawn multiple threads for DNS resolution for same url. In
our case we are observing the DNS resolution is taking some time and during
that interval if another requests comes in it creates another threads and
thread count increases by 30 under load.
Yeah, this is a known area suitable for improvement as mentioned in the TODO
file here:
https://curl.haxx.se/docs/todo.html#Better_support_for_same_name_res
Question is, what is the best way to handle this problem.
Pretty much as the TODO item above says:
If a name resolve has been initiated for name NN and a second easy handle
wants to resolve that name as well, make it wait for the first resolve to end
up in the cache instead of doing a second separate resolve.
... but since it doesn't work like that now, we need to write code that makes
libcurl act like this!
--
/ daniel.haxx.se
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
