On 01/11/23 18:32, Dmitry Karpov via curl-library wrote:
-----Original Message-----
From: Luis Carlos Chalaca Figueira<luisccfigue...@gmail.com>
Sent: Wednesday, November 1, 2023 9:18 AM
To: libcurl development<curl-library@lists.haxx.se>
Cc: Dmitry Karpov<dkar...@roku.com>
Subject: [EXTERNAL] Re: "Could not resolve host" error only on the multi
interface
On 31/10/23 18:32, Dmitry Karpov via curl-library wrote:
I would recommend to try without explicitly setting DNS server and timeout -
just to use the default DNS system settings.
Maybe, the "8.8.8.8" is not routable in your setup.
Also, I would run Wireshark or tcpdump to see if your DNS queries/response were
actually sent/received on the wire.
Thanks,
Dmitry Karpov
-----Original Message-----
From: curl-library<curl-library-boun...@lists.haxx.se> On Behalf Of
Luis Carlos Chalaca Figueira via curl-library
Sent: Tuesday, October 31, 2023 10:08 AM
To:curl-library@lists.haxx.se
Cc: Luis Carlos Chalaca Figueira<luisccfigue...@gmail.com>
Subject: [EXTERNAL] "Could not resolve host" error only on the multi
interface
Hello everybody,
I'm building a web crawler.
Started with python but the need for speed led me to libcurl.
The obstacle I'm having is: when using the multi interface a big part of the the
connections cloe because of the error "Could not resolve host:"
What I've tried:
1 - Confirmed that all URLs are correct.
2 - Confirmed that using the easy interface URLs are resolved.
3 - Confirmed that other programs like dig, resolvectl, nslookup can resolve
URLs that multi interface couldn't.
4 - Set a "reputable" DNS server doing: curl_easy_setopt(handles[i], CURLOPT_DNS_SERVERS,
"8.8.8.8");.
5 - In the file /etc/nsswitch.conf, line "hosts: files nDNS[NOTFOUNDED=return]
dns" I deleted [NOTFOUNDED=return].
6 - I've turned verbose on - Doesn't help me much.
I don't know what to do more as it seems a problem inside the library, so i
posted here before going digging in the library.
System info:
O.S. - Ubuntu 22.04.3 LTS
Harware - HP HP Pavilion Laptop 14-ce3xxx
Processor - Intel® Core™ i7-1065G7 CPU @ 1.30GHz × 8
libcurl version - 7.81.0
Attached I send the source code file I'm using do debug this problem.
Best Regards,
Luis Figueira
Dimitry,
Thanks for the response,
However it looks like my e-mail was not clear, maybe because English is my
second language.
Those 6 steps I took were made after I realized that most requests names could
not be resolved therefore they are not the problem. However I've reset all of
them to the default state.
I've used tcpdump and the output only revealed what the the system sends
packets only to one address, the stub address (Which discovered by using
nslookup). Below an exctract from the output:
19:09:50.875611 IP6 ubuntu.39274 > 2001:8a0:776c:b200::1.domain: 53886+ AAAA?
e10952.a.akamaiedge.net. (41)
19:09:50.876703 IP6 ubuntu.51246 > 2001:8a0:776c:b200::1.domain: 27951+ AAAA?
e-cultura.pt. (30)
19:09:50.880193 IP6 ubuntu.58635 > 2001:8a0:776c:b200::1.domain: 52688+ AAAA?
orientacao-vocacional.com.pt. (46)
19:09:50.881306 IP6 2001:8a0:776c:b200::1.domain > ubuntu.47511: 24563
0/1/0 (104)
19:09:50.882880 IP6 2001:8a0:776c:b200::1.domain > ubuntu.35687: 64715
0/1/0 (110)
19:09:50.883430 IP6 2001:8a0:776c:b200::1.domain > ubuntu.43052: 35782
0/1/0 (106)
What I will do now is ask curl to give me every type of error code for each
request and see if I can get what is happening.
--
Cumprimentos,
Luis Figueira
" What I will do now is ask curl to give me every type of error code for each
request and see if I can get what is happening."
I don't think libcurl can easily do that. It works with different DNS backends
(i.e. c-ares or libc) and they don't provide the same error information.
In the past, I proposed to add an "info" option which would return a resolver
error code, so it could be logged on remote devices and reported,
but the proposal was rejected because of differences between multiple resolver
backends.
Thanks,
Dmitry
It looks like it might be the fact that my code is multi-threaded that
is creating these errors, because using both interfaces single-threaded
does not yield any error but threading any of them leads to errors.
Therefore I wave 2 questions:
1 - About multithreaded name resolving: - I would say Ubuntu 22.04.3 LTS
provides thread safe name resolving system calls ? - How can I find out?
2 - About multithreading TLS: - folowed the threaded-ssl example but got
the compiling error:
/usr/bin/ld: crawler.o: undefined reference to symbol
'CRYPTO_malloc@@OPENSSL_3.0.0'
/usr/bin/ld: /lib/x86_64-linux-gnu/libcrypto.so.3: error adding symbols:
DSO missing from command line
collect2: error: ld returned 1 exit status
make: *** [Makefile:5: crawler] Error 1
What am I missing?
--
Cumprimentos,
Luis Figueira
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
