Re: multi_socket and epoll example
On Thu, Aug 9, 2018 at 7:56 AM, Daniel Stenberg wrote:
> On Thu, 9 Aug 2018, James Read wrote:
>
> Everything seems to work fine. With a single URL and with multiple URLs.
>> The only issue I have is the throughput.
>>
>
> Could be vphiperfifo.c example issues.
>
> For example, I don't see how it handles API timeouts properly: the timeout
> that libcurl tells the CURLMOPT_TIMERFUNCTION callback should fire after
> that time and curl_multi_socket_action( ... CURL_SOCKET_TIMEOUT ...) should
> be called.
>
>
As far as I can see curl_multi_socket_action is called.
/* Update the timer after curl_multi library does it's thing. Curl will
* inform us through this callback what it wants the new timeout to be,
* after it does some work. */
static int multi_timer_cb(CURLM *multi, long timeout_ms, GlobalInfo *g)
{
struct itimerspec its;
CURLMcode rc;
fprintf(MSG_OUT, "multi_timer_cb: Setting timeout to %ld ms\n",
timeout_ms);
timerfd_settime(g->tfd, /*flags=*/ 0, , NULL);
if(timeout_ms > 0) {
its.it_interval.tv_sec = 1;
its.it_interval.tv_nsec = 0;
its.it_value.tv_sec = timeout_ms / 1000;
its.it_value.tv_nsec = (timeout_ms % 1000) * 1000;
timerfd_settime(g->tfd, /*flags=*/ 0, , NULL);
}
else if(timeout_ms == 0) {
rc = curl_multi_socket_action(g->multi,
CURL_SOCKET_TIMEOUT, 0,
>still_running);
mcode_or_die("multi_timer_cb: curl_multi_socket_action", rc);
}
else {
memset(, 0, sizeof(struct itimerspec));
timerfd_settime(g->tfd, /*flags=*/ 0, , NULL);
}
return 0;
}
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: multi_socket and epoll example
> > and make sure the application works correctly when using a small amount of >> transfers - perhaps a single one to start with - and then add more when >> you've confirmed that. > > Everything seems to work fine. With a single URL and with multiple URLs. The only issue I have is the throughput. --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: multi_socket and epoll example
On Tue, 7 Aug 2018, James Read wrote: As long as libcurl is built to use the threaded resolver (default) or the c-ares backend, it will do name resolving in a non-blocking manner. I just did a standard install using the latest git clone. Then it probably built with the threaded resolver. At the end of the configure run it says which one it will build with (and the curl -V output subtly reveals the info too). On another note. I've just tested ephiperfifo with a 100k domains. Throughput is below what I was expecting (hoping for). It started off at about 600KB/sec and then dropped to around 40KB/sec. There are like a million factors that all work together and in this statement you put them all together. First, I think you should understand the API properly and make sure that's used correctly. Then you should start small and get data from URLs that really will give you high speed transfers and make sure the application works correctly when using a small amount of transfers - perhaps a single one to start with - and then add more when you've confirmed that. -- / daniel.haxx.se --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: multi_socket and epoll example
On Tue, Aug 7, 2018 at 11:11 PM, Daniel Stenberg wrote: > On Tue, 7 Aug 2018, James Read wrote: > > What about DNS resolution? Is this already asynchronous or do I need to >> make some changes? >> > > As long as libcurl is built to use the threaded resolver (default) or the > c-ares backend, it will do name resolving in a non-blocking manner. > > -- > > / daniel.haxx.se > I just did a standard install using the latest git clone. On another note. I've just tested ephiperfifo with a 100k domains. Throughput is below what I was expecting (hoping for). It started off at about 600KB/sec and then dropped to around 40KB/sec. Any ideas how I could optimise the code for performance gains. I ideally want to max out my bandwidth (don't need to worry about other users). --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: multi_socket and epoll example
On Tue, 7 Aug 2018, James Read wrote: What about DNS resolution? Is this already asynchronous or do I need to make some changes? As long as libcurl is built to use the threaded resolver (default) or the c-ares backend, it will do name resolving in a non-blocking manner. -- / daniel.haxx.se --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: multi_socket and epoll example
On Tue, Aug 7, 2018 at 9:37 PM, Daniel Stenberg wrote: > On Tue, 7 Aug 2018, James Read via curl-library wrote: > > Does anyone have a working example using multi_socket with epoll? >> > > Josh Bialkowski provided one not too long ago, available here: > > https://curl.haxx.se/libcurl/c/ephiperfifo.html > > -- > > / daniel.haxx.se > Thanks. That seems to be what I'm looking for. What about DNS resolution? Is this already asynchronous or do I need to make some changes? --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: multi_socket and epoll example
On Tue, 7 Aug 2018, James Read via curl-library wrote: Does anyone have a working example using multi_socket with epoll? Josh Bialkowski provided one not too long ago, available here: https://curl.haxx.se/libcurl/c/ephiperfifo.html -- / daniel.haxx.se --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
