On Thu, Aug 10, 2017 at 6:33 AM, Daniel Stenberg <dan...@haxx.se> wrote:

> On Wed, 9 Aug 2017, fan fan via curl-library wrote:
>
>    In my application, I using libcurl to send 40 HTTP requests in every
>> one second. As I test, this consume much CPU, my application such as:
>>
>>  main thread for sending request and using epoll listen fds whether read
>> or write.
>>
>>    while(running) {
>>          int ret = epoll_wait(epollFd, events, maxEventNum, 10); //10ms
>>
>
> Unconditionally returning every 10 milliseconds like this will make you
> spend a lot of CPU in vein. Don't do that. libcurl tells you how long it
> wants you to wait at most. No need to spin around a lap when there's
> nothing for libcurl to do.
>
> Do you also call libcurl to tell there's a timeout if the 10 milliseconds
> lapse or when do you do that?


   [Fan] Maybe this  is not root cause, because when no http request send,
CPU almost 0.1%. of course I need to expand waiting  time.
           this epoll_wait for listening every easy handle fd, I
set CURLMOPT_SOCKETFUNCTION, when callback, I add this fd to epoll events.
           of course I set CURLMOPT_TIMERFUNCTION, when timeout, I
call curl_multi_socket_action
again.

>
>
>          for (i = 0; i < ret; i++) {
>>               if (events[i].events & EPOLLIN) {
>>                      curl_multi_socket_action   // set CURL_CSELECT_IN
>>                      curl_multi_info_read  // for reading
>>
>
> I think you should put the curl_multi_info_read() call outside of the loop
> checking for activities, since it might not have anything to return and
> then its a bit of a waste to call it up to 40 times.
>
> But sure, curl_multi_socket_action() may of course consume CPU since
> that's the function that will handle the traffic on the socket you give it
> (and possibly on a few others if necessary).


   [Fan] So, when do I call  curl_multi_info_read?
            I refer to:
            https://curl.haxx.se/libcurl/c/hiperfifo.html
            when socket have something to read, call curl_multi_info_read,
is it correct?

>
>
> As I test,  send 100 or 200 or 300 or 400 or 500 in parallel  maybe consume
>> cpu 100%.
>>
>
> On modern hardware that shouldn't happen.

    [Fan] Agree

>
>
> I found there are too much EPOLLIN event, this consume much cpu.
>>
>
> What do you mean with too much EPOLLIN event? If libcurl tells you to wait
> for that, shouldn't you then be glad you get it so that you can download
> the data?


   [Fan] I add socket fd to epoll event when CURLMOPT_SOCKETFUNCTION
callback.
             If there has a lot of easy handle in processing, trigger
frequently.
             CURLMOPT_TIMERFUNCTION not about epoll fd, is it right?
             CURLMOPT_TIMERFUNCTION  only indicating whether time out or
not.

>
>
> 1. Does my use libcurl wrong?
>>
>
> Ineffective at least. But you also didn't show the whole program so its
> not possible to tell the whole story.

   [Fan] Agree

>
>
> 2. Have any performance measurement for libcurl process in parallel?
>>
>
> It should easily handle thousands of parallel connections, CPU wise. Very
> likely tens of thousands too. Of course they may have to struggle with each
> other for other resources (like bandwidth).

   [Fan]Awesome

>
>
> --
>
>  / daniel.haxx.se
>
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Reply via email to