curl_multi_socket_action, epoll and HTTP redirect

[Erik Johansson]

Hi,

While trying to use curl_multi_socket_action together with epoll I
noticed that it didn't work in combination with HTTP redirect.

When a socket is first connected, CURLMOPT_SOCKETFUNCTION is called
and the socket is added to the epoll set. Later on when cURL connects
to the HTTP server and sees the redirect it closes the socket (which
makes the kernel remove the socket from the epoll set) and opens a
new. Unfortunately this new socket has the same number as the old, and
thus the socket function callback is never called to add it to the
epoll set.

My current work-around is to keep track of all sockets added in the
socket function callback and also listen to CURLOPT_SOCKOPTFUNCTION to
know when a new socket has been added that wasn't notified in the
socket callback. This works, but seems a bit hackish.

So, have I misunderstood something and it actually works if done right
or is this an issue in cURL?

I'm attaching two files that I have used to reproduce this problem:
httpserver.py - Starts two servers on two different ports where one
redirects to the other.
curlepoll.c - Test program that reproduce the problem.

Output from a test run:

# ./httpserver.py
Connect to http://127.0.0.1:38209 to be redirected to http://127.0.0.1:47067
...

Direct connection works as expected:
# ./curlepoll http://127.0.0.1:47067
Using libcurl/7.29.0 OpenSSL/1.0.1c zlib/1.2.7 libidn/1.25 librtmp/2.3
Hello Port 47067!
bye bye

But connecting to the redirect server hangs:
# ./curlepoll http://127.0.0.1:38209
Using libcurl/7.29.0 OpenSSL/1.0.1c zlib/1.2.7 libidn/1.25 librtmp/2.3
^C

strace shows the problem:
# strace -esocket,connect,close,epoll_ctl ./curlepoll http://127.0.0.1:38209
...
Using libcurl/7.29.0 OpenSSL/1.0.1c zlib/1.2.7 libidn/1.25 librtmp/2.3
...
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(38209),
sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now
in progress)
epoll_ctl(3, EPOLL_CTL_ADD, 4, {EPOLLIN, {u32=4, u64=4}}) = 0
close(4)                                = 0
...
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(47067),
sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now
in progress)
^C

// Erik

-- 
Erik Johansson
Home Page: http://ejohansson.se/
PGP Key: http://ejohansson.se/erik.asc

#include <assert.h>
#include <curl/curl.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/epoll.h>
#include <unistd.h>

void error(const char* string)
{
  perror(string);
  exit(1);
}

int epollFd;
int timeout = -1;

int socketCallback(CURL* easy, curl_socket_t fd, int action, void* u, void* s)
{
  struct epoll_event event;
  event.events = 0;
  event.data.fd = fd;

  if (action == CURL_POLL_REMOVE) {
    int res = epoll_ctl(epollFd, EPOLL_CTL_DEL, fd, &event);
    if (res == -1 && errno != EBADF)
      perror("epoll_ctl(DEL)");
    return 0;
  }

  if (action == CURL_POLL_IN || action == CURL_POLL_INOUT)
    event.events |= EPOLLIN;
  if (action == CURL_POLL_OUT || action == CURL_POLL_INOUT)
    event.events |= EPOLLOUT;

  if (event.events != 0) {
    int res = epoll_ctl(epollFd, EPOLL_CTL_ADD, fd, &event);
    if (res == -1)
      res = epoll_ctl(epollFd, EPOLL_CTL_MOD, fd, &event);
    if (res == -1)
      error("epoll_ctl(MOD)");
  }

  return 0;
}

int timerCallback(CURLM* multi, long timeout_ms, void* u)
{
  timeout = timeout_ms;
  return 0;
}

int main(int argc, char** argv)
{
  // argv[1] == URL
  assert(argc == 2);

  epollFd = epoll_create(1);
  if (epollFd == -1)
    error("epoll_create");

  curl_global_init(CURL_GLOBAL_ALL);
  printf("Using %s\n", curl_version());

  CURL* easy = curl_easy_init();
  curl_easy_setopt(easy, CURLOPT_URL, argv[1]);
  curl_easy_setopt(easy, CURLOPT_FOLLOWLOCATION, 1);

  CURLM* multi = curl_multi_init();
  curl_multi_setopt(multi, CURLMOPT_SOCKETFUNCTION, socketCallback);
  curl_multi_setopt(multi, CURLMOPT_TIMERFUNCTION, timerCallback);

  curl_multi_add_handle(multi, easy);

  int running_handles = 1;
  while (running_handles > 0) {
    struct epoll_event event;
    int res = epoll_wait(epollFd, &event, 1, timeout);
    if (res == -1)
      error("epoll_wait");
    else if (res == 0) {
      curl_multi_socket_action(multi, CURL_SOCKET_TIMEOUT, 0, &running_handles);
    }
    else {
      curl_multi_socket_action(multi, event.data.fd, 0, &running_handles);
    }
  }

  curl_global_cleanup();
  close(epollFd);

  printf("bye bye\n");
  return 0;
}

httpserver.py
Description: Binary data

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html