[ 
https://issues.apache.org/jira/browse/MESOS-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Mahler updated MESOS-7934:
-----------------------------------
    Description: 
LibeventSSLSocket can return 0 from send incorrectly, which leads the caller to 
send the data twice!

See here: 
https://github.com/apache/mesos/blob/1.3.1/3rdparty/libprocess/src/libevent_ssl_socket.cpp#L396-L398

In some particular cases, it's possible that the caller keeps getting back 0 
and loops infinitely, blowing up the memory and OOMing the process.

One example is when a send occurs after a shutdown:

{code}
// This test was added due to an OOM issue: MESOS-7934.
TEST_F(SSLTest, ShutdownThenSend)
{
  Clock::pause();

  Try<Socket> server = setup_server({
      {"LIBPROCESS_SSL_ENABLED", "true"},
      {"LIBPROCESS_SSL_KEY_FILE", key_path().string()},
      {"LIBPROCESS_SSL_CERT_FILE", certificate_path().string()}});

  ASSERT_SOME(server);
  ASSERT_SOME(server.get().address());
  ASSERT_SOME(server.get().address().get().hostname());

  Future<Socket> socket = server.get().accept();

  Clock::settle();
  EXPECT_TRUE(socket.isPending());

  Try<Socket> client = Socket::create(SocketImpl::Kind::SSL);
  ASSERT_SOME(client);
  AWAIT_ASSERT_READY(client->connect(server->address().get()));

  AWAIT_ASSERT_READY(socket);

  EXPECT_SOME(Socket(socket.get()).shutdown());

  // This loops forever!
  AWAIT_FAILED(Socket(socket.get()).send("Hello World"));
}
{code}

  was:
LibeventSSLSocket can return 0 from send incorrectly, which leads the caller to 
send the data twice!

See here: 
https://github.com/apache/mesos/blob/1.3.1/3rdparty/libprocess/src/libevent_ssl_socket.cpp#L396-L398

In some particular cases, it's possible that the caller keeps getting back 0 
and loops infinitely, blowing up the memory and OOMing the process.

The following test case triggers the a send loop:

{code}
TEST_F(SSLTest, Bug)
{
  Clock::pause();

  Try<Socket> server = setup_server({
      {"LIBPROCESS_SSL_ENABLED", "true"},
      {"LIBPROCESS_SSL_KEY_FILE", key_path().string()},
      {"LIBPROCESS_SSL_CERT_FILE", certificate_path().string()}});

  ASSERT_SOME(server);
  ASSERT_SOME(server.get().address());
  ASSERT_SOME(server.get().address().get().hostname());

  Future<Socket> socket = server.get().accept();

  Clock::settle();
  EXPECT_TRUE(socket.isPending());

  Try<Socket> client = Socket::create(SocketImpl::Kind::SSL);
  ASSERT_SOME(client);
  AWAIT_ASSERT_READY(client->connect(server->address().get()));

  AWAIT_ASSERT_READY(socket);

  // Now write data to the pipe and splice to the file and the redirect hook.
  string data =
    "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do "
    "eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim "
    "ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut "
    "aliquip ex ea commodo consequat. Duis aute irure dolor in "
    "reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla "
    "pariatur. Excepteur sint occaecat cupidatat non proident, sunt in "
    "culpa qui officia deserunt mollit anim id est laborum.";

  // Create more data!
  while (Bytes(data.size()) < Megabytes(10)) {
    data.append(data);
  }

  // Construct response and send(server side).
  const string buffer =
    string("HTTP/1.1 200 OK\r\n") +
    "Content-Length : " +
    stringify(data.length()) + "\r\n" +
    "\r\n" +
    data;

  Future<string> recvC = client->recv();
  Future<string> recvS = Socket(socket.get()).recv();
  Future<Nothing> send = Socket(socket.get()).send(buffer);

  ::shutdown(client->get(), SHUT_WR);

  os::sleep(Milliseconds(1));

  Socket(socket.get()).shutdown();

  os::sleep(Seconds(10));
}
{code}


> OOM due to LibeventSSLSocket send incorrectly returning 0 after shutdown.
> -------------------------------------------------------------------------
>
>                 Key: MESOS-7934
>                 URL: https://issues.apache.org/jira/browse/MESOS-7934
>             Project: Mesos
>          Issue Type: Bug
>          Components: libprocess
>            Reporter: Benjamin Mahler
>            Assignee: Benjamin Mahler
>            Priority: Blocker
>
> LibeventSSLSocket can return 0 from send incorrectly, which leads the caller 
> to send the data twice!
> See here: 
> https://github.com/apache/mesos/blob/1.3.1/3rdparty/libprocess/src/libevent_ssl_socket.cpp#L396-L398
> In some particular cases, it's possible that the caller keeps getting back 0 
> and loops infinitely, blowing up the memory and OOMing the process.
> One example is when a send occurs after a shutdown:
> {code}
> // This test was added due to an OOM issue: MESOS-7934.
> TEST_F(SSLTest, ShutdownThenSend)
> {
>   Clock::pause();
>   Try<Socket> server = setup_server({
>       {"LIBPROCESS_SSL_ENABLED", "true"},
>       {"LIBPROCESS_SSL_KEY_FILE", key_path().string()},
>       {"LIBPROCESS_SSL_CERT_FILE", certificate_path().string()}});
>   ASSERT_SOME(server);
>   ASSERT_SOME(server.get().address());
>   ASSERT_SOME(server.get().address().get().hostname());
>   Future<Socket> socket = server.get().accept();
>   Clock::settle();
>   EXPECT_TRUE(socket.isPending());
>   Try<Socket> client = Socket::create(SocketImpl::Kind::SSL);
>   ASSERT_SOME(client);
>   AWAIT_ASSERT_READY(client->connect(server->address().get()));
>   AWAIT_ASSERT_READY(socket);
>   EXPECT_SOME(Socket(socket.get()).shutdown());
>   // This loops forever!
>   AWAIT_FAILED(Socket(socket.get()).send("Hello World"));
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to