[
https://issues.apache.org/jira/browse/MESOS-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Mahler updated MESOS-7934:
-----------------------------------
Description:
LibeventSSLSocket can return 0 from send incorrectly, which leads the caller to
send the data twice!
See here:
https://github.com/apache/mesos/blob/1.3.1/3rdparty/libprocess/src/libevent_ssl_socket.cpp#L396-L398
In some particular cases, it's possible that the caller keeps getting back 0
and loops infinitely, blowing up the memory and OOMing the process.
One example is when a send occurs after a shutdown:
{code}
// This test was added due to an OOM issue: MESOS-7934.
TEST_F(SSLTest, ShutdownThenSend)
{
Clock::pause();
Try<Socket> server = setup_server({
{"LIBPROCESS_SSL_ENABLED", "true"},
{"LIBPROCESS_SSL_KEY_FILE", key_path().string()},
{"LIBPROCESS_SSL_CERT_FILE", certificate_path().string()}});
ASSERT_SOME(server);
ASSERT_SOME(server.get().address());
ASSERT_SOME(server.get().address().get().hostname());
Future<Socket> socket = server.get().accept();
Clock::settle();
EXPECT_TRUE(socket.isPending());
Try<Socket> client = Socket::create(SocketImpl::Kind::SSL);
ASSERT_SOME(client);
AWAIT_ASSERT_READY(client->connect(server->address().get()));
AWAIT_ASSERT_READY(socket);
EXPECT_SOME(Socket(socket.get()).shutdown());
// This loops forever!
AWAIT_FAILED(Socket(socket.get()).send("Hello World"));
}
{code}
was:
LibeventSSLSocket can return 0 from send incorrectly, which leads the caller to
send the data twice!
See here:
https://github.com/apache/mesos/blob/1.3.1/3rdparty/libprocess/src/libevent_ssl_socket.cpp#L396-L398
In some particular cases, it's possible that the caller keeps getting back 0
and loops infinitely, blowing up the memory and OOMing the process.
The following test case triggers the a send loop:
{code}
TEST_F(SSLTest, Bug)
{
Clock::pause();
Try<Socket> server = setup_server({
{"LIBPROCESS_SSL_ENABLED", "true"},
{"LIBPROCESS_SSL_KEY_FILE", key_path().string()},
{"LIBPROCESS_SSL_CERT_FILE", certificate_path().string()}});
ASSERT_SOME(server);
ASSERT_SOME(server.get().address());
ASSERT_SOME(server.get().address().get().hostname());
Future<Socket> socket = server.get().accept();
Clock::settle();
EXPECT_TRUE(socket.isPending());
Try<Socket> client = Socket::create(SocketImpl::Kind::SSL);
ASSERT_SOME(client);
AWAIT_ASSERT_READY(client->connect(server->address().get()));
AWAIT_ASSERT_READY(socket);
// Now write data to the pipe and splice to the file and the redirect hook.
string data =
"Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do "
"eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim "
"ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut "
"aliquip ex ea commodo consequat. Duis aute irure dolor in "
"reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla "
"pariatur. Excepteur sint occaecat cupidatat non proident, sunt in "
"culpa qui officia deserunt mollit anim id est laborum.";
// Create more data!
while (Bytes(data.size()) < Megabytes(10)) {
data.append(data);
}
// Construct response and send(server side).
const string buffer =
string("HTTP/1.1 200 OK\r\n") +
"Content-Length : " +
stringify(data.length()) + "\r\n" +
"\r\n" +
data;
Future<string> recvC = client->recv();
Future<string> recvS = Socket(socket.get()).recv();
Future<Nothing> send = Socket(socket.get()).send(buffer);
::shutdown(client->get(), SHUT_WR);
os::sleep(Milliseconds(1));
Socket(socket.get()).shutdown();
os::sleep(Seconds(10));
}
{code}
> OOM due to LibeventSSLSocket send incorrectly returning 0 after shutdown.
> -------------------------------------------------------------------------
>
> Key: MESOS-7934
> URL: https://issues.apache.org/jira/browse/MESOS-7934
> Project: Mesos
> Issue Type: Bug
> Components: libprocess
> Reporter: Benjamin Mahler
> Assignee: Benjamin Mahler
> Priority: Blocker
>
> LibeventSSLSocket can return 0 from send incorrectly, which leads the caller
> to send the data twice!
> See here:
> https://github.com/apache/mesos/blob/1.3.1/3rdparty/libprocess/src/libevent_ssl_socket.cpp#L396-L398
> In some particular cases, it's possible that the caller keeps getting back 0
> and loops infinitely, blowing up the memory and OOMing the process.
> One example is when a send occurs after a shutdown:
> {code}
> // This test was added due to an OOM issue: MESOS-7934.
> TEST_F(SSLTest, ShutdownThenSend)
> {
> Clock::pause();
> Try<Socket> server = setup_server({
> {"LIBPROCESS_SSL_ENABLED", "true"},
> {"LIBPROCESS_SSL_KEY_FILE", key_path().string()},
> {"LIBPROCESS_SSL_CERT_FILE", certificate_path().string()}});
> ASSERT_SOME(server);
> ASSERT_SOME(server.get().address());
> ASSERT_SOME(server.get().address().get().hostname());
> Future<Socket> socket = server.get().accept();
> Clock::settle();
> EXPECT_TRUE(socket.isPending());
> Try<Socket> client = Socket::create(SocketImpl::Kind::SSL);
> ASSERT_SOME(client);
> AWAIT_ASSERT_READY(client->connect(server->address().get()));
> AWAIT_ASSERT_READY(socket);
> EXPECT_SOME(Socket(socket.get()).shutdown());
> // This loops forever!
> AWAIT_FAILED(Socket(socket.get()).send("Hello World"));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)