Shizuo Fujita created CASSCPP-14:
------------------------------------
Summary: Use-after-free: re-binding a CassStatement during an
in-flight async execution
Key: CASSCPP-14
URL: https://issues.apache.org/jira/browse/CASSCPP-14
Project: Apache Cassandra C/C++ driver
Issue Type: Bug
Reporter: Shizuo Fujita
Attachments: repro-uuid-use-after-free.tar.gz
If an application binds new values into a {{CassStatement}} after
{{cass_session_execute()}} has returned but while that execution is still in
flight, the driver reads freed memory and may send freed heap contents to the
server. Nothing in the API documentation forbids this pattern:
{{cass_session_execute()}} returns immediately, the thread-safety notes only
say objects must not be used {_}from multiple application threads{_}, and here
all
API calls happen on one application thread. The official examples even free the
statement right after {{{}execute(){}}}, which suggests the driver has taken
everything it needs — it has not.
h3. Root cause
# {{cass_session_execute()}} enqueues the request; the wire encoding happens
later, on an IO thread, at socket-write time: {{Connection::write}} ->
{{SocketWriteBase::write}} -> {{RequestCallback::encode}} ->
{{Statement::encode_values}} (src/statement.cpp:438), which reads the
statement's {{AbstractData::Element}} buffers via plain, unsynchronized
{{Buffer}} copies (src/buffer.hpp {{{}Buffer::copy{}}}).
# Meanwhile {{cass_statement_bind_*_by_name}} on the application thread
replaces
the same {{Element}} ({{{}AbstractData::set{}}} -> {{Element::operator=}} ->
{{{}Buffer::operator={}}}), which dec_refs and frees the previous value's
{{{}RefBuffer{}}}.
# {{{}Buffer{}}}'s copy of the size/pointer pair and the {{RefBuffer}} refcount
hand-off are not atomic, so the IO thread can end up holding (and writev-ing)
a freed {{{}RefBuffer{}}}, or both threads dec_ref the same buffer (double
free).
The heap path is taken only when the serialized value is strictly larger than
{{Buffer::FIXED_BUFFER_SIZE}} (16 bytes); {{Buffer::copy}} compares with
{{{}size > FIXED_BUFFER_SIZE{}}}, so 16 bytes still uses the inline {{fixed}}
buffer and 17+ bytes allocates a {{{}RefBuffer{}}}. Serialized size includes the
4-byte length prefix, giving, for the columns the original reporter binds:
||bound value||serialized size||buffer||
|null|4|inline|
|boolean|4 + 1 = 5|inline|
|int|4 + 4 = 8|inline|
|bigint|4 + 8 = 12|inline|
|short text ("hello i")|4 + 7 = 11|inline|
|*uuid*|*4 + 16 = 20*|*heap (RefBuffer)*|
A {{uuid}} is 20 bytes, so it is the only value in the original tests that
takes the heap path, and thus the only one whose race corrupts the heap; the
inline values instead risk silently sending torn bytes (a data-integrity
issue). This threshold is why the bug was originally observed as "crashes only
when a UUID column is used" (DataStax JIRA CPP-996 /
[https://github.com/Watson1978/ilios/issues/12], where glibc intermittently
aborted with {{{}tcache_thread_shutdown(): unaligned tcache chunk detected{}}}).
Any value serializing to >16 bytes reproduces identically — e.g. a text column
with a 13+ byte payload (4 + 13 = 17); the attached {{reuse-text}} mode uses a
32-byte string (36 bytes serialized) — so the bug is not UUID-specific.
h3. Steps to reproduce
Attached {{uuid_async_repro.c}} is self-contained (creates its own
keyspace/table). A {{Makefile}} fetches + builds the driver with a sanitizer
and builds the reproducer; with a local Cassandra running, {{make && make run}}
reproduces. Core loop:
{code:c}
CassStatement* stmt = cass_prepared_bind(prepared); /* created once */
for (;;) {
for (int i = 0; i < 50; i++) {
cass_statement_bind_int64_by_name(stmt, "id", i);
cass_uuid_from_string(random_uuid, &uuid);
cass_statement_bind_uuid_by_name(stmt, "uuid", uuid); /* re-bind */
futures[i] = cass_session_execute(session, stmt); /* 50 in flight */
}
/* wait all futures, repeat */
}
{code}
Build the driver and the program with {{{}-fsanitize=address -g{}}}; crashes
within
~1500 executions in every attempt (4/4 on 2.17.1, 2/2 on master). Control modes
in the same file: binding null instead of a uuid -> clean; creating a fresh
statement per execution ({{{}fresh-uuid{}}}) -> clean.
h3. Expected vs actual
* *Expected:* either the statement's values are snapshotted by
{{cass_session_execute()}} (so later re-binding is safe), or the documentation
clearly states a statement must not be modified until its execution future
completes.
* *Actual:* use-after-free / double free; freed heap memory can be written to
the socket. Without sanitizers this surfaces as rare, hard-to-diagnose glibc
heap-corruption aborts or SEGVs far from the cause (often at session teardown).
h3. AddressSanitizer (2.17.1, excerpt)
{noformat}
==ERROR: AddressSanitizer: heap-use-after-free ... READ of size 20 ... thread T2
#1 writev
#4 uv_write2
#5 SocketWrite::flush() src/socket.cpp:58
#6 ConnectionPool::flush() src/connection_pool.cpp:113
...
freed by thread T0 here:
#1 Buffer::copy(Buffer const&) src/buffer.hpp:201
#3 AbstractData::Element::operator=(Element&&) src/abstract_data.hpp:44
#4 AbstractData::set(unsigned long, CassUuid_) src/abstract_data.hpp:110
#6 cass_statement_bind_uuid_by_name src/statement.cpp:204
previously allocated by thread T0 here:
#3 RefBuffer::create(unsigned long) src/ref_counted.hpp:149
#6 encode_with_length(CassUuid_) src/encode.hpp:115
#9 cass_statement_bind_uuid_by_name src/statement.cpp:204
{noformat}
h3. ThreadSanitizer (2.17.1, racing pair)
{noformat}
Read of size 8 by thread T2 (IO thread):
#0 Buffer::copy(Buffer const&) src/buffer.hpp:194
#2 AbstractData::Element::get_buffer() src/abstract_data.cpp:116
#3 Statement::encode_values(...) src/statement.cpp:438
#5 RequestCallback::encode(...) src/request_callback.cpp:81
#6 SocketWriteBase::write(...) src/socket.cpp:236
Previous write of size 8 by main thread:
#0 Buffer::copy(Buffer const&) src/buffer.hpp:195
#2 AbstractData::Element::operator=(Element&&) src/abstract_data.hpp:44
#5 cass_statement_bind_uuid_by_name src/statement.cpp:204
{noformat}
TSan also reports the symmetric double-free: main thread dec_ref on a
{{RefBuffer}} already freed by the IO thread in
{{{}SocketWriteBase::clear(){}}}/{{{}handle_write{}}} after the write completed.
h3. Suggested fix directions
* Snapshot the parameter buffers on the application thread, inside
{{cass_session_execute()}} (a {{Buffer}} copy per element is cheap — inc_ref
for heap buffers, <=16-byte memcpy for inline ones); or
* deep-copy the {{Statement}} at submission; or, at minimum,
* document explicitly (cassandra.h {{CassStatement}} docs + thread-safety
section) that a statement must not be modified between submission and future
completion, and note that this also silently produces torn inline values
(<=16 bytes) — a data-integrity issue, not just a crash.
h3. History
First reported (symptom only) as DataStax JIRA CPP-996 (Feb 2024, not migrated
to ASF JIRA and no longer accessible), from
[https://github.com/Watson1978/ilios/issues/12] — a Ruby binding that re-binds
one statement across concurrent {{execute_async}} calls. That project had to
stop binding UUID values in its tests to avoid the crashes; the attached
reproducer distills their exact pattern to C.
Related to [https://datastax-oss.atlassian.net/browse/CPP-996]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
