Andrew Kyle Purtell created ZOOKEEPER-5133:
----------------------------------------------

             Summary: Native Rust client and drop-in C/C++ client replacement
                 Key: ZOOKEEPER-5133
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5133
             Project: ZooKeeper
          Issue Type: New Feature
            Reporter: Andrew Kyle Purtell
            Assignee: Andrew Kyle Purtell


We have developed a new Rust ZooKeeper client and would like to contribute it.

A single, shared protocol and I/O core produces two deliverables:

*Drop-in C/C++ Replacement Libraries (`libzookeeper_mt` and 
`libzookeeper_st`)*: Binary- and source-compatible replacements for the legacy 
C client libraries. They expose identical C headers, identical symbol tables, 
and identical wire and callback semantics, diverging only where documented to 
remediate memory corruption, data races, undefined behavior, or protocol 
defects identified in the legacy C client implementation.

*Native Rust Client (`apache-zookeeper` and `apache-zookeeper-recipes`)*: A 
strongly typed, idiomatic Rust package supporting both asynchronous (via Rust's 
well known `tokio`) and synchronous execution runtimes, persistent and 
recursive watches, transaction batching, and higher level coordination 
primitives such as distributed locks, leader latches, and tree caches that will 
interoperate in a wire compatible way with Apache Curator.

The primary objective is the elimination of memory safety and concurrency 
defect classes endemic to C systems programming, such as buffer overreads on 
untrusted network frames, use-after-free during concurrent session teardown, 
lock inversions across reentrant application callbacks, and uninitialized 
memory exposure. We achieve this without requiring modifications to existing 
C/C++  ZooKeeper client applications or runtime configurations. Both 
deliverables integrate TLS natively via `rustls`, removing OpenSSL from the 
dependency and linkage graph.

The legacy C client remains intact and untouched in the repository and 
continues to serve as the default build target unless the Rust core is selected 
during the build configuration process.

*Highlights*

*Safety and Formal Verification:* Across all shipped crates, "unsafe" Rust is 
forbidden everywhere except a single audited Foreign Function Interface (FFI) 
module. Compile time ownership and lock exclusion proof checking ensure memory- 
and lock-safety at compile time, and these assurances are further reinforced 
with formal verification using loom's concurrency permutation model checker, 
Miri's strict provenance abstract machine interpretation, and continuous 
property based fuzzing using cargo-fuzz.

*Performance Superiority:* In a comprehensive head-to-head evaluation, the Rust 
client outperforms the legacy C client with a 95% confidence interval in 898 of 
953 cases (94.2%), achieves statistical parity in 33 cases (3.5%), and trails 
in 22 cases (2.3%). Client CPU consumption is lower than the legacy C client in 
284 of 294 measured cases (96.6%). The performance of the multithreaded drop-in 
libzookeeper_mt is better than or at statistical parity with the C client in 
99.7% of the test cases. The single threaded drop-in libzookeeper_st improves 
small operation latency across all cases by up to 20%. Under high concurrency 
pipelining with a window of 1,000 concurrent requests, throughput scales up to 
32x with the drop-in MT library and up to 54x with Rust's Tokio, compared to 
the legacy C MT library's measured scalability at 1x. TLS session establishment 
is 27% faster than compared with the C client.

*Superior Reliability:* In concurrent close stress testing (36,000 
close-under-load iterations), the Rust drop-in libraries recorded zero lost, 
duplicate, or stale completions, resolving a long standing teardown race 
condition in the legacy C library, which often fails this test.

*Overview*

At the foundation of the Rust client stack sits a serialization tier, which 
houses the generated wire data structures and encoding codecs derived directly 
from the protocol schema. This foundation performs no network transport 
operations and validates buffer boundaries on every read, using an incremental 
frame decoder that allocates memory strictly as data arrives from the network 
to protect against malformed frame headers and exhaustion attacks.

Directly above serialization lies the protocol core ("IOCore"), designed as a 
deterministic state machine operating entirely independent of sockets, 
execution threads, or ambient clocks. By consuming raw byte streams, abstract 
request representations, and discrete timesteps provided by the driver layer, 
the state machine produces outbound wire streams, ordered application event 
queues, and connection deadlines. All mutable session context, including 
request pipelines, transaction counters, keepalive intervals, server resolution 
tracking, watch tables, authentication state, and staged connection teardown, 
is contained exclusively within this single-owner protocol machine.

The driver tier bridges the IOCore to the operating system network facilities, 
multiplexed polling engines, wakeup primitives, connectivity probes, and TLS 
sessions. A common execution pipeline compiles into four distinct runtime 
environments selected by feature gates: a single threaded runtime matching 
legacy single-threaded semantics, a multithreaded runtime matching legacy 
multithreaded semantics, a single threaded runtime driven directly by caller 
event loops, and a tokio-compatible runtime for async applications.

For Rust-native applications, a high level client layer provides typed request 
handling, asynchronous pipelining, watch notification streams, and builder 
interfaces, complemented by Curator-compatible coordination recipes including 
distributed mutual exclusion, leader election, and hierarchical tree caching. 
For C-based applications, a foreign function interface bridges the Rust side to 
a thin C/C++ compatibility shim, exposing a C ABI identical to today's C 
client, while ensuring the compatibility bridge introduces no overheads.

The shared protocol state machine and I/O pipeline execute unified logic across 
all deliverables without dynamic branching. Strict boundaries are maintained 
across the compilation units, forbidding unsafe operations throughout the 
codebase with the exception of the isolated foreign function bridge.

The build harness includes comprehensive automated verification suites 
including a deterministic fault-injection framework, scripted mock servers, 
differential cross-checks against legacy implementations, binary layout probes, 
and continuous fuzz testing to provide confidence that the Rust client 
implementation is both as memory- and concurrency-safe as promised and fully 
compatible with legacy client behavior.

*Architecture*

The foundational invariant is that application-defined code never executes 
while a Rust stack frame remains active on the calling thread. This rule 
insulates user logic, including asynchronous completion callbacks, watch 
notifications, authentication completions, custom logging hooks, and security 
mechanism exchanges, from the IOCore. Isolation is realized through a 
cooperative step-and-drain protocol. When the outer runtime invokes the core 
engine, the core processes pending transport and protocol state, populates an 
outbox queue with notifications, and returns control after unwinding all Rust 
frames and releasing internal locks. The surrounding compatibility shim then 
iterates through the outbox in native host frames, delivering results to 
application callbacks or managing thread coordination. When protocol 
progression depends on the observable outcome of an application notification, 
such as a session closure or authentication state inspection initiated from 
within a callback, the engine halts at a synchronization barrier and yields, 
employing a corountine/continuation model. Once the outer layer completes event 
dispatch, execution resumes seamlessly from the saved corountine state.

The isolation strategy adapts naturally across the various runtime 
environments. In the multithreaded compatibility library, the two-thread 
architecture of the legacy client is preserved: one background thread drives 
the network I/O loop while a separate dispatch thread handles completion and 
watch events in strict wire order. Synchronous API calls submit asynchronous 
requests internally and wait on native condition variables outside all Rust 
frames, populating caller buffers only upon wake-up, while any attempt to issue 
synchronous requests from the network thread is rejected immediately to 
eliminate deadlock. Rather than blocking within the engine, thread join or 
detach commands are yielded to the outer layer, allowing foreign threads, 
completion callbacks, or network callbacks to terminate cleanly without 
deadlocking. In single-threaded environments, no background threads are 
created. The application drives I/O directly from its own event loop, 
processing connection events, outbound flushes, and inbound frames in lockstep 
and delivering completions inline at historical synchronization points. In 
Rust-native environments, requests are validated and enqueued synchronously at 
call time to ensure submission order matches wire transmission order, routing 
responses to asynchronous futures, one-shot watch channels, and persistent 
watch streams with integrated backpressure and consumer lag tracking.

The drop-in replacement libraries combine a compiled static archive of the 
bridge crate with a thin compatibility shim and unmodified legacy serialization 
units. Public data structures remain compiled by the platform C compiler to 
guarantee exact binary layout and ensure dynamic linking functions without 
divergence. Callback dispatch routines run entirely within host stack frames, 
ensuring any non-local jump or exception unwinds only native frames while 
safely returning unconsumed events to the core. Thread creation, joins, detach 
operations, and synchronous waits similarly reside in the outer layer so 
cancellation points never span active core frames. Exported global symbols and 
Cyrus SASL glue remain isolated within dedicated C units so that external C 
header details do not leak into the Rust driver.

*Verification*

A battery of verification tools guards against regressions, undefined behavior, 
and concurrency hazards. Abstract machine interpretation under strict pointer 
provenance rules systematically exercises unit and integration paths to confirm 
the total absence of dangling references, uninitialized memory accesses, and 
aliasing violations across the driver core and foreign interface bridge. 
Concurrency permutation model checking exhaustively explores thread scheduling 
interleavings across queue implementations, shutdown sequences, and handle 
lifecycle operations to ensure synchronization freedom. Continuous 
coverage-guided fuzzing targets serialization formats, incremental frame 
reconstruction, transaction batches, stream decoders, authentication 
handshakes, and cryptographic processing pipelines under address and leak 
sanitization. Structural layout checkers compile native probe programs across 
all supported build configurations to assert that every bridge type and public 
structure matches expected target alignment and padding. Specialized memory 
scans verify that sensitive cryptographic key material and session credentials 
are wiped from deallocated heap memory. These verification passes complement 
extensive deterministic simulation suites, property-based tests, and 
integration test suites spanning thousands of test scenarios.

This verification strategy eliminates whole classes of memory and concurrency 
vulnerabilities present in the C client implementation. Invalid or 
uninitialized client handle references are intercepted at entry boundaries and 
converted into clean argument rejections rather than triggering null-pointer 
faults. Buffer operations on data payloads, string collections, and access 
control lists validate dimensions and capacities before memory copies occur, 
eliminating heap corruption and buffer overreads. Framing logic guarantees that 
malformed or truncated network frames result in graceful connection termination 
rather than parsing beyond allocated packet bounds or exposing uninitialized 
stack data to user callbacks. Transaction batch failures consistently 
initialize all caller result slots with defined error codes rather than leaving 
uninitialized memory in place. Shared process-wide counters and credential 
buffers are fully encapsulated within isolated per-session states, eliminating 
unsynchronized cross-thread mutations. Internal locks are never held while 
invoking application notifications, preventing deadlock when user callbacks 
make reentrant client calls. Host failover state machines advance 
monotonically, eliminating cursor progression defects that previously bypassed 
ensemble nodes during connection retries. Finally, teardown sequences 
coordinate thread termination and event delivery so that queued completions are 
guaranteed to reach waiting callers rather than being dropped during shutdown.

Applications that employ external authentication mechanisms across multiple 
threads must continue to configure thread synchronization within the 
third-party library, as those external caches reside outside the client 
boundary.

*Compatibility*

An existing application built against the original C client can link directly 
against or dynamically load the replacement libraries with no modifications to 
application code, build configurations, or operational deployments. Complete 
fidelity is maintained across all observable dimensions of the programming 
interface: installed public headers are byte-for-byte identical, platform 
library names and major shared object versions match historical targets, and 
exported symbol sets maintain exact parity in symbol names, function 
signatures, data object sizes, and mutability without introducing extraneous 
symbols or omitting existing ones. Protocol framing, request sequencing, 
timeout limits, hash-table watcher traversal order, error return codes, 
operating system error hints, callback delivery sequences, and formatted 
diagnostic messages similarly reflect legacy behaviors. Runtime execution 
characteristics are fully preserved, running asynchronous completions and watch 
events on a dedicated completion thread in multithreaded builds while 
dispatching them inline during event loop processing in single-threaded builds.

Symbol visibility boundaries are strictly enforced during linking, ensuring 
internal helpers and runtime components remain fully unexported while matching 
the historical public inventory across every build configuration. In every 
build variant, a fixed subset of the public surface represents exported data 
objects, split between read-only constants and mutable configuration variables. 
The legacy build infrastructure requires only minimal adjustments to delegate 
to the new core.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to