Heads up folks! ZooKeeper Rust client is on the way …
> On Nov 17, 2026, at 7:51 PM, Andrew Kyle Purtell (Jira) via dev > <[email protected]> wrote: > > Andrew Kyle Purtell created ZOOKEEPER-5133: > ---------------------------------------------- > > Summary: Native Rust client and drop-in C/C++ client replacement > Key: ZOOKEEPER-5133 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5133 > Project: ZooKeeper > Issue Type: New Feature > Reporter: Andrew Kyle Purtell > Assignee: Andrew Kyle Purtell > > > We have developed a new Rust ZooKeeper client and would like to contribute it. > > A single, shared protocol and I/O core produces two deliverables: > > *Drop-in C/C++ Replacement Libraries (`libzookeeper_mt` and > `libzookeeper_st`)*: Binary- and source-compatible replacements for the > legacy C client libraries. They expose identical C headers, identical symbol > tables, and identical wire and callback semantics, diverging only where > documented to remediate memory corruption, data races, undefined behavior, or > protocol defects identified in the legacy C client implementation. > > *Native Rust Client (`apache-zookeeper` and `apache-zookeeper-recipes`)*: A > strongly typed, idiomatic Rust package supporting both asynchronous (via > Rust's well known `tokio`) and synchronous execution runtimes, persistent and > recursive watches, transaction batching, and higher level coordination > primitives such as distributed locks, leader latches, and tree caches that > will interoperate in a wire compatible way with Apache Curator. > > The primary objective is the elimination of memory safety and concurrency > defect classes endemic to C systems programming, such as buffer overreads on > untrusted network frames, use-after-free during concurrent session teardown, > lock inversions across reentrant application callbacks, and uninitialized > memory exposure. We achieve this without requiring modifications to existing > C/C++ ZooKeeper client applications or runtime configurations. Both > deliverables integrate TLS natively via `rustls`, removing OpenSSL from the > dependency and linkage graph. > > The legacy C client remains intact and untouched in the repository and > continues to serve as the default build target unless the Rust core is > selected during the build configuration process. > > *Highlights* > > *Safety and Formal Verification:* Across all shipped crates, "unsafe" Rust is > forbidden everywhere except a single audited Foreign Function Interface (FFI) > module. Compile time ownership and lock exclusion proof checking ensure > memory- and lock-safety at compile time, and these assurances are further > reinforced with formal verification using loom's concurrency permutation > model checker, Miri's strict provenance abstract machine interpretation, and > continuous property based fuzzing using cargo-fuzz. > > *Performance Superiority:* In a comprehensive head-to-head evaluation, the > Rust client outperforms the legacy C client with a 95% confidence interval in > 898 of 953 cases (94.2%), achieves statistical parity in 33 cases (3.5%), and > trails in 22 cases (2.3%). Client CPU consumption is lower than the legacy C > client in 284 of 294 measured cases (96.6%). The performance of the > multithreaded drop-in libzookeeper_mt is better than or at statistical parity > with the C client in 99.7% of the test cases. The single threaded drop-in > libzookeeper_st improves small operation latency across all cases by up to > 20%. Under high concurrency pipelining with a window of 1,000 concurrent > requests, throughput scales up to 32x with the drop-in MT library and up to > 54x with Rust's Tokio, compared to the legacy C MT library's measured > scalability at 1x. TLS session establishment is 27% faster than compared with > the C client. > > *Superior Reliability:* In concurrent close stress testing (36,000 > close-under-load iterations), the Rust drop-in libraries recorded zero lost, > duplicate, or stale completions, resolving a long standing teardown race > condition in the legacy C library, which often fails this test. > > *Overview* > > At the foundation of the Rust client stack sits a serialization tier, which > houses the generated wire data structures and encoding codecs derived > directly from the protocol schema. This foundation performs no network > transport operations and validates buffer boundaries on every read, using an > incremental frame decoder that allocates memory strictly as data arrives from > the network to protect against malformed frame headers and exhaustion attacks. > > Directly above serialization lies the protocol core ("IOCore"), designed as a > deterministic state machine operating entirely independent of sockets, > execution threads, or ambient clocks. By consuming raw byte streams, abstract > request representations, and discrete timesteps provided by the driver layer, > the state machine produces outbound wire streams, ordered application event > queues, and connection deadlines. All mutable session context, including > request pipelines, transaction counters, keepalive intervals, server > resolution tracking, watch tables, authentication state, and staged > connection teardown, is contained exclusively within this single-owner > protocol machine. > > The driver tier bridges the IOCore to the operating system network > facilities, multiplexed polling engines, wakeup primitives, connectivity > probes, and TLS sessions. A common execution pipeline compiles into four > distinct runtime environments selected by feature gates: a single threaded > runtime matching legacy single-threaded semantics, a multithreaded runtime > matching legacy multithreaded semantics, a single threaded runtime driven > directly by caller event loops, and a tokio-compatible runtime for async > applications. > > For Rust-native applications, a high level client layer provides typed > request handling, asynchronous pipelining, watch notification streams, and > builder interfaces, complemented by Curator-compatible coordination recipes > including distributed mutual exclusion, leader election, and hierarchical > tree caching. For C-based applications, a foreign function interface bridges > the Rust side to a thin C/C++ compatibility shim, exposing a C ABI identical > to today's C client, while ensuring the compatibility bridge introduces no > overheads. > > The shared protocol state machine and I/O pipeline execute unified logic > across all deliverables without dynamic branching. Strict boundaries are > maintained across the compilation units, forbidding unsafe operations > throughout the codebase with the exception of the isolated foreign function > bridge. > > The build harness includes comprehensive automated verification suites > including a deterministic fault-injection framework, scripted mock servers, > differential cross-checks against legacy implementations, binary layout > probes, and continuous fuzz testing to provide confidence that the Rust > client implementation is both as memory- and concurrency-safe as promised and > fully compatible with legacy client behavior. > > *Architecture* > > The foundational invariant is that application-defined code never executes > while a Rust stack frame remains active on the calling thread. This rule > insulates user logic, including asynchronous completion callbacks, watch > notifications, authentication completions, custom logging hooks, and security > mechanism exchanges, from the IOCore. Isolation is realized through a > cooperative step-and-drain protocol. When the outer runtime invokes the core > engine, the core processes pending transport and protocol state, populates an > outbox queue with notifications, and returns control after unwinding all Rust > frames and releasing internal locks. The surrounding compatibility shim then > iterates through the outbox in native host frames, delivering results to > application callbacks or managing thread coordination. When protocol > progression depends on the observable outcome of an application notification, > such as a session closure or authentication state inspection initiated from > within a callback, the engine halts at a synchronization barrier and yields, > employing a corountine/continuation model. Once the outer layer completes > event dispatch, execution resumes seamlessly from the saved corountine state. > > The isolation strategy adapts naturally across the various runtime > environments. In the multithreaded compatibility library, the two-thread > architecture of the legacy client is preserved: one background thread drives > the network I/O loop while a separate dispatch thread handles completion and > watch events in strict wire order. Synchronous API calls submit asynchronous > requests internally and wait on native condition variables outside all Rust > frames, populating caller buffers only upon wake-up, while any attempt to > issue synchronous requests from the network thread is rejected immediately to > eliminate deadlock. Rather than blocking within the engine, thread join or > detach commands are yielded to the outer layer, allowing foreign threads, > completion callbacks, or network callbacks to terminate cleanly without > deadlocking. In single-threaded environments, no background threads are > created. The application drives I/O directly from its own event loop, > processing connection events, outbound flushes, and inbound frames in > lockstep and delivering completions inline at historical synchronization > points. In Rust-native environments, requests are validated and enqueued > synchronously at call time to ensure submission order matches wire > transmission order, routing responses to asynchronous futures, one-shot watch > channels, and persistent watch streams with integrated backpressure and > consumer lag tracking. > > The drop-in replacement libraries combine a compiled static archive of the > bridge crate with a thin compatibility shim and unmodified legacy > serialization units. Public data structures remain compiled by the platform C > compiler to guarantee exact binary layout and ensure dynamic linking > functions without divergence. Callback dispatch routines run entirely within > host stack frames, ensuring any non-local jump or exception unwinds only > native frames while safely returning unconsumed events to the core. Thread > creation, joins, detach operations, and synchronous waits similarly reside in > the outer layer so cancellation points never span active core frames. > Exported global symbols and Cyrus SASL glue remain isolated within dedicated > C units so that external C header details do not leak into the Rust driver. > > *Verification* > > A battery of verification tools guards against regressions, undefined > behavior, and concurrency hazards. Abstract machine interpretation under > strict pointer provenance rules systematically exercises unit and integration > paths to confirm the total absence of dangling references, uninitialized > memory accesses, and aliasing violations across the driver core and foreign > interface bridge. Concurrency permutation model checking exhaustively > explores thread scheduling interleavings across queue implementations, > shutdown sequences, and handle lifecycle operations to ensure synchronization > freedom. Continuous coverage-guided fuzzing targets serialization formats, > incremental frame reconstruction, transaction batches, stream decoders, > authentication handshakes, and cryptographic processing pipelines under > address and leak sanitization. Structural layout checkers compile native > probe programs across all supported build configurations to assert that every > bridge type and public structure matches expected target alignment and > padding. Specialized memory scans verify that sensitive cryptographic key > material and session credentials are wiped from deallocated heap memory. > These verification passes complement extensive deterministic simulation > suites, property-based tests, and integration test suites spanning thousands > of test scenarios. > > This verification strategy eliminates whole classes of memory and concurrency > vulnerabilities present in the C client implementation. Invalid or > uninitialized client handle references are intercepted at entry boundaries > and converted into clean argument rejections rather than triggering > null-pointer faults. Buffer operations on data payloads, string collections, > and access control lists validate dimensions and capacities before memory > copies occur, eliminating heap corruption and buffer overreads. Framing logic > guarantees that malformed or truncated network frames result in graceful > connection termination rather than parsing beyond allocated packet bounds or > exposing uninitialized stack data to user callbacks. Transaction batch > failures consistently initialize all caller result slots with defined error > codes rather than leaving uninitialized memory in place. Shared process-wide > counters and credential buffers are fully encapsulated within isolated > per-session states, eliminating unsynchronized cross-thread mutations. > Internal locks are never held while invoking application notifications, > preventing deadlock when user callbacks make reentrant client calls. Host > failover state machines advance monotonically, eliminating cursor progression > defects that previously bypassed ensemble nodes during connection retries. > Finally, teardown sequences coordinate thread termination and event delivery > so that queued completions are guaranteed to reach waiting callers rather > than being dropped during shutdown. > > Applications that employ external authentication mechanisms across multiple > threads must continue to configure thread synchronization within the > third-party library, as those external caches reside outside the client > boundary. > > *Compatibility* > > An existing application built against the original C client can link directly > against or dynamically load the replacement libraries with no modifications > to application code, build configurations, or operational deployments. > Complete fidelity is maintained across all observable dimensions of the > programming interface: installed public headers are byte-for-byte identical, > platform library names and major shared object versions match historical > targets, and exported symbol sets maintain exact parity in symbol names, > function signatures, data object sizes, and mutability without introducing > extraneous symbols or omitting existing ones. Protocol framing, request > sequencing, timeout limits, hash-table watcher traversal order, error return > codes, operating system error hints, callback delivery sequences, and > formatted diagnostic messages similarly reflect legacy behaviors. Runtime > execution characteristics are fully preserved, running asynchronous > completions and watch events on a dedicated completion thread in > multithreaded builds while dispatching them inline during event loop > processing in single-threaded builds. > > Symbol visibility boundaries are strictly enforced during linking, ensuring > internal helpers and runtime components remain fully unexported while > matching the historical public inventory across every build configuration. In > every build variant, a fixed subset of the public surface represents exported > data objects, split between read-only constants and mutable configuration > variables. The legacy build infrastructure requires only minimal adjustments > to delegate to the new core. > > > > -- > This message was sent by Atlassian Jira > (v10.3.26#10030026)
