https://github.com/apache/pulsar/pull/26447 Rendered version: https://github.com/merlimat/pulsar/blob/mmerli/pip-494/pip/pip-494.md
------ # PIP-494: Scalable Topics Client Specification — authoritative reference and change process *Sub-PIP of [PIP-460: Scalable Topics](pip-460.md)* > **Draft.** This PIP establishes a **process**; it deliberately does not > restate the specification's > content, which lives in the repository at `spec/scalable-topics/`. The > current work-in-progress > specification — > <https://github.com/merlimat/pulsar/tree/mmerli/scalable-topics-spec/spec/scalable-topics> > — will be > adopted as **version 1.0** of the specification. It reflects the current > client API and > semantics as designed in the scalable-topics PIPs ([PIP-460](pip-460.md), > [PIP-468](pip-468.md), > [PIP-483](pip-483.md), [PIP-486](pip-486.md)) and as implemented on the > `master` branch. ## Motivation Scalable topics ([PIP-460](pip-460.md) and its sub-PIPs — the controller [PIP-468](pip-468.md), auto split/merge [PIP-483](pip-483.md), key-shared consumption [PIP-486](pip-486.md)) introduce a client model that is materially different from classic Pulsar topics: no application-visible partitions, a dynamic per-cluster segment layout the client must track, client-side key routing, three distinct consumer modes with different ordering/acknowledgment contracts, a checkpoint position type, a controller-driven consumer-assignment protocol, and a family of new binary-protocol commands. Today the only complete description of that client behavior is the **Java V5 client's source code**, with the design rationale scattered across the PIPs. That is not a workable basis for the ecosystem: - **Other-language SDKs** (Go, Python, C++, Rust, Node, .NET) need a precise, language-neutral definition of the API contract, the required client-side mechanisms, and the exact wire interactions — not a Java implementation to reverse-engineer. - **Interoperability** depends on every client routing keys identically, applying layouts and assignments with the same staleness rules, and honoring the same ordering and acknowledgment semantics. Without a normative reference, divergent clients are inevitable and hard to diagnose. - **PIPs are design proposals, not contracts.** They explain *why* and record decisions, but they are point-in-time documents that are never updated after acceptance. The living behavior needs a living, normative document. - **Stability and deprecation** need defined rules so SDK authors know what they can rely on across Pulsar releases. This PIP establishes a **Scalable Topics Client Specification** as the single authoritative description of the client API, its semantics, and its protocol-level interactions — the base document for anyone implementing scalable-topics support in a Pulsar client SDK — and defines the **process** by which that specification is maintained and changed. ## Background knowledge **PIPs vs. a specification.** A PIP proposes and ratifies a change; once merged it is a historical record of a decision. A specification is a *current* normative statement of behavior that implementations conform to, and it evolves as behavior evolves. Mature cross-implementation ecosystems keep both: the OpenTelemetry project, for example, maintains a versioned specification with explicit stability tiers and a formal change process, separate from the proposals that motivated each change. This PIP adopts that model for scalable topics. **Reference implementation.** The Java V5 client (`pulsar-client-api-v5`, `pulsar-client-v5`) is the reference implementation of scalable-topics client behavior. The specification is derived from it and verified against it, but the specification — not the Java code — is the source of truth for conformance. ## Goals ### In Scope - Establish `spec/scalable-topics/` in the `apache/pulsar` repository as the **authoritative, language-neutral specification** for scalable-topics clients: the API contract, its semantics, the required client-side mechanisms, and the wire-protocol interactions. - Define the specification as the **required base for any new client SDK** adding scalable-topics support, and the reference against which SDK **conformance** is judged. - Define the **change process**: how normative changes to the specification are proposed, reviewed, and landed, and how the specification is kept in sync with the reference implementation. - Define the specification's **versioning, stability tiers, and deprecation/removal rules**, aligned with the Apache Pulsar release (LTS) cadence. ### Out of Scope - The specification's content itself. It is a separate deliverable (already drafted) and is not restated here; this PIP governs it, it does not define client behavior. - Client behavior for classic (non-scalable) topics — the specification covers scalable topics and their migration bridge from classic topics only. - The mechanics of publishing the specification to the user-facing documentation site (`apache/pulsar-site`); that is a follow-up task once the process is in place. ## High Level Design ### What is established 1. **Location and status.** The specification lives at `spec/scalable-topics/` in `apache/pulsar`, as a set of Markdown documents with a top-level index (`README.md`). It is a first-class artifact of the project, maintained alongside the code it describes. 2. **Authority.** For scalable-topics client behavior, the specification is **normative**. Where an implementation (including the Java reference client) and the specification disagree, the specification is the source of truth and the implementation is the bug — unless a PIP changes the specification. 3. **Audience and use.** The specification is the base document for implementing a scalable-topics client in any language. It separates the transport-agnostic **API contract** (what an application observes: operations, inputs, guarantees, errors) from the **implementation requirements** (what a client must do internally: routing, layout tracking, fan-out, retries) and from the **wire protocol** (the exact command exchanges, with sequence diagrams), so an implementer can build against the contract and bind it to the Pulsar protocol. 4. **Conformance.** The specification defines what a conformant client MUST implement (the core producer, the three consumer modes, transactions) and which capabilities are optional. An SDK claims scalable-topics support by conforming to the specification's conformance document. ### How it is governed The specification is **not itself a PIP**. It is a living document maintained under the change process below, which ties every normative change to a PIP and keeps the document in lockstep with the reference implementation. ## Detailed Design ### 1. Change process - **Every normative change to the specification MUST be made through a PIP.** A "normative change" is any change to a requirement (MUST/SHOULD/MAY), a guarantee, an operation's contract, or a wire interaction. The PIP carries the design and rationale; reviewers evaluate the proposed behavior and the specification edits **together**. - **Specification edits land with the PIP.** When a PIP that changes scalable-topics client behavior is accepted, the corresponding edits to `spec/scalable-topics/` are merged together with it — normally in the **same pull request** as the implementation — so the contract and its decision record never drift. A PIP that changes client-observable behavior is not complete until its specification edits are merged. - **Non-normative edits** (typos, wording clarifications that change no requirement, added examples or diagrams, cross-reference fixes) do not require a PIP and follow the normal pull-request review. - **PIPs remain the rationale and decision record**; the specification is the normative contract kept continuously in sync with the ratified PIPs. A reader who wants to know *what* a client must do reads the specification; a reader who wants to know *why* follows its references to the PIPs. ### 2. Relationship to the reference implementation - The Java V5 client is the **reference implementation**. The specification's Stable content describes behavior that is **implemented in a released broker and the reference client, or ratified via an accepted PIP and committed to ship**; an SDK built to it interoperates with current and announced brokers. - A change to the reference implementation that alters client-observable behavior MUST be accompanied by the matching specification change (via §1). Conversely, the specification MUST NOT describe behavior the reference implementation does not (or will not, per an accepted PIP) exhibit. - The specification includes a **Java reference mapping** appendix that maps each language-neutral operation and type to the Java V5 surface, so the reference implementation and the specification can be checked against each other and other SDKs can align their naming. ### 3. Versioning, aligned with Pulsar LTS releases - The specification carries its own version of the form `MAJOR.MINOR`, declared in its index. - **MAJOR** increments on a backward-incompatible change to a *Stable* requirement and is **tied to the Apache Pulsar LTS cadence**: each new MAJOR corresponds to a new Pulsar LTS release. Backward-incompatible changes and removals of deprecated features therefore land only at an LTS boundary, never in an interim release. - **MINOR** increments on backward-compatible additions or clarifications and may occur in any release. - The work-in-progress draft is currently at `0.x` (**Draft**). Acceptance of this PIP adopts it as **`1.0`**, from which point the stability guarantees below take full effect. Any later MAJOR transition is a normative change made through the change process (§1). ### 4. Stability tiers and deprecation Every feature in the specification carries a stability tier expressing its change contract: - **Stable** — will not change incompatibly without a MAJOR bump; conformant clients MUST implement all Stable, non-optional requirements. - **Experimental** — may change in any way, including removal, in a MINOR version; clients MAY implement it, and omitting it never makes a client non-conformant. Used for designs that are documentable but not yet settled. - **Deprecated** — Stable but scheduled for removal; retained for compatibility, not to be newly adopted. **Deprecation and removal mechanism.** A feature enters Deprecated only through the change process (a PIP) that records the reason, the replacement (or that none exists), and the earliest version at which removal is permitted. While Deprecated, brokers MUST keep honoring it, clients SHOULD migrate, and implementations SHOULD emit a non-fatal deprecation signal. A Deprecated Stable feature MUST NOT be removed before the next MAJOR version **and** MUST remain available at least until the next Pulsar LTS release, guaranteeing an LTS-to-LTS upgrade path in which the feature and its replacement coexist. ### 5. Scope of the specification - **In scope:** everything a client SDK must implement for scalable topics — topic identity, the data model (message, message identifier, checkpoint, schema), the producer and the three consumer modes, transactions, error semantics, the client-side mechanisms, the complete wire protocol, conformance criteria, and compatibility with classic topics (the migration bridge). - **Out of scope:** broker-internal behavior (controller, split/merge execution, storage) except where observable by a client; and designs whose specification is not yet settled (currently geo-replication of scalable topics), which enter the specification — initially as Experimental — only once ratified. ### 6. Obligations on new client SDKs An SDK adding scalable-topics support: - MUST implement the specification's core (Stable, non-optional) requirements and MAY implement optional capabilities, each fully per its section, gating on broker support where the specification requires. - MUST use the specification — not the Java client's source — as its reference, and SHOULD map its idiomatic API to the specification's language-neutral operations (as the Java mapping appendix does for Java), so cross-SDK behavior stays comparable. - SHOULD contribute clarifications back through the change process when the specification is found ambiguous or incomplete, rather than resolving the ambiguity privately. ## Public-facing changes - **New repository artifact:** `spec/scalable-topics/` — the specification document set. The work-in-progress draft linked above is adopted as **version 1.0**: it reflects the current client API and semantics as designed in the scalable-topics PIPs and as implemented on `master`; the stability guarantees of §3–§4 apply from that version. - **Process:** the change process in §1 applies to every subsequent PIP that touches scalable-topics client behavior, and to pull requests changing the reference implementation's client-observable behavior. - No code, wire-protocol, configuration, API, or metrics changes are introduced by this PIP. ## Backward & Forward Compatibility Not applicable to this PIP itself: it introduces a document and a process, not a behavior change. The compatibility guarantees the specification *makes* (protocol extensibility by addition, cross-version restorability of serialized identifiers and checkpoints, the LTS-anchored deprecation window) are defined within the specification and are governed by §3–§4 of this PIP. ## Alternatives - **Keep the PIPs as the only description.** Rejected: PIPs are point-in-time proposals that are never updated after acceptance; they cannot serve as a living, normative contract, and they describe design rationale rather than exact client obligations. - **Make the specification itself a PIP.** Rejected: a PIP is ratified once and then frozen, whereas the specification must evolve continuously with the implementation. Governing the specification *through* PIPs (§1) captures the review rigor without freezing the document. - **Per-language documentation only (e.g. Javadoc as the reference).** Rejected: language-specific documentation cannot define language-neutral contracts, wire interactions, or conformance, and it invites divergence between SDKs. - **Publish the specification only on the documentation site, not in the repository.** Rejected for the source of truth: keeping it in `apache/pulsar` lets specification edits merge in the same pull request as the implementation change (§1). Mirroring to the docs site is a follow-up. ## Links <!-- Updated afterwards --> * Specification (work in progress): <https://github.com/merlimat/pulsar/tree/mmerli/scalable-topics-spec/spec/scalable-topics> — will live at `spec/scalable-topics/` in `apache/pulsar` (index: `README.md`) * Related: [PIP-460](pip-460.md), [PIP-468](pip-468.md), [PIP-483](pip-483.md), [PIP-486](pip-486.md) -- Matteo Merli <[email protected]>
