+1 (binding) Regards, Penghui
On Mon, Sep 14, 2026 at 5:07 PM Matteo Merli <[email protected]> wrote: > https://github.com/apache/pulsar/pull/26447 > Rendered version: > https://github.com/merlimat/pulsar/blob/mmerli/pip-494/pip/pip-494.md > > ------ > > # PIP-494: Scalable Topics Client Specification — authoritative > reference and change process > > *Sub-PIP of [PIP-460: Scalable Topics](pip-460.md)* > > > **Draft.** This PIP establishes a **process**; it deliberately does not > restate the specification's > > content, which lives in the repository at `spec/scalable-topics/`. The > current work-in-progress > > specification — > > < > https://github.com/merlimat/pulsar/tree/mmerli/scalable-topics-spec/spec/scalable-topics> > — will be > > adopted as **version 1.0** of the specification. It reflects the current > client API and > > semantics as designed in the scalable-topics PIPs > ([PIP-460](pip-460.md), [PIP-468](pip-468.md), > > [PIP-483](pip-483.md), [PIP-486](pip-486.md)) and as implemented on the > `master` branch. > > ## Motivation > > Scalable topics ([PIP-460](pip-460.md) and its sub-PIPs — the > controller [PIP-468](pip-468.md), auto > split/merge [PIP-483](pip-483.md), key-shared consumption > [PIP-486](pip-486.md)) introduce a client model > that is materially different from classic Pulsar topics: no > application-visible partitions, a dynamic > per-cluster segment layout the client must track, client-side key > routing, three distinct consumer > modes with different ordering/acknowledgment contracts, a checkpoint > position type, a controller-driven > consumer-assignment protocol, and a family of new binary-protocol commands. > > Today the only complete description of that client behavior is the > **Java V5 client's source code**, with > the design rationale scattered across the PIPs. That is not a workable > basis for the ecosystem: > > - **Other-language SDKs** (Go, Python, C++, Rust, Node, .NET) need a > precise, language-neutral definition > of the API contract, the required client-side mechanisms, and the > exact wire interactions — not a Java > implementation to reverse-engineer. > - **Interoperability** depends on every client routing keys > identically, applying layouts and assignments > with the same staleness rules, and honoring the same ordering and > acknowledgment semantics. Without a > normative reference, divergent clients are inevitable and hard to > diagnose. > - **PIPs are design proposals, not contracts.** They explain *why* and > record decisions, but they are > point-in-time documents that are never updated after acceptance. The > living behavior needs a living, > normative document. > - **Stability and deprecation** need defined rules so SDK authors know > what they can rely on across > Pulsar releases. > > This PIP establishes a **Scalable Topics Client Specification** as the > single authoritative description > of the client API, its semantics, and its protocol-level interactions > — the base document for anyone > implementing scalable-topics support in a Pulsar client SDK — and > defines the **process** by which that > specification is maintained and changed. > > ## Background knowledge > > **PIPs vs. a specification.** A PIP proposes and ratifies a change; > once merged it is a historical > record of a decision. A specification is a *current* normative > statement of behavior that implementations > conform to, and it evolves as behavior evolves. Mature > cross-implementation ecosystems keep both: the > OpenTelemetry project, for example, maintains a versioned > specification with explicit stability tiers and > a formal change process, separate from the proposals that motivated > each change. This PIP adopts that > model for scalable topics. > > **Reference implementation.** The Java V5 client > (`pulsar-client-api-v5`, `pulsar-client-v5`) is the > reference implementation of scalable-topics client behavior. The > specification is derived from it and > verified against it, but the specification — not the Java code — is > the source of truth for conformance. > > ## Goals > > ### In Scope > > - Establish `spec/scalable-topics/` in the `apache/pulsar` repository > as the **authoritative, > language-neutral specification** for scalable-topics clients: the > API contract, its semantics, the > required client-side mechanisms, and the wire-protocol interactions. > - Define the specification as the **required base for any new client > SDK** adding scalable-topics > support, and the reference against which SDK **conformance** is judged. > - Define the **change process**: how normative changes to the > specification are proposed, reviewed, and > landed, and how the specification is kept in sync with the reference > implementation. > - Define the specification's **versioning, stability tiers, and > deprecation/removal rules**, aligned with > the Apache Pulsar release (LTS) cadence. > > ### Out of Scope > > - The specification's content itself. It is a separate deliverable > (already drafted) and is not > restated here; this PIP governs it, it does not define client behavior. > - Client behavior for classic (non-scalable) topics — the > specification covers scalable topics and their > migration bridge from classic topics only. > - The mechanics of publishing the specification to the user-facing > documentation site > (`apache/pulsar-site`); that is a follow-up task once the process is in > place. > > ## High Level Design > > ### What is established > > 1. **Location and status.** The specification lives at > `spec/scalable-topics/` in `apache/pulsar`, as a > set of Markdown documents with a top-level index (`README.md`). It > is a first-class artifact of the > project, maintained alongside the code it describes. > > 2. **Authority.** For scalable-topics client behavior, the > specification is **normative**. Where an > implementation (including the Java reference client) and the > specification disagree, the > specification is the source of truth and the implementation is the > bug — unless a PIP changes the > specification. > > 3. **Audience and use.** The specification is the base document for > implementing a scalable-topics > client in any language. It separates the transport-agnostic **API > contract** (what an application > observes: operations, inputs, guarantees, errors) from the > **implementation requirements** (what a > client must do internally: routing, layout tracking, fan-out, > retries) and from the **wire protocol** > (the exact command exchanges, with sequence diagrams), so an > implementer can build against the > contract and bind it to the Pulsar protocol. > > 4. **Conformance.** The specification defines what a conformant client > MUST implement (the core > producer, the three consumer modes, transactions) and which > capabilities are optional. An SDK claims > scalable-topics support by conforming to the specification's > conformance document. > > ### How it is governed > > The specification is **not itself a PIP**. It is a living document > maintained under the change process > below, which ties every normative change to a PIP and keeps the > document in lockstep with the reference > implementation. > > ## Detailed Design > > ### 1. Change process > > - **Every normative change to the specification MUST be made through a > PIP.** A "normative change" is > any change to a requirement (MUST/SHOULD/MAY), a guarantee, an > operation's contract, or a wire > interaction. The PIP carries the design and rationale; reviewers > evaluate the proposed behavior and the > specification edits **together**. > - **Specification edits land with the PIP.** When a PIP that changes > scalable-topics client behavior is > accepted, the corresponding edits to `spec/scalable-topics/` are > merged together with it — normally in > the **same pull request** as the implementation — so the contract > and its decision record never > drift. A PIP that changes client-observable behavior is not complete > until its specification edits are > merged. > - **Non-normative edits** (typos, wording clarifications that change > no requirement, added examples or > diagrams, cross-reference fixes) do not require a PIP and follow the > normal pull-request review. > - **PIPs remain the rationale and decision record**; the specification > is the normative contract kept > continuously in sync with the ratified PIPs. A reader who wants to > know *what* a client must do reads > the specification; a reader who wants to know *why* follows its > references to the PIPs. > > ### 2. Relationship to the reference implementation > > - The Java V5 client is the **reference implementation**. The > specification's Stable content describes > behavior that is **implemented in a released broker and the > reference client, or ratified via an > accepted PIP and committed to ship**; an SDK built to it > interoperates with current and announced > brokers. > - A change to the reference implementation that alters > client-observable behavior MUST be accompanied > by the matching specification change (via §1). Conversely, the > specification MUST NOT describe > behavior the reference implementation does not (or will not, per an > accepted PIP) exhibit. > - The specification includes a **Java reference mapping** appendix > that maps each language-neutral > operation and type to the Java V5 surface, so the reference > implementation and the specification can > be checked against each other and other SDKs can align their naming. > > ### 3. Versioning, aligned with Pulsar LTS releases > > - The specification carries its own version of the form `MAJOR.MINOR`, > declared in its index. > - **MAJOR** increments on a backward-incompatible change to a *Stable* > requirement and is **tied to the > Apache Pulsar LTS cadence**: each new MAJOR corresponds to a new > Pulsar LTS release. Backward-incompatible > changes and removals of deprecated features therefore land only at > an LTS boundary, never in an interim > release. > - **MINOR** increments on backward-compatible additions or > clarifications and may occur in any release. > - The work-in-progress draft is currently at `0.x` (**Draft**). > Acceptance of this PIP adopts it as > **`1.0`**, from which point the stability guarantees below take full > effect. Any later MAJOR > transition is a normative change made through the change process (§1). > > ### 4. Stability tiers and deprecation > > Every feature in the specification carries a stability tier expressing > its change contract: > > - **Stable** — will not change incompatibly without a MAJOR bump; > conformant clients MUST implement all > Stable, non-optional requirements. > - **Experimental** — may change in any way, including removal, in a > MINOR version; clients MAY implement > it, and omitting it never makes a client non-conformant. Used for > designs that are documentable but not > yet settled. > - **Deprecated** — Stable but scheduled for removal; retained for > compatibility, not to be newly adopted. > > **Deprecation and removal mechanism.** A feature enters Deprecated > only through the change process (a > PIP) that records the reason, the replacement (or that none exists), > and the earliest version at which > removal is permitted. While Deprecated, brokers MUST keep honoring it, > clients SHOULD migrate, and > implementations SHOULD emit a non-fatal deprecation signal. A > Deprecated Stable feature MUST NOT be > removed before the next MAJOR version **and** MUST remain available at > least until the next Pulsar LTS > release, guaranteeing an LTS-to-LTS upgrade path in which the feature > and its replacement coexist. > > ### 5. Scope of the specification > > - **In scope:** everything a client SDK must implement for scalable > topics — topic identity, the data > model (message, message identifier, checkpoint, schema), the > producer and the three consumer modes, > transactions, error semantics, the client-side mechanisms, the > complete wire protocol, conformance > criteria, and compatibility with classic topics (the migration bridge). > - **Out of scope:** broker-internal behavior (controller, split/merge > execution, storage) except where > observable by a client; and designs whose specification is not yet > settled (currently geo-replication > of scalable topics), which enter the specification — initially as > Experimental — only once ratified. > > ### 6. Obligations on new client SDKs > > An SDK adding scalable-topics support: > > - MUST implement the specification's core (Stable, non-optional) > requirements and MAY implement optional > capabilities, each fully per its section, gating on broker support > where the specification requires. > - MUST use the specification — not the Java client's source — as its > reference, and SHOULD map its > idiomatic API to the specification's language-neutral operations (as > the Java mapping appendix does for > Java), so cross-SDK behavior stays comparable. > - SHOULD contribute clarifications back through the change process > when the specification is found > ambiguous or incomplete, rather than resolving the ambiguity privately. > > ## Public-facing changes > > - **New repository artifact:** `spec/scalable-topics/` — the > specification document set. The > work-in-progress draft linked above is adopted as **version 1.0**: > it reflects the current client > API and semantics as designed in the scalable-topics PIPs and as > implemented on `master`; the > stability guarantees of §3–§4 apply from that version. > - **Process:** the change process in §1 applies to every subsequent > PIP that touches scalable-topics > client behavior, and to pull requests changing the reference > implementation's client-observable > behavior. > - No code, wire-protocol, configuration, API, or metrics changes are > introduced by this PIP. > > ## Backward & Forward Compatibility > > Not applicable to this PIP itself: it introduces a document and a > process, not a behavior change. The > compatibility guarantees the specification *makes* (protocol > extensibility by addition, cross-version > restorability of serialized identifiers and checkpoints, the > LTS-anchored deprecation window) are defined > within the specification and are governed by §3–§4 of this PIP. > > ## Alternatives > > - **Keep the PIPs as the only description.** Rejected: PIPs are > point-in-time proposals that are never > updated after acceptance; they cannot serve as a living, normative > contract, and they describe design > rationale rather than exact client obligations. > - **Make the specification itself a PIP.** Rejected: a PIP is ratified > once and then frozen, whereas the > specification must evolve continuously with the implementation. > Governing the specification *through* > PIPs (§1) captures the review rigor without freezing the document. > - **Per-language documentation only (e.g. Javadoc as the reference).** > Rejected: language-specific > documentation cannot define language-neutral contracts, wire > interactions, or conformance, and it > invites divergence between SDKs. > - **Publish the specification only on the documentation site, not in > the repository.** Rejected for the > source of truth: keeping it in `apache/pulsar` lets specification > edits merge in the same pull request > as the implementation change (§1). Mirroring to the docs site is a > follow-up. > -- > Matteo Merli > <[email protected]> >
