+1 (binding) -Lari
On Thu, 1 Oct 2026 at 03:20, PengHui Li <[email protected]> wrote: > > +1 (binding) > > Regards, > Penghui > > On Mon, Sep 14, 2026 at 5:07 PM Matteo Merli <[email protected]> wrote: > > > https://github.com/apache/pulsar/pull/26447 > > Rendered version: > > https://github.com/merlimat/pulsar/blob/mmerli/pip-494/pip/pip-494.md > > > > ------ > > > > # PIP-494: Scalable Topics Client Specification — authoritative > > reference and change process > > > > *Sub-PIP of [PIP-460: Scalable Topics](pip-460.md)* > > > > > **Draft.** This PIP establishes a **process**; it deliberately does not > > restate the specification's > > > content, which lives in the repository at `spec/scalable-topics/`. The > > current work-in-progress > > > specification — > > > < > > https://github.com/merlimat/pulsar/tree/mmerli/scalable-topics-spec/spec/scalable-topics> > > — will be > > > adopted as **version 1.0** of the specification. It reflects the current > > client API and > > > semantics as designed in the scalable-topics PIPs > > ([PIP-460](pip-460.md), [PIP-468](pip-468.md), > > > [PIP-483](pip-483.md), [PIP-486](pip-486.md)) and as implemented on the > > `master` branch. > > > > ## Motivation > > > > Scalable topics ([PIP-460](pip-460.md) and its sub-PIPs — the > > controller [PIP-468](pip-468.md), auto > > split/merge [PIP-483](pip-483.md), key-shared consumption > > [PIP-486](pip-486.md)) introduce a client model > > that is materially different from classic Pulsar topics: no > > application-visible partitions, a dynamic > > per-cluster segment layout the client must track, client-side key > > routing, three distinct consumer > > modes with different ordering/acknowledgment contracts, a checkpoint > > position type, a controller-driven > > consumer-assignment protocol, and a family of new binary-protocol commands. > > > > Today the only complete description of that client behavior is the > > **Java V5 client's source code**, with > > the design rationale scattered across the PIPs. That is not a workable > > basis for the ecosystem: > > > > - **Other-language SDKs** (Go, Python, C++, Rust, Node, .NET) need a > > precise, language-neutral definition > > of the API contract, the required client-side mechanisms, and the > > exact wire interactions — not a Java > > implementation to reverse-engineer. > > - **Interoperability** depends on every client routing keys > > identically, applying layouts and assignments > > with the same staleness rules, and honoring the same ordering and > > acknowledgment semantics. Without a > > normative reference, divergent clients are inevitable and hard to > > diagnose. > > - **PIPs are design proposals, not contracts.** They explain *why* and > > record decisions, but they are > > point-in-time documents that are never updated after acceptance. The > > living behavior needs a living, > > normative document. > > - **Stability and deprecation** need defined rules so SDK authors know > > what they can rely on across > > Pulsar releases. > > > > This PIP establishes a **Scalable Topics Client Specification** as the > > single authoritative description > > of the client API, its semantics, and its protocol-level interactions > > — the base document for anyone > > implementing scalable-topics support in a Pulsar client SDK — and > > defines the **process** by which that > > specification is maintained and changed. > > > > ## Background knowledge > > > > **PIPs vs. a specification.** A PIP proposes and ratifies a change; > > once merged it is a historical > > record of a decision. A specification is a *current* normative > > statement of behavior that implementations > > conform to, and it evolves as behavior evolves. Mature > > cross-implementation ecosystems keep both: the > > OpenTelemetry project, for example, maintains a versioned > > specification with explicit stability tiers and > > a formal change process, separate from the proposals that motivated > > each change. This PIP adopts that > > model for scalable topics. > > > > **Reference implementation.** The Java V5 client > > (`pulsar-client-api-v5`, `pulsar-client-v5`) is the > > reference implementation of scalable-topics client behavior. The > > specification is derived from it and > > verified against it, but the specification — not the Java code — is > > the source of truth for conformance. > > > > ## Goals > > > > ### In Scope > > > > - Establish `spec/scalable-topics/` in the `apache/pulsar` repository > > as the **authoritative, > > language-neutral specification** for scalable-topics clients: the > > API contract, its semantics, the > > required client-side mechanisms, and the wire-protocol interactions. > > - Define the specification as the **required base for any new client > > SDK** adding scalable-topics > > support, and the reference against which SDK **conformance** is judged. > > - Define the **change process**: how normative changes to the > > specification are proposed, reviewed, and > > landed, and how the specification is kept in sync with the reference > > implementation. > > - Define the specification's **versioning, stability tiers, and > > deprecation/removal rules**, aligned with > > the Apache Pulsar release (LTS) cadence. > > > > ### Out of Scope > > > > - The specification's content itself. It is a separate deliverable > > (already drafted) and is not > > restated here; this PIP governs it, it does not define client behavior. > > - Client behavior for classic (non-scalable) topics — the > > specification covers scalable topics and their > > migration bridge from classic topics only. > > - The mechanics of publishing the specification to the user-facing > > documentation site > > (`apache/pulsar-site`); that is a follow-up task once the process is in > > place. > > > > ## High Level Design > > > > ### What is established > > > > 1. **Location and status.** The specification lives at > > `spec/scalable-topics/` in `apache/pulsar`, as a > > set of Markdown documents with a top-level index (`README.md`). It > > is a first-class artifact of the > > project, maintained alongside the code it describes. > > > > 2. **Authority.** For scalable-topics client behavior, the > > specification is **normative**. Where an > > implementation (including the Java reference client) and the > > specification disagree, the > > specification is the source of truth and the implementation is the > > bug — unless a PIP changes the > > specification. > > > > 3. **Audience and use.** The specification is the base document for > > implementing a scalable-topics > > client in any language. It separates the transport-agnostic **API > > contract** (what an application > > observes: operations, inputs, guarantees, errors) from the > > **implementation requirements** (what a > > client must do internally: routing, layout tracking, fan-out, > > retries) and from the **wire protocol** > > (the exact command exchanges, with sequence diagrams), so an > > implementer can build against the > > contract and bind it to the Pulsar protocol. > > > > 4. **Conformance.** The specification defines what a conformant client > > MUST implement (the core > > producer, the three consumer modes, transactions) and which > > capabilities are optional. An SDK claims > > scalable-topics support by conforming to the specification's > > conformance document. > > > > ### How it is governed > > > > The specification is **not itself a PIP**. It is a living document > > maintained under the change process > > below, which ties every normative change to a PIP and keeps the > > document in lockstep with the reference > > implementation. > > > > ## Detailed Design > > > > ### 1. Change process > > > > - **Every normative change to the specification MUST be made through a > > PIP.** A "normative change" is > > any change to a requirement (MUST/SHOULD/MAY), a guarantee, an > > operation's contract, or a wire > > interaction. The PIP carries the design and rationale; reviewers > > evaluate the proposed behavior and the > > specification edits **together**. > > - **Specification edits land with the PIP.** When a PIP that changes > > scalable-topics client behavior is > > accepted, the corresponding edits to `spec/scalable-topics/` are > > merged together with it — normally in > > the **same pull request** as the implementation — so the contract > > and its decision record never > > drift. A PIP that changes client-observable behavior is not complete > > until its specification edits are > > merged. > > - **Non-normative edits** (typos, wording clarifications that change > > no requirement, added examples or > > diagrams, cross-reference fixes) do not require a PIP and follow the > > normal pull-request review. > > - **PIPs remain the rationale and decision record**; the specification > > is the normative contract kept > > continuously in sync with the ratified PIPs. A reader who wants to > > know *what* a client must do reads > > the specification; a reader who wants to know *why* follows its > > references to the PIPs. > > > > ### 2. Relationship to the reference implementation > > > > - The Java V5 client is the **reference implementation**. The > > specification's Stable content describes > > behavior that is **implemented in a released broker and the > > reference client, or ratified via an > > accepted PIP and committed to ship**; an SDK built to it > > interoperates with current and announced > > brokers. > > - A change to the reference implementation that alters > > client-observable behavior MUST be accompanied > > by the matching specification change (via §1). Conversely, the > > specification MUST NOT describe > > behavior the reference implementation does not (or will not, per an > > accepted PIP) exhibit. > > - The specification includes a **Java reference mapping** appendix > > that maps each language-neutral > > operation and type to the Java V5 surface, so the reference > > implementation and the specification can > > be checked against each other and other SDKs can align their naming. > > > > ### 3. Versioning, aligned with Pulsar LTS releases > > > > - The specification carries its own version of the form `MAJOR.MINOR`, > > declared in its index. > > - **MAJOR** increments on a backward-incompatible change to a *Stable* > > requirement and is **tied to the > > Apache Pulsar LTS cadence**: each new MAJOR corresponds to a new > > Pulsar LTS release. Backward-incompatible > > changes and removals of deprecated features therefore land only at > > an LTS boundary, never in an interim > > release. > > - **MINOR** increments on backward-compatible additions or > > clarifications and may occur in any release. > > - The work-in-progress draft is currently at `0.x` (**Draft**). > > Acceptance of this PIP adopts it as > > **`1.0`**, from which point the stability guarantees below take full > > effect. Any later MAJOR > > transition is a normative change made through the change process (§1). > > > > ### 4. Stability tiers and deprecation > > > > Every feature in the specification carries a stability tier expressing > > its change contract: > > > > - **Stable** — will not change incompatibly without a MAJOR bump; > > conformant clients MUST implement all > > Stable, non-optional requirements. > > - **Experimental** — may change in any way, including removal, in a > > MINOR version; clients MAY implement > > it, and omitting it never makes a client non-conformant. Used for > > designs that are documentable but not > > yet settled. > > - **Deprecated** — Stable but scheduled for removal; retained for > > compatibility, not to be newly adopted. > > > > **Deprecation and removal mechanism.** A feature enters Deprecated > > only through the change process (a > > PIP) that records the reason, the replacement (or that none exists), > > and the earliest version at which > > removal is permitted. While Deprecated, brokers MUST keep honoring it, > > clients SHOULD migrate, and > > implementations SHOULD emit a non-fatal deprecation signal. A > > Deprecated Stable feature MUST NOT be > > removed before the next MAJOR version **and** MUST remain available at > > least until the next Pulsar LTS > > release, guaranteeing an LTS-to-LTS upgrade path in which the feature > > and its replacement coexist. > > > > ### 5. Scope of the specification > > > > - **In scope:** everything a client SDK must implement for scalable > > topics — topic identity, the data > > model (message, message identifier, checkpoint, schema), the > > producer and the three consumer modes, > > transactions, error semantics, the client-side mechanisms, the > > complete wire protocol, conformance > > criteria, and compatibility with classic topics (the migration bridge). > > - **Out of scope:** broker-internal behavior (controller, split/merge > > execution, storage) except where > > observable by a client; and designs whose specification is not yet > > settled (currently geo-replication > > of scalable topics), which enter the specification — initially as > > Experimental — only once ratified. > > > > ### 6. Obligations on new client SDKs > > > > An SDK adding scalable-topics support: > > > > - MUST implement the specification's core (Stable, non-optional) > > requirements and MAY implement optional > > capabilities, each fully per its section, gating on broker support > > where the specification requires. > > - MUST use the specification — not the Java client's source — as its > > reference, and SHOULD map its > > idiomatic API to the specification's language-neutral operations (as > > the Java mapping appendix does for > > Java), so cross-SDK behavior stays comparable. > > - SHOULD contribute clarifications back through the change process > > when the specification is found > > ambiguous or incomplete, rather than resolving the ambiguity privately. > > > > ## Public-facing changes > > > > - **New repository artifact:** `spec/scalable-topics/` — the > > specification document set. The > > work-in-progress draft linked above is adopted as **version 1.0**: > > it reflects the current client > > API and semantics as designed in the scalable-topics PIPs and as > > implemented on `master`; the > > stability guarantees of §3–§4 apply from that version. > > - **Process:** the change process in §1 applies to every subsequent > > PIP that touches scalable-topics > > client behavior, and to pull requests changing the reference > > implementation's client-observable > > behavior. > > - No code, wire-protocol, configuration, API, or metrics changes are > > introduced by this PIP. > > > > ## Backward & Forward Compatibility > > > > Not applicable to this PIP itself: it introduces a document and a > > process, not a behavior change. The > > compatibility guarantees the specification *makes* (protocol > > extensibility by addition, cross-version > > restorability of serialized identifiers and checkpoints, the > > LTS-anchored deprecation window) are defined > > within the specification and are governed by §3–§4 of this PIP. > > > > ## Alternatives > > > > - **Keep the PIPs as the only description.** Rejected: PIPs are > > point-in-time proposals that are never > > updated after acceptance; they cannot serve as a living, normative > > contract, and they describe design > > rationale rather than exact client obligations. > > - **Make the specification itself a PIP.** Rejected: a PIP is ratified > > once and then frozen, whereas the > > specification must evolve continuously with the implementation. > > Governing the specification *through* > > PIPs (§1) captures the review rigor without freezing the document. > > - **Per-language documentation only (e.g. Javadoc as the reference).** > > Rejected: language-specific > > documentation cannot define language-neutral contracts, wire > > interactions, or conformance, and it > > invites divergence between SDKs. > > - **Publish the specification only on the documentation site, not in > > the repository.** Rejected for the > > source of truth: keeping it in `apache/pulsar` lets specification > > edits merge in the same pull request > > as the implementation change (§1). Mirroring to the docs site is a > > follow-up. > > -- > > Matteo Merli > > <[email protected]> > >
