merlimat opened a new pull request, #25875: URL: https://github.com/apache/pulsar/pull/25875
## Summary Third implementation PR for [PIP-475: Regular-to-Scalable Topic Migration](https://github.com/apache/pulsar/blob/master/pip/pip-475.md). Builds on the broker synthetic-layout lookup (#25822) and the V5 SDK support (#25850). Adds the admin command that converts an existing regular (partitioned or non-partitioned) topic into a scalable topic, plus the broker guard that keeps legacy v4 clients off a migrated topic. The v4-guard work originally scoped as a separate PR is folded in here. ### Migration command (admin REST + client + CLI) - `POST /admin/v2/scalable-topics/{t}/{n}/{topic}/migrate?force=` (requires `produce` permission). - `ScalableTopics.migrateToScalable(topic, force)` / `...Async`; `pulsar-admin scalable-topics migrate <topic> [--force]`. - Orchestration: reject if already scalable (409) or topic missing (404); unless `--force`, reject if any legacy v4 connection is attached (409); build the migrated layout; create the new child segment topics; CAS-write the scalable metadata (commit point — connected V5 lookup sessions transition from the synthetic layout to the real DAG via the metadata watch); terminate the old topic(s) so no further v4 writes land — they become the drainable sealed parent segments. ### Migrated layout (`ScalableTopicController.createMigratedMetadata`) - N sealed legacy parent segments (the old partitions, each spanning the full hash range since v4 mod-N scattered keys everywhere) + N active range-based children with full parent fan-in; `nextSegmentId = 2N`. ### Legacy-vs-V5 connection detection - New `ScalableTopicConstants.V5_MANAGED_METADATA_KEY` marker; the V5 SDK sets it on its per-segment producers/consumers **only when attaching to a legacy segment** (a wrapped `persistent://` topic). The migration pre-check counts producers/consumers whose metadata lacks the marker as legacy v4 connections, so a topic served solely by upgraded V5 clients passes the check. ### v4 write guard - Post-migration termination blocks attached/new v4 producers at send. `BrokerService.isAllowAutoTopicCreationAsync` also refuses to auto-create a `persistent://` topic shadowed by a scalable topic (reuses `TopicTerminatedError`, metadata-cache-backed, gated by `scalableTopicsEnabled`) — enforcing "once scalable, always scalable" even after the old ledgers are GC'd. ## Test plan - [x] `ScalableTopicControllerTest`: `createMigratedMetadata` for partitioned + non-partitioned sources. - [x] `ScalableTopicMigrationTest` (7 cases): migrate partitioned + non-partitioned, already-scalable 409, missing 404, legacy-connection 409 without force, success with `--force`, and the auto-create shadow guard. - [x] Full `pulsar-client-v5` unit suite + `SegmentRouterTest` regression. - [ ] End-to-end migration test (full operator timeline: v4 produce → upgrade to V5 → migrate → per-key ordering / drain / termination / v4 lockout) — follow-up PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
