Hi Pulsar community, I'd like to propose PIP-496: Pulsar Functions and IO support for the V5 client and scalable topics. The Functions and IO runtime uses the v4 client throughout, so today a function, source or sink cannot read from or write to a scalable topic (topic://, PIP-460), which only the V5 client (PIP-466) supports. Functions and connectors also block the regular-to-scalable topic migration (PIP-475): the migration refuses while any v4 producer or consumer is attached, and every function or connector on a topic attaches one.
The proposal picks the client per component from its topics, with the same rule the pulsar-client and pulsar-perf CLIs already follow: topic:// topics use the V5 client with no configuration, and anything else keeps the v4 client. A new optional clientApi setting (--client-api V5) lets the V5 client drive persistent:// topics too, so the topics of a running component can be migrated. Subscription types map onto the V5 consumer models: - Shared uses the V5 queue consumer, with the same ack, negative-ack, timeout and dead letter behavior as today. - Failover, Key_Shared and EFFECTIVELY_ONCE use the V5 stream consumer, which gives per-key order across the component's instances. The proposal states where these semantics differ from v4. Every setting the V5 runtime cannot honor is rejected when the component is submitted. The user-facing API change is minimal: a single new default method, BaseContext.getPulsarClientV5(). Context.newOutputMessage() publishes to topic:// topics with the V5 client in every component. Existing components are unaffected, since the default keeps the v4 client unless a component uses topic:// topics. One open question I'd especially like feedback on: should a single component be allowed to mix topic:// topics with other topics? The proposal keeps the CLIs' stricter one-domain-per-component rule, which can be relaxed later without breaking anything. The feature targets Pulsar 5.0. A reference implementation with unit and end-to-end tests is available. Full proposal: https://github.com/lhotari/pulsar/blob/lh-pip-496/pip/pip-496.md PIP PR: https://github.com/apache/pulsar/pull/26698 Reference implementation: https://github.com/apache/pulsar/pull/26697 Looking forward to your feedback! Lari
