This proposal sounds very useful. Thank you for taking the initiative to address this gap in Pulsar Functions.
When thinking of a phased approach for implementation, I'd recommend prioritizing the "automatic transaction wrapping" approach since it wouldn't require changes to existing public APIs other than adding a configuration option for enabling this. After the "automatic transaction wrapping" has been implemented, the second phase could be to add the explicit transaction management support. The benefit of implementing in phases would be that we could simplify the implementation and get the first phase delivered more quickly than implementing both approaches at once. There might also be a chance to backport this feature to Pulsar 4.0.x LTS if we don't change public interfaces. -Lari On 2025/07/31 13:06:19 Ramzan MAGOMADOW wrote: > Hi Pulsar community, > > I'm planning to work on adding transactional support to Pulsar Functions and > would like to start a discussion about the approach before creating a formal > PIP. > > Background > > Currently, Pulsar Functions cannot publish to multiple topics > transactionally, which is a significant limitation for use cases requiring > atomic multi-topic publishing. As noted in the Pulsar Transactions > documentation<https://pulsar.apache.org/docs/4.0.x/txn-what/#use-case>, > "Support for Pulsar Functions and other connectors will be added in future > releases." > > Proposed Approach > > I've outlined a context-based approach in GitHub issue > #24588<https://github.com/apache/pulsar/issues/24588> that would add > transaction support through the existing Function Context interface. > > The key ideas include: > > 1. > Explicit Transaction Management > 2. > > context.beginTransaction(); > try { > context.newOutputMessage("topic-a", Schema.STRING).value("abc").send(); > context.newOutputMessage("topic-b", Schema.INT32).value(1).send(); > context.transactionallyAcknowledgeRecord(); > context.commitTransaction(); > } catch (Exception e) { > context.rollbackTransaction(); > throw e; > } > > This approach would require turning off automatic acknowledgment and letting > the context handle record acknowledgment transactionally. > 3. > Automatic Transaction Wrapping > A configuration option to automatically wrap function execution in a > transactional context, eliminating the need for explicit transaction > management in user code. > > Any feedback, suggestions, or experiences from the community would be greatly > appreciated before I proceed with the formal proposal. Has anyone else been > working on similar functionality or have thoughts on this approach? > > Best regards, > Ramzan Magomadow > This message and any attachment ("the Message") are confidential. If you have > received the Message in error, please notify the sender immediately and > delete the Message from your system, any use of the Message is forbidden. > Correspondence via e-mail is primarily for information purposes. RBI neither > makes nor accepts legally binding statements via e-mail unless explicitly > agreed otherwise. Information pursuant to ? 14 Austrian Companies Code: > Raiffeisen Bank International AG; Registered Office: Am Stadtpark 9, 1030 > Vienna, Austria; Company Register Number: FN 122119m at the Commercial Court > of Vienna (Handelsgericht Wien). > > Classification: GENERAL >
