lhotari commented on issue #24588:
URL: https://github.com/apache/pulsar/issues/24588#issuecomment-3139163959
> Thank you for the feedback!
>
> I think adding transaction support through a context-based approach would
be ideal. Here's what I have in mind:
>
> public class TransactionalTupleFunction implements Function<String, Void> {
> @Override
> public Void process(String input, Context context) throws Exception {
> context.beginTransaction();
> try {
> context.newOutputMessage(input + "_processed",
"topic-a").send();
> context.newOutputMessage(input.length(), "topic-b").send();
> context.commitTransaction();
> } catch (Exception e) {
> context.rollbackTransaction();
> throw e;
> }
> return null;
> }
> }
> Alternatively we could add a configuration option to automatically run
functions within a transactional context. The Java Instance could wrap the
entire function execution in a transaction, eliminating the need for explicit
transaction management in user code. This would be particularly useful for
functions that always require transactional behavior.
>
> With automatic transaction wrapping, the same function would look much
cleaner:
>
> public class TransactionalTupleFunction implements Function<String, Void> {
> @Override
> public Void process(String input, Context context) throws Exception {
> // Transaction is automatically managed by the runtime
> context.newOutputMessage(input + "_processed", "topic-a").send();
> context.newOutputMessage(input.length(), "topic-b").send();
> return null;
> }
> }
> The implementation details could also handle the transactional
acknowledgement of the processed input message, ensuring that message
consumption, processing, and output are all part of the same atomic operation.
@wzhramc Thanks, looks very good and this all makes sense. Would you like to
start contributing this improvement to Pulsar?
In Pulsar, we have [the PIP (Pulsar Improvement
Process)](https://github.com/apache/pulsar/blob/master/pip/README.md) for
handling such changes. The first step would be to [join the dev mailing
list](https://pulsar.apache.org/contact/#mailing-lists) and start a discussion
thread about the initial plan. You could then go ahead and create the PIP
document to capture at least the high level design that impacts interfaces.
LLM helps a lot in creating the first draft when you provide an example PIP
(such as [PIP-379](https://github.com/apache/pulsar/blob/master/pip/pip-379.md)
or [PIP-430](https://github.com/apache/pulsar/blob/master/pip/pip-430.md)) and
[the template](https://github.com/apache/pulsar/blob/master/pip/TEMPLATE.md) as
reference examples of the PIP document structure.
Many PIPs contain a lot of implementation level details, but I personally
think that it's better to get alignment in the Pulsar project about the
direction first before putting too much effort on implementation details of how
this can be implemented. The implementation details such as what internal
changes are needed in the underlying PulsarSink/*PulsarSource classes to
support transactions can be covered in the PRs and PR reviews. Obviously if you
are able to do a PoC already at the time of creating the PIP, it helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
