GitHub user KIC created a discussion: Pulsar functions should be able to return none and multiple values
First of all, I love pulsar functions! It is almost exactly what I have always wanted to build. But just by myself and as a hobby project I never really finished it. So I am really happy someone else did it! However I am missing one important feature. **Is your feature request related to a problem? Please describe.** As I understand the current implementation of pulsar functions is a 1:1 relation ship. One event in -> one event out. This is very limiting as one can not even write a function to filter out events. Also there are use cases when you get a batched message and you need to "unpack" it into single events. Or when you need to interpolate values from a previous event (which is held in the state). **Describe the solution you'd like** I propose that the interface should be something along the lines `Function<I, ? extends Collection<O>>`. This way you can either return nothing (an empty list), a single element, but also a collection of like interpolated values. **Describe alternatives you've considered** If you consider a `PublishFunction` then I see here the following problem. In the very moment you also need to store a state via the `Context` you get a timing issue. What if you stored the state but then for some reason you are not able to send to the topic. Or even worse what if you could send _n_ of _m_ messages and then the network fails? I would be clean and easier when pulsar handles all these cases outside of the function implementation. **Additional context** One not necessarily needs to use atomic transactions over different storage solutions for this use case. Functions just need to be deterministic. So during startup (or retry) you just need to know what is needed to reproduce the failed "state" (nacked message) and you need to know what was the last message which has been sent to the target topic. You then store the state and only send the missing messages after the last one which was already sent. GitHub link: https://github.com/apache/pulsar/discussions/19657 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
