GitHub user tenthe added a comment to the discussion: Feature Request: Edge-Side Buffering for StreamPipes Adapters
Thanks for bringing up this idea — I think adding an edge-side buffering mechanism is a great idea, but we should also consider the implications it might have on downstream functionalities A few initial thoughts: **1. Where should the buffering take place?** I can think of two different levels: * **Client/Broker level:** A local MQTT/Kafka client could persist messages and automatically resend them once the connection to the remote system is restored. * **StreamPipes adapter level:** Implementing buffering directly inside the StreamPipes adapter independent of the used protocol Personally, I think the second approach is better, as we are more flexible when we integrate the functionality ourselves. **Open question:** What happens in the current implementation if the connection is interrupted? Does the behavior differ from the protocol used? **2. Handling out-of-order events in pipelines and functions** Once events are buffered and later resent, pipelines and functions may receive data out of chronological order. Currently, it is generally assumed that events arrive time-synchronized or at least in order. For many functionalities this may not be an issue, but we should identify which ones could be affected. So we should determine: * which StreamPipes functionalities are sensitive to out-of-order events, * how much time/data we want to buffer, * and whether we need serialization of events or if keeping them in memory is sufficient. What do you think? Are there any other implications we should consider? And do you have suggestions on where a good starting point for the implementation would be? GitHub link: https://github.com/apache/streampipes/discussions/3932#discussioncomment-14990291 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
