GitHub user amanchaudhary-95 added a comment to the discussion: Feature Request: Edge-Side Buffering for StreamPipes Adapters
@tenthe Let me explain my current setup. I'm running two independent StreamPipes instances on two separate edge systems. Both StreamPipes instances connect to the same PLC, and together they act as an HA / failover pair. At the edge, each StreamPipes instance sends data to Kafka, which I use as an edge buffer. Kafka persists messages but cannot automatically forward data to the remote system, so I'm using Apache Flink on the remote side to pull messages from both Kafka topics and write them into my remote time-series database (IoTDB). At the edge, I am also de-duplicating the messages before they are finally pushed to IoTDB. Below are screenshots from one of the edge StreamPipes systems: **At the Edge** * StreamPipes adapter reading from PLC <img width="1502" height="484" alt="image" src="https://github.com/user-attachments/assets/c9bcd0a5-bf44-4995-aa9f-007204cf8917"; /> * Pipeline element sending the events into Kafka <img width="1042" height="684" alt="image" src="https://github.com/user-attachments/assets/cab9a641-c5a2-4762-a2e3-7c3f0be0a6c5"; /> However, I believe your suggestion: > **StreamPipes adapter level:** > Implement buffering directly inside the StreamPipes adapter independent of > the protocol used. …would indeed be a much better long-term solution. Answer to your first question: > What happens in the current implementation if the connection is interrupted? > Does the behavior differ based on the protocol? Right now, the adapter does not store data locally. If the connection breaks, the data is lost permanently. So yes, this behavior is independent of the protocol — the adapter has no built-in persistence. Answer to your second question: > Handling out-of-order events in pipelines and functions. Normally the sensor data arrives in order, but this is still a real concern. In my current setup, Kafka guarantees message order per partition, so offsets ensure that events are processed correctly and consistently. If we implement buffering at the StreamPipes adapter level, we can simply add a timestamp tag (if the PLC does not already provide one). This timestamp can then be used to handle out-of-order events during replay or buffering. I'll submit a separate idea to implement streampipes in cluster so that 2 or more streampipes can act in tandem and can act as HA/Failover. GitHub link: https://github.com/apache/streampipes/discussions/3932#discussioncomment-14996896 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
