GitHub user amanchaudhary-95 created a discussion: Feature Request: Edge-Side Buffering for StreamPipes Adapters
**Summary** Implement an edge-side buffering mechanism in Apache StreamPipes so that data collected from field devices (e.g., PLCs) is not lost when network connectivity to the remote StreamPipes backend is interrupted. **Problem Statement** In many industrial and IoT scenarios, StreamPipes is deployed at the edge, collecting data from PLCs via adapters and forwarding it to a remote central StreamPipes instance. However, if the communication between edge → remote breaks (network outage, WAN failure, server maintenance, etc.), all in-flight data is lost because the edge installation currently does not buffer data locally. This creates reliability issues in environments where continuous, lossless telemetry is critical. **Requested Feature** * Add a local buffer/queue mechanism at the edge StreamPipes instance that: * Stores incoming events from adapters (e.g., PLC adapter) on local disk or durable queue. * Automatically retries sending buffered data when the remote system becomes available. * Flushes the buffer in order (FIFO) to maintain event consistency. **Allows configurable policies:** * max buffer size * retention period * retry/backoff strategy * fallback to “drop oldest” or “stop processing” modes **Why This Is Needed** * Industrial connectivity is not always reliable. * A temporary WAN outage should not cause data loss. * Many IoT/edge platforms support offline buffering (e.g., MQTT brokers, Kafka edge nodes, Telegraf, EMQX). * Adding this would significantly improve StreamPipes’ reliability in edge → cloud deployments. GitHub link: https://github.com/apache/streampipes/discussions/3932 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
