Hi all, I also think it should be a standalone feature. And the RateLimiter should allow users to configure;
- a window (time or count) - partitioning field (because there can be many sensors sending data to the same stream) - event to emit (first or last event of the window) - retry logic. (something like; if there are no new events on the current window, emit the last event of the previous window) - etc... So we can use that with existing filters, sequences (which we can improve for event change capture), and notification sinks to tackle most of the above scenarios. Even with Siddhi, we have to use several queries to create a pipeline and achieve the same. i.e; define stream TemperatureStream (sensorId string, temperature float, > timestamp int); > -- Filter[1] to capture tempStatus > from TemperatureStream > select sensorId, ifThenElse((temperature < -10 || 50 < temperature), > "abnormal", "normal") as as tempStatus > insert into ProcessedTemperatureStream; > -- Sequence[2] to tempStatus change (realtime) > from every e1=ProcessedTemperatureStream, > e2=ProcessedTemperatureStream[e1.tempStatus != tempStatus] > group by sensorId > select e1.sensorId, e2.tempStatus > insert into NotificationStream; > -- Ratelimit[3] to send only the last notification per 15 min (somewhat > batching) > from NotificationStream > select * > group by sensorId > output last every 15 min > insert into RateLimitedNotificationStream; Further, I'll do some research on how similar products have done this, and update the thread with how we can implement the same with StreamPipes. What do you think? [1] https://siddhi.io/en/v5.1/docs/examples/if-then-else/ [2] https://siddhi.io/en/v5.1/docs/examples/simple-sequence/ [3] https://siddhi.io/en/v5.1/docs/examples/time-rate-limit/ Grainier Perera. On Wed, 27 May 2020 at 14:02, Patrick Wiener <[email protected]> wrote: > I think this should not be a feature included in the notification sinks. > Rather it should be part of the dedicated PE either > as a „standalone“ feature such as a rate limiter or as part of the > application logic itself. > > For the latter, lets consider a CEP like use case where we want to monitor > sensor temperature in a given range, i.e. where > there exists a „normal“ range (e.g. 50+-10°C). Everything below or above > would be abnormal and should throw a warning. > > Now the question is, when is that warning thrown and how often? And can > the warning be reset e.g. by getting back in the > normal range within a given time/count window? > > These involve questions in how to design such a pattern or pattern > sequences, how to handle consecutive occurrences of > events satisfying the pattern. > > Now we could imagine various output options: > > 1) output warning once and never again > 2) output warning and wait for „reset“ of warning state, e.g. temperature > back in normal range for a given time > 2) output warning and set a timeout, after timeout if pattern still > satisfied output another reminder warning event > 3) output warning every time pattern is satisfied > > IMO, these choices greatly depend on the individual users. However, we > should somehow provide her/him with possible > options to suit their use case and prevent spamming notifications. > > @Grainier: would such a scenario be possible to implement out of the box > using Siddhi? > > For all other simple scenarios, I guess an output rate-limiter giving the > user a simple way of „reducing“ the messages > send (e.g. send 1 of 10 events) out to the notification sink (Slack, > Telegram etc) should be feasible. > > Any other thoughts? > > Patrick > > > > > Am 27.05.2020 um 06:54 schrieb Philipp Zehnder <[email protected]>: > > > > I agree, maybe it is better to keep the notification-sinks as simple as > possible. > > Then users could also use multiple channels (e.g. Email, Slack, and > Telegram) for their notification and have a single output rate-limiter in > the pipeline. > > I was just wondering, if that’s a feature a user would expect in each > data sink or if this is an additional functionality for a separate pipeline > element? > > > > Which functionalities would such a rate-limiter need? > > I think it should not ‘just’ remove events, e.g. when a user is not > reacting to the situation we might have to send the notification again > after some time. > > Any thoughts on that? > > > > Philipp > > > > > >> On 26. May 2020, at 19:11, Grainier Perera <[email protected]> > wrote: > >> > >> IMO, the sink shouldn't bother which events to publish. Instead, we can > >> have event change capture & output rate-limiting as > >> separate pipeline elements which can be used with notification sinks. > So it > >> gives users the flexibility to limit events or not, depending on their > >> use-case. What do you think? > >> > >> pipeline: [event-source -> output rate-limiter -> notification-sink] > >> > >> Grainier. > >> > >> On Tue, 26 May 2020 at 21:47, Philipp Zehnder <[email protected]> > wrote: > >> > >>> Hi Grainier, > >>> > >>> this sink is very cool and useful. I directly merged it. > >>> > >>> @all I think we still have a little conceptual problem with our > >>> notification sinks (e.g. Notification, Email, …) > >>> Currently we emit a notification for each event, even if the event did > not > >>> change. This could annoy users and also be counter productive as you > don't > >>> want to receive hundreds of messages. > >>> We need a solution to set the maximum number of notification or just > >>> notify the user when the situation changed. > >>> Any ideas on that? > >>> > >>> Philipp > >>> > >>>> On 26. May 2020, at 16:28, Patrick Wiener <[email protected]> wrote: > >>>> > >>>> Hi Grainier, > >>>> > >>>> this is really nice. We were at some point actually talking about > such a > >>> sink so super cool > >>>> that you implemented it. I used telegram once in another project where > >>> one was informed > >>>> about alerts/warnings and quite liked it. > >>>> > >>>> Thanks for the contribution and valuable work. > >>>> > >>>> Patrick > >>>> > >>>> > >>>>> Am 26.05.2020 um 15:25 schrieb Grainier Perera <[email protected] > >: > >>>>> > >>>>> Hi all, > >>>>> > >>>>> I've implemented a publisher to send notifications to a Telegram > >>> channel. > >>>>> Since Telegram[1] is a free and cross-platform application, I think > this > >>>>> will be a good addition to notification sinks. What do you think? > >>>>> > >>>>> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-144 > >>>>> PR: > https://github.com/apache/incubator-streampipes-extensions/pull/16 > >>>>> > >>>>> [1] https://telegram.org/ > >>>>> > >>>>> Thanks, > >>>>> Grainier. > >>>> > >>> > >>> > >>> > > > > > >
