also like the idea to check how others are tackling such scenarios. maybe we can adapt and improve from potential suitable designs.
other than that I’m with Dominik to start with a basic functionality for a rate limiting processor and further extend it. @Grainier: true, in Siddhi this would also include having multiple „steps“ or queries. I still think it’s worth extending our current toolbox with new Siddhi processors to also support more lightweight deployments, e.g. the aggregation processor [1] is currently only available in flink - but that’s another topic :) Patrick [1] https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-processors-aggregation-flink/src/main/java/org/apache/streampipes/processors/aggregation/flink/processor/aggregation <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-processors-aggregation-flink/src/main/java/org/apache/streampipes/processors/aggregation/flink/processor/aggregation> > Am 27.05.2020 um 19:08 schrieb Dominik Riemer <rie...@apache.org>: > > Hi, > > +1 for a new pipeline element/standalone feature to rate limit events. > > Just as an idea, in a version 2 or 3 we could also extend this further: > - allow users to configure to not only emit the first or last event, but also > to perform some aggregation function before emitting, e.g., output the > average value from a 15-minute time window. > - output the event at fixed intervals (e.g., every full hour) > > @Grainier, great I think it's a good idea to analyze how other tools are > doing this! > > Dominik > > -----Original Message----- > From: Grainier Perera <grain...@apache.org> > Sent: Wednesday, May 27, 2020 2:01 PM > To: dev@streampipes.apache.org > Subject: Re: Telegram publisher to send event notifications > > Hi all, > > I also think it should be a standalone feature. And the RateLimiter should > allow users to configure; > > - a window (time or count) > - partitioning field (because there can be many sensors sending data to > the same stream) > - event to emit (first or last event of the window) > - retry logic. (something like; if there are no new events on the > current window, emit the last event of the previous window) > - etc... > > So we can use that with existing filters, sequences (which we can improve for > event change capture), and notification sinks to tackle most of the above > scenarios. > Even with Siddhi, we have to use several queries to create a pipeline and > achieve the same. i.e; > > define stream TemperatureStream (sensorId string, temperature float, >> timestamp int); > > >> -- Filter[1] to capture tempStatus >> from TemperatureStream >> select sensorId, ifThenElse((temperature < -10 || 50 < temperature), >> "abnormal", "normal") as as tempStatus insert into >> ProcessedTemperatureStream; > > >> -- Sequence[2] to tempStatus change (realtime) from every >> e1=ProcessedTemperatureStream, >> e2=ProcessedTemperatureStream[e1.tempStatus != tempStatus] group by >> sensorId select e1.sensorId, e2.tempStatus insert into >> NotificationStream; > > >> -- Ratelimit[3] to send only the last notification per 15 min >> (somewhat >> batching) >> from NotificationStream >> select * >> group by sensorId >> output last every 15 min >> insert into RateLimitedNotificationStream; > > > Further, I'll do some research on how similar products have done this, and > update the thread with how we can implement the same with StreamPipes. > What do you think? > > [1] https://siddhi.io/en/v5.1/docs/examples/if-then-else/ > [2] https://siddhi.io/en/v5.1/docs/examples/simple-sequence/ > [3] https://siddhi.io/en/v5.1/docs/examples/time-rate-limit/ > > Grainier Perera. > > > On Wed, 27 May 2020 at 14:02, Patrick Wiener <wie...@apache.org> wrote: > >> I think this should not be a feature included in the notification sinks. >> Rather it should be part of the dedicated PE either as a „standalone“ >> feature such as a rate limiter or as part of the application logic >> itself. >> >> For the latter, lets consider a CEP like use case where we want to >> monitor sensor temperature in a given range, i.e. where there exists a >> „normal“ range (e.g. 50+-10°C). Everything below or above would be >> abnormal and should throw a warning. >> >> Now the question is, when is that warning thrown and how often? And >> can the warning be reset e.g. by getting back in the normal range >> within a given time/count window? >> >> These involve questions in how to design such a pattern or pattern >> sequences, how to handle consecutive occurrences of events satisfying >> the pattern. >> >> Now we could imagine various output options: >> >> 1) output warning once and never again >> 2) output warning and wait for „reset“ of warning state, e.g. >> temperature back in normal range for a given time >> 2) output warning and set a timeout, after timeout if pattern still >> satisfied output another reminder warning event >> 3) output warning every time pattern is satisfied >> >> IMO, these choices greatly depend on the individual users. However, we >> should somehow provide her/him with possible options to suit their use >> case and prevent spamming notifications. >> >> @Grainier: would such a scenario be possible to implement out of the >> box using Siddhi? >> >> For all other simple scenarios, I guess an output rate-limiter giving >> the user a simple way of „reducing“ the messages send (e.g. send 1 of >> 10 events) out to the notification sink (Slack, Telegram etc) should >> be feasible. >> >> Any other thoughts? >> >> Patrick >> >> >> >>> Am 27.05.2020 um 06:54 schrieb Philipp Zehnder <zehn...@apache.org>: >>> >>> I agree, maybe it is better to keep the notification-sinks as simple >>> as >> possible. >>> Then users could also use multiple channels (e.g. Email, Slack, and >> Telegram) for their notification and have a single output rate-limiter >> in the pipeline. >>> I was just wondering, if that’s a feature a user would expect in >>> each >> data sink or if this is an additional functionality for a separate >> pipeline element? >>> >>> Which functionalities would such a rate-limiter need? >>> I think it should not ‘just’ remove events, e.g. when a user is not >> reacting to the situation we might have to send the notification again >> after some time. >>> Any thoughts on that? >>> >>> Philipp >>> >>> >>>> On 26. May 2020, at 19:11, Grainier Perera >>>> <grainier.per...@gmail.com> >> wrote: >>>> >>>> IMO, the sink shouldn't bother which events to publish. Instead, we >>>> can have event change capture & output rate-limiting as separate >>>> pipeline elements which can be used with notification sinks. >> So it >>>> gives users the flexibility to limit events or not, depending on >>>> their use-case. What do you think? >>>> >>>> pipeline: [event-source -> output rate-limiter -> >>>> notification-sink] >>>> >>>> Grainier. >>>> >>>> On Tue, 26 May 2020 at 21:47, Philipp Zehnder <zehn...@apache.org> >> wrote: >>>> >>>>> Hi Grainier, >>>>> >>>>> this sink is very cool and useful. I directly merged it. >>>>> >>>>> @all I think we still have a little conceptual problem with our >>>>> notification sinks (e.g. Notification, Email, …) Currently we emit >>>>> a notification for each event, even if the event did >> not >>>>> change. This could annoy users and also be counter productive as >>>>> you >> don't >>>>> want to receive hundreds of messages. >>>>> We need a solution to set the maximum number of notification or >>>>> just notify the user when the situation changed. >>>>> Any ideas on that? >>>>> >>>>> Philipp >>>>> >>>>>> On 26. May 2020, at 16:28, Patrick Wiener <wie...@apache.org> wrote: >>>>>> >>>>>> Hi Grainier, >>>>>> >>>>>> this is really nice. We were at some point actually talking about >> such a >>>>> sink so super cool >>>>>> that you implemented it. I used telegram once in another project >>>>>> where >>>>> one was informed >>>>>> about alerts/warnings and quite liked it. >>>>>> >>>>>> Thanks for the contribution and valuable work. >>>>>> >>>>>> Patrick >>>>>> >>>>>> >>>>>>> Am 26.05.2020 um 15:25 schrieb Grainier Perera >>>>>>> <grain...@apache.org >>> : >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I've implemented a publisher to send notifications to a Telegram >>>>> channel. >>>>>>> Since Telegram[1] is a free and cross-platform application, I >>>>>>> think >> this >>>>>>> will be a good addition to notification sinks. What do you think? >>>>>>> >>>>>>> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-144 >>>>>>> PR: >> https://github.com/apache/incubator-streampipes-extensions/pull/16 >>>>>>> >>>>>>> [1] https://telegram.org/ >>>>>>> >>>>>>> Thanks, >>>>>>> Grainier. >>>>>> >>>>> >>>>> >>>>> >>> >>> >> >> >
