We'll start with a basic implementation and improve further. I've also created issues to track the progress. I'll start working on the [1] and update [2] as time permits.
[1] RateLimitter: https://issues.apache.org/jira/browse/STREAMPIPES-147 [2] Siddhi Processor improvements: https://issues.apache.org/jira/browse/STREAMPIPES-148 Grainier Perera. On Wed, 27 May 2020 at 23:25, Patrick Wiener <[email protected]> wrote: > also like the idea to check how others are tackling such scenarios. maybe > we can > adapt and improve from potential suitable designs. > > other than that I’m with Dominik to start with a basic functionality for a > rate limiting processor > and further extend it. > > @Grainier: true, in Siddhi this would also include having multiple „steps“ > or queries. > I still think it’s worth extending our current toolbox with new Siddhi > processors to also > support more lightweight deployments, e.g. the aggregation processor [1] > is currently only > available in flink - but that’s another topic :) > > Patrick > > [1] > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-processors-aggregation-flink/src/main/java/org/apache/streampipes/processors/aggregation/flink/processor/aggregation > < > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-processors-aggregation-flink/src/main/java/org/apache/streampipes/processors/aggregation/flink/processor/aggregation > > > > > Am 27.05.2020 um 19:08 schrieb Dominik Riemer <[email protected]>: > > > > Hi, > > > > +1 for a new pipeline element/standalone feature to rate limit events. > > > > Just as an idea, in a version 2 or 3 we could also extend this further: > > - allow users to configure to not only emit the first or last event, but > also to perform some aggregation function before emitting, e.g., output the > average value from a 15-minute time window. > > - output the event at fixed intervals (e.g., every full hour) > > > > @Grainier, great I think it's a good idea to analyze how other tools are > doing this! > > > > Dominik > > > > -----Original Message----- > > From: Grainier Perera <[email protected]> > > Sent: Wednesday, May 27, 2020 2:01 PM > > To: [email protected] > > Subject: Re: Telegram publisher to send event notifications > > > > Hi all, > > > > I also think it should be a standalone feature. And the RateLimiter > should allow users to configure; > > > > - a window (time or count) > > - partitioning field (because there can be many sensors sending data to > > the same stream) > > - event to emit (first or last event of the window) > > - retry logic. (something like; if there are no new events on the > > current window, emit the last event of the previous window) > > - etc... > > > > So we can use that with existing filters, sequences (which we can > improve for event change capture), and notification sinks to tackle most of > the above scenarios. > > Even with Siddhi, we have to use several queries to create a pipeline > and achieve the same. i.e; > > > > define stream TemperatureStream (sensorId string, temperature float, > >> timestamp int); > > > > > >> -- Filter[1] to capture tempStatus > >> from TemperatureStream > >> select sensorId, ifThenElse((temperature < -10 || 50 < temperature), > >> "abnormal", "normal") as as tempStatus insert into > >> ProcessedTemperatureStream; > > > > > >> -- Sequence[2] to tempStatus change (realtime) from every > >> e1=ProcessedTemperatureStream, > >> e2=ProcessedTemperatureStream[e1.tempStatus != tempStatus] group by > >> sensorId select e1.sensorId, e2.tempStatus insert into > >> NotificationStream; > > > > > >> -- Ratelimit[3] to send only the last notification per 15 min > >> (somewhat > >> batching) > >> from NotificationStream > >> select * > >> group by sensorId > >> output last every 15 min > >> insert into RateLimitedNotificationStream; > > > > > > Further, I'll do some research on how similar products have done this, > and update the thread with how we can implement the same with StreamPipes. > > What do you think? > > > > [1] https://siddhi.io/en/v5.1/docs/examples/if-then-else/ > > [2] https://siddhi.io/en/v5.1/docs/examples/simple-sequence/ > > [3] https://siddhi.io/en/v5.1/docs/examples/time-rate-limit/ > > > > Grainier Perera. > > > > > > On Wed, 27 May 2020 at 14:02, Patrick Wiener <[email protected]> wrote: > > > >> I think this should not be a feature included in the notification sinks. > >> Rather it should be part of the dedicated PE either as a „standalone“ > >> feature such as a rate limiter or as part of the application logic > >> itself. > >> > >> For the latter, lets consider a CEP like use case where we want to > >> monitor sensor temperature in a given range, i.e. where there exists a > >> „normal“ range (e.g. 50+-10°C). Everything below or above would be > >> abnormal and should throw a warning. > >> > >> Now the question is, when is that warning thrown and how often? And > >> can the warning be reset e.g. by getting back in the normal range > >> within a given time/count window? > >> > >> These involve questions in how to design such a pattern or pattern > >> sequences, how to handle consecutive occurrences of events satisfying > >> the pattern. > >> > >> Now we could imagine various output options: > >> > >> 1) output warning once and never again > >> 2) output warning and wait for „reset“ of warning state, e.g. > >> temperature back in normal range for a given time > >> 2) output warning and set a timeout, after timeout if pattern still > >> satisfied output another reminder warning event > >> 3) output warning every time pattern is satisfied > >> > >> IMO, these choices greatly depend on the individual users. However, we > >> should somehow provide her/him with possible options to suit their use > >> case and prevent spamming notifications. > >> > >> @Grainier: would such a scenario be possible to implement out of the > >> box using Siddhi? > >> > >> For all other simple scenarios, I guess an output rate-limiter giving > >> the user a simple way of „reducing“ the messages send (e.g. send 1 of > >> 10 events) out to the notification sink (Slack, Telegram etc) should > >> be feasible. > >> > >> Any other thoughts? > >> > >> Patrick > >> > >> > >> > >>> Am 27.05.2020 um 06:54 schrieb Philipp Zehnder <[email protected]>: > >>> > >>> I agree, maybe it is better to keep the notification-sinks as simple > >>> as > >> possible. > >>> Then users could also use multiple channels (e.g. Email, Slack, and > >> Telegram) for their notification and have a single output rate-limiter > >> in the pipeline. > >>> I was just wondering, if that’s a feature a user would expect in > >>> each > >> data sink or if this is an additional functionality for a separate > >> pipeline element? > >>> > >>> Which functionalities would such a rate-limiter need? > >>> I think it should not ‘just’ remove events, e.g. when a user is not > >> reacting to the situation we might have to send the notification again > >> after some time. > >>> Any thoughts on that? > >>> > >>> Philipp > >>> > >>> > >>>> On 26. May 2020, at 19:11, Grainier Perera > >>>> <[email protected]> > >> wrote: > >>>> > >>>> IMO, the sink shouldn't bother which events to publish. Instead, we > >>>> can have event change capture & output rate-limiting as separate > >>>> pipeline elements which can be used with notification sinks. > >> So it > >>>> gives users the flexibility to limit events or not, depending on > >>>> their use-case. What do you think? > >>>> > >>>> pipeline: [event-source -> output rate-limiter -> > >>>> notification-sink] > >>>> > >>>> Grainier. > >>>> > >>>> On Tue, 26 May 2020 at 21:47, Philipp Zehnder <[email protected]> > >> wrote: > >>>> > >>>>> Hi Grainier, > >>>>> > >>>>> this sink is very cool and useful. I directly merged it. > >>>>> > >>>>> @all I think we still have a little conceptual problem with our > >>>>> notification sinks (e.g. Notification, Email, …) Currently we emit > >>>>> a notification for each event, even if the event did > >> not > >>>>> change. This could annoy users and also be counter productive as > >>>>> you > >> don't > >>>>> want to receive hundreds of messages. > >>>>> We need a solution to set the maximum number of notification or > >>>>> just notify the user when the situation changed. > >>>>> Any ideas on that? > >>>>> > >>>>> Philipp > >>>>> > >>>>>> On 26. May 2020, at 16:28, Patrick Wiener <[email protected]> > wrote: > >>>>>> > >>>>>> Hi Grainier, > >>>>>> > >>>>>> this is really nice. We were at some point actually talking about > >> such a > >>>>> sink so super cool > >>>>>> that you implemented it. I used telegram once in another project > >>>>>> where > >>>>> one was informed > >>>>>> about alerts/warnings and quite liked it. > >>>>>> > >>>>>> Thanks for the contribution and valuable work. > >>>>>> > >>>>>> Patrick > >>>>>> > >>>>>> > >>>>>>> Am 26.05.2020 um 15:25 schrieb Grainier Perera > >>>>>>> <[email protected] > >>> : > >>>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I've implemented a publisher to send notifications to a Telegram > >>>>> channel. > >>>>>>> Since Telegram[1] is a free and cross-platform application, I > >>>>>>> think > >> this > >>>>>>> will be a good addition to notification sinks. What do you think? > >>>>>>> > >>>>>>> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-144 > >>>>>>> PR: > >> https://github.com/apache/incubator-streampipes-extensions/pull/16 > >>>>>>> > >>>>>>> [1] https://telegram.org/ > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Grainier. > >>>>>> > >>>>> > >>>>> > >>>>> > >>> > >>> > >> > >> > > > >
