Hi, I really like the ideas that came up for the rate limiter and thanks for opening the issue.
Philipp > On 28. May 2020, at 06:42, Grainier Perera <[email protected]> wrote: > > We'll start with a basic implementation and improve further. I've also > created issues to track the progress. > I'll start working on the [1] and update [2] as time permits. > > [1] RateLimitter: https://issues.apache.org/jira/browse/STREAMPIPES-147 > [2] Siddhi Processor improvements: > https://issues.apache.org/jira/browse/STREAMPIPES-148 > > Grainier Perera. > > > On Wed, 27 May 2020 at 23:25, Patrick Wiener <[email protected]> wrote: > >> also like the idea to check how others are tackling such scenarios. maybe >> we can >> adapt and improve from potential suitable designs. >> >> other than that I’m with Dominik to start with a basic functionality for a >> rate limiting processor >> and further extend it. >> >> @Grainier: true, in Siddhi this would also include having multiple „steps“ >> or queries. >> I still think it’s worth extending our current toolbox with new Siddhi >> processors to also >> support more lightweight deployments, e.g. the aggregation processor [1] >> is currently only >> available in flink - but that’s another topic :) >> >> Patrick >> >> [1] >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-processors-aggregation-flink/src/main/java/org/apache/streampipes/processors/aggregation/flink/processor/aggregation >> < >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-processors-aggregation-flink/src/main/java/org/apache/streampipes/processors/aggregation/flink/processor/aggregation >>> >> >>> Am 27.05.2020 um 19:08 schrieb Dominik Riemer <[email protected]>: >>> >>> Hi, >>> >>> +1 for a new pipeline element/standalone feature to rate limit events. >>> >>> Just as an idea, in a version 2 or 3 we could also extend this further: >>> - allow users to configure to not only emit the first or last event, but >> also to perform some aggregation function before emitting, e.g., output the >> average value from a 15-minute time window. >>> - output the event at fixed intervals (e.g., every full hour) >>> >>> @Grainier, great I think it's a good idea to analyze how other tools are >> doing this! >>> >>> Dominik >>> >>> -----Original Message----- >>> From: Grainier Perera <[email protected]> >>> Sent: Wednesday, May 27, 2020 2:01 PM >>> To: [email protected] >>> Subject: Re: Telegram publisher to send event notifications >>> >>> Hi all, >>> >>> I also think it should be a standalone feature. And the RateLimiter >> should allow users to configure; >>> >>> - a window (time or count) >>> - partitioning field (because there can be many sensors sending data to >>> the same stream) >>> - event to emit (first or last event of the window) >>> - retry logic. (something like; if there are no new events on the >>> current window, emit the last event of the previous window) >>> - etc... >>> >>> So we can use that with existing filters, sequences (which we can >> improve for event change capture), and notification sinks to tackle most of >> the above scenarios. >>> Even with Siddhi, we have to use several queries to create a pipeline >> and achieve the same. i.e; >>> >>> define stream TemperatureStream (sensorId string, temperature float, >>>> timestamp int); >>> >>> >>>> -- Filter[1] to capture tempStatus >>>> from TemperatureStream >>>> select sensorId, ifThenElse((temperature < -10 || 50 < temperature), >>>> "abnormal", "normal") as as tempStatus insert into >>>> ProcessedTemperatureStream; >>> >>> >>>> -- Sequence[2] to tempStatus change (realtime) from every >>>> e1=ProcessedTemperatureStream, >>>> e2=ProcessedTemperatureStream[e1.tempStatus != tempStatus] group by >>>> sensorId select e1.sensorId, e2.tempStatus insert into >>>> NotificationStream; >>> >>> >>>> -- Ratelimit[3] to send only the last notification per 15 min >>>> (somewhat >>>> batching) >>>> from NotificationStream >>>> select * >>>> group by sensorId >>>> output last every 15 min >>>> insert into RateLimitedNotificationStream; >>> >>> >>> Further, I'll do some research on how similar products have done this, >> and update the thread with how we can implement the same with StreamPipes. >>> What do you think? >>> >>> [1] https://siddhi.io/en/v5.1/docs/examples/if-then-else/ >>> [2] https://siddhi.io/en/v5.1/docs/examples/simple-sequence/ >>> [3] https://siddhi.io/en/v5.1/docs/examples/time-rate-limit/ >>> >>> Grainier Perera. >>> >>> >>> On Wed, 27 May 2020 at 14:02, Patrick Wiener <[email protected]> wrote: >>> >>>> I think this should not be a feature included in the notification sinks. >>>> Rather it should be part of the dedicated PE either as a „standalone“ >>>> feature such as a rate limiter or as part of the application logic >>>> itself. >>>> >>>> For the latter, lets consider a CEP like use case where we want to >>>> monitor sensor temperature in a given range, i.e. where there exists a >>>> „normal“ range (e.g. 50+-10°C). Everything below or above would be >>>> abnormal and should throw a warning. >>>> >>>> Now the question is, when is that warning thrown and how often? And >>>> can the warning be reset e.g. by getting back in the normal range >>>> within a given time/count window? >>>> >>>> These involve questions in how to design such a pattern or pattern >>>> sequences, how to handle consecutive occurrences of events satisfying >>>> the pattern. >>>> >>>> Now we could imagine various output options: >>>> >>>> 1) output warning once and never again >>>> 2) output warning and wait for „reset“ of warning state, e.g. >>>> temperature back in normal range for a given time >>>> 2) output warning and set a timeout, after timeout if pattern still >>>> satisfied output another reminder warning event >>>> 3) output warning every time pattern is satisfied >>>> >>>> IMO, these choices greatly depend on the individual users. However, we >>>> should somehow provide her/him with possible options to suit their use >>>> case and prevent spamming notifications. >>>> >>>> @Grainier: would such a scenario be possible to implement out of the >>>> box using Siddhi? >>>> >>>> For all other simple scenarios, I guess an output rate-limiter giving >>>> the user a simple way of „reducing“ the messages send (e.g. send 1 of >>>> 10 events) out to the notification sink (Slack, Telegram etc) should >>>> be feasible. >>>> >>>> Any other thoughts? >>>> >>>> Patrick >>>> >>>> >>>> >>>>> Am 27.05.2020 um 06:54 schrieb Philipp Zehnder <[email protected]>: >>>>> >>>>> I agree, maybe it is better to keep the notification-sinks as simple >>>>> as >>>> possible. >>>>> Then users could also use multiple channels (e.g. Email, Slack, and >>>> Telegram) for their notification and have a single output rate-limiter >>>> in the pipeline. >>>>> I was just wondering, if that’s a feature a user would expect in >>>>> each >>>> data sink or if this is an additional functionality for a separate >>>> pipeline element? >>>>> >>>>> Which functionalities would such a rate-limiter need? >>>>> I think it should not ‘just’ remove events, e.g. when a user is not >>>> reacting to the situation we might have to send the notification again >>>> after some time. >>>>> Any thoughts on that? >>>>> >>>>> Philipp >>>>> >>>>> >>>>>> On 26. May 2020, at 19:11, Grainier Perera >>>>>> <[email protected]> >>>> wrote: >>>>>> >>>>>> IMO, the sink shouldn't bother which events to publish. Instead, we >>>>>> can have event change capture & output rate-limiting as separate >>>>>> pipeline elements which can be used with notification sinks. >>>> So it >>>>>> gives users the flexibility to limit events or not, depending on >>>>>> their use-case. What do you think? >>>>>> >>>>>> pipeline: [event-source -> output rate-limiter -> >>>>>> notification-sink] >>>>>> >>>>>> Grainier. >>>>>> >>>>>> On Tue, 26 May 2020 at 21:47, Philipp Zehnder <[email protected]> >>>> wrote: >>>>>> >>>>>>> Hi Grainier, >>>>>>> >>>>>>> this sink is very cool and useful. I directly merged it. >>>>>>> >>>>>>> @all I think we still have a little conceptual problem with our >>>>>>> notification sinks (e.g. Notification, Email, …) Currently we emit >>>>>>> a notification for each event, even if the event did >>>> not >>>>>>> change. This could annoy users and also be counter productive as >>>>>>> you >>>> don't >>>>>>> want to receive hundreds of messages. >>>>>>> We need a solution to set the maximum number of notification or >>>>>>> just notify the user when the situation changed. >>>>>>> Any ideas on that? >>>>>>> >>>>>>> Philipp >>>>>>> >>>>>>>> On 26. May 2020, at 16:28, Patrick Wiener <[email protected]> >> wrote: >>>>>>>> >>>>>>>> Hi Grainier, >>>>>>>> >>>>>>>> this is really nice. We were at some point actually talking about >>>> such a >>>>>>> sink so super cool >>>>>>>> that you implemented it. I used telegram once in another project >>>>>>>> where >>>>>>> one was informed >>>>>>>> about alerts/warnings and quite liked it. >>>>>>>> >>>>>>>> Thanks for the contribution and valuable work. >>>>>>>> >>>>>>>> Patrick >>>>>>>> >>>>>>>> >>>>>>>>> Am 26.05.2020 um 15:25 schrieb Grainier Perera >>>>>>>>> <[email protected] >>>>> : >>>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I've implemented a publisher to send notifications to a Telegram >>>>>>> channel. >>>>>>>>> Since Telegram[1] is a free and cross-platform application, I >>>>>>>>> think >>>> this >>>>>>>>> will be a good addition to notification sinks. What do you think? >>>>>>>>> >>>>>>>>> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-144 >>>>>>>>> PR: >>>> https://github.com/apache/incubator-streampipes-extensions/pull/16 >>>>>>>>> >>>>>>>>> [1] https://telegram.org/ >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Grainier. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>> >>>> >>> >> >>
