I have a use case which I am trying to solve using Flink. Need an advice to decide on the correct approach.
Use case: ---------- I have a stream of events partitioned by a key. For some events, I need to start a timer (conside this as a SLA i.e. if something is not done in x secs/minutes do something). Now when that timer expires I need to perform some arbitrary action (like writing to database etc). There will be some events which can cancel the timers. (i.e. if event comes before x secs we need not run SLA violation action etc.). We are considering flink because then we can reuse the scaling, fault tolerance provided by the engine rather than building our own. Current rps is ~ 200-300 but it can be expected to increase quickly. Solutions in mind: ------------------- 1. We can think it like CEP use case, where with encoding like "event1 followed by event2" with "not" in x seconds. i.e. when event 2 is "not" arrived in x seconds. I assume there will be NOT operator support. I am not sure about memory consumption in CEP. Because x seconds can be x days as well and I do not need any batching of events in memory. I just need to start a timer of x days/hours (lets say) and when it is fired just trigger something. So there is no notion of window as such. Can CEP fit in this type of use case? If the timer between events is in days, how about the memory consumption? 2. Use Flink for event processing and delegate the tasks of timers to another service i.e. when event occurs send it to kafka with timer information and then another service handles timers and send back the event again once that is done etc. Looks like many hops in this process and latency will be high if SLA is in seconds (I am thinking of using Kafka here). Is anyone aware of a better way of doing this in flink? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-timer-implementation-using-Flink-tp12581.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
