Hi Alexander, Thank you for responding. The solution you proposed uses statically defined windows. What I need a are dynamically created windows determined by metadata in the stream element.
I want the stream element to define the window. That’s what I’m trying to research, or an alternate solution. Again, thank you for your input. > On Jan 26, 2022, at 1:32 PM, Alexander Fedulov <alexan...@ververica.com> > wrote: > > > Hi Marco, > > Not sure if I get your problem correctly, but you can process those windows > on data "split" from the same input within the same Flink job. > Something along these lines: > > DataStream<SomePojo> stream = ... > DataStream<SomePojo> a = stream.filter( /* time series name == "a" */); > a.keyBy(...).window(TumblingEventTimeWindows.of(Time.minutes(1))); > > DataStream<SomePojo> b = stream.filter( /* time series name == "b" */); > b.keyBy(...).window(TumblingEventTimeWindows.of(Time.minutes(5))); > > If needed, you can then union all of the separate results streams together. > a.union(b, c ...); > > There is no need for separate Flink deployments to create such a pipeline. > > Best, > Alexander Fedulov > >> On Wed, Jan 26, 2022 at 6:47 PM Marco Villalobos <mvillalo...@kineteque.com> >> wrote: >> Hi, >> >> I am working with time series data in the form of (timestamp, name, value), >> and an event time that is the timestamp when the data was published onto >> kafka, and I have a business requirement in which each stream element >> becomes enriched, and then processing requires different time series names >> to be processed in different windows with different time averages. >> >> For example, time series with name "a" >> >> might require a one minute window, and five minute window. >> >> time series with name "b" requires no windowing. >> >> time series with name "c" requires a two minute window and 10 minute window. >> >> Does flink support this style of windowing? I think it doesn't. Also, does >> any streaming platform support that type of windowing? >> >> I was thinking that this type of windowing support might require a different >> flink deployment per each window. Would that scale though, if there are >> tens of thousands of time series names / windows? >> >> Any help or advice would be appreciated. Thank you. >> >> Marco A. Villalobos >> >>
