Hi, I created a wiki page and added our initial thoughts: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158868087
I'll go through the existing wrapper implementation and add any ideas to the wiki. If you have any ideas, feel free to edit the page! Dominik -----Original Message----- From: Patrick Wiener <wie...@apache.org> Sent: Friday, July 17, 2020 7:35 AM To: dev@streampipes.apache.org Subject: Re: Adding window semantics to the sdk/core I also tend towards option 2. It makes it more intuative to develop windowed operations by having a dedicated windowed event processor. We can collect the design choices etc in the wiki and discussion points in the wiki. Patrick > Am 16.07.2020 um 22:55 schrieb Philipp Zehnder <zehn...@apache.org>: > > Hi, > > +1 for creating a wiki page and collecting all the requirements and > functionalities. > > Philipp > >> On 16. Jul 2020, at 20:00, Dominik Riemer <rie...@apache.org> wrote: >> >> Hi, >> >> I'm also leaning towards option 2 - we could introduce new engine and >> controller classes for windowed event processors (e.g., >> WindowedDataProcessor) which handle the windowing logic internally and only >> expose the higher-level methods to users (as you proposed, >> onCurrentEvent...). Window parameters could be automatically added to the >> controller without further configuration. >> Probably the biggest hurdle is to introduce an API design that works for all >> available wrappers, without re-implementing a new higher-level engine. But >> as Philipp said, it probably makes sense to start with one of the >> lightweight wrappers such as Java or Siddhi. >> Should we create a wiki page to collect design requirements for such a >> feature? >> >> Dominik >> >> -----Original Message----- >> From: Philipp Zehnder <zehn...@apache.org> >> Sent: Wednesday, July 15, 2020 4:13 PM >> To: Grainier Perera <grain...@apache.org> >> Cc: dev@streampipes.apache.org >> Subject: Re: Adding window semantics to the sdk/core >> >> Hi Grainier, >> >> That's a very good point. >> I am leaning towards proposal 2, because if we have a separate PE, the user >> must be aware of windowing and how to use windows. >> I think it is easier for the user to configure the relevant parameters >> directly in the processor configuration itself. Whats your opinion on that? >> However, I'm afraid that we will then have to implement all functionalities >> twice, one version for single values and one version for the windowed >> version (e.g. numerical filter). >> Perhaps we can find a good solution for this. (e.g. make the window function >> optional). >> >> Maybe we can also make a list of processors that we think need this >> functionality? >> The first thing that came to my mind was the aggregation component. >> Currently we only have a Flink version of it, and I think it would be very >> helpful to have a lighter version as well (e.g. Java, or Siddhi). >> >> Philipp >> PS: The JS Evaluator is really awesome, I use it a lot and it’s a big time >> saver! >> >>> On 15. Jul 2020, at 07:06, Grainier Perera <grain...@apache.org> wrote: >>> >>> Hi all, >>> >>> Hope you are doing well. I'm starting this thread to discuss $title. As we >>> discussed in the mail thread [1], having a common window semantic that can >>> be leveraged by every PE can be very useful. >>> >>> I'm thinking of two ways to achieve the same; Introduce a dedicated >>> Window PE which can be used before any existing PE. >>> No need for API changes. >>> But, might have to flag event in a way for the next processor to identify, >>> to which window the event belongs to (i.e with sliding windows, etc...). >>> So, have to change the processing logic of existing PEs to check event flag >>> and process accordingly. >>> So all the existing PEs might not work with window semantics. In >>> that case, we need a way to show window compatible PEs (because, >>> having a window before a normal PE might result in un-expected outputs) >>> Introduce windowed EventProcessor/EventSink APIs which allows users to >>> write their own windowed extensions (i.e aggregators, etc...) No need for >>> event flagging. Can introduce API methods like onCurrentEvent, >>> onExpiredEvent, onResetEvent, etc... >>> Need API changes/refactoring in EventProcessor/EventSink, as well as in >>> existing PEs. >>> Need a way to expose the window related parameters through existing PEs >>> DataProcessorDescription. >>> Since there're pros/cons to both, what do you think is the best approach? >>> Or is there any other approach that we can try out? >>> >>> [1] PE to rate-limit events >>> >>> Grainier Perera. >> >> >
