Grainier Perera created STREAMPIPES-292:
-------------------------------------------
Summary: Introduce event windowing to the StreamPipes core/sdk
Key: STREAMPIPES-292
URL: https://issues.apache.org/jira/browse/STREAMPIPES-292
Project: StreamPipes
Issue Type: New Feature
Components: Backend
Reporter: Grainier Perera
h3. +*Apache StreamPipes*+
Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to
enable non-technical users to connect, analyze and explore IoT data streams.
StreamPipes offers several modules including StreamPipes Connect to easily
connect data from industrial IoT sources, the Pipeline Editor to quickly create
processing pipelines and several visualization modules for live and historic
data exploration. Under the hood, StreamPipes utilizes an event-driven
microservice paradigm of standalone, so-called analytics microservices making
the system easy to extend for individual needs.
h3. +*Background*+
Currently, window logic can be individually defined per pipeline element. The
whole windowing logic needs to be declared in the controller and runtime logic
needs to be individually added based on the selected runtime wrapper (Java,
Siddhi, Flink, etc...).
As many data processors benefit from using window-functions (i.e PEs such as
Event Counter, Count Aggregation, Rate Limiter), windowing logic is often
duplicated as it needs to be implemented for every new pipeline element. In
addition, the feature set of supported window operators differs (and often
depends on the developer) as it is unclear which windows and parameters
should/can be offered.
Therefore, adding support for explicit window semantics to the SDK/Core would
make implementing data processors and sinks using windows much easier and less
error-prone.
+**+
h3. +*Tasks*+
# Design and introduce new processor and controller classes for windowed event
processors (e.g., WindowedDataProcessor) which handle the windowing logic
internally and only expose the higher-level methods to users (i.e
onCurrentEvent, onExpiredEvent, etc...).
# Implement internal logic for few window functions (i.e TimeWindow,
LengthWindow, TimeBatchWindow, LengthBatchWindow, etc...)
# Write a few sample pipeline-elements using your new API!
h3. +*Relevant Skills*+
* Basic knowledge in StreamPipes core (cloning the repo, going through the
codebase/documents would do).
* Basic knowledge of stream analytics window functions (this is not a must,
but it's awesome if you know your way around analytics window functions).
* Some Java experience.
h3. +*Learning Material*+
+For StreamPipes:+
* [https://streampipes.apache.org/docs/]
* [https://streampipes.apache.org/media.html]
* [https://github.com/apache/incubator-streampipes]
* [https://github.com/apache/incubator-streampipes-extensions]
+For Streaming Analytics:+
*
[https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-window-functions]
*
[https://www.mikulskibartosz.name/difference-between-tumbling-and-sliding-window/]
*
[https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/developing-storm-applications/content/understanding_sliding_and_tumbling_windows.html]
+For the context for the issue:+
* [https://www.mail-archive.com/[email protected]/msg00868.html]
+*Mentor*+
- Grainier Perera.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
