This is an automated email from the ASF dual-hosted git repository.
riemer pushed a commit to branch dev
in repository
https://gitbox.apache.org/repos/asf/incubator-streampipes-website.git
The following commit(s) were added to refs/heads/dev by this push:
new 6e429da [STREAMPIPES-197] Merge existing documentation pages into new
structure
6e429da is described below
commit 6e429dab0434dbd919515ffa23e539b30a7ca8d7
Author: Dominik Riemer <[email protected]>
AuthorDate: Tue Apr 27 21:51:26 2021 +0200
[STREAMPIPES-197] Merge existing documentation pages into new structure
---
documentation/docs/03_use-pipeline-editor.md | 15 +-
.../{dev-guide-ssl.md => 05_deploy-use-ssl.md} | 10 +-
documentation/docs/06_extend-archetypes.md | 138 +++++-
...event-model.md => 06_extend-sdk-event-model.md} | 4 +-
...egies.md => 06_extend-sdk-output-strategies.md} | 4 +-
...rties.md => 06_extend-sdk-static-properties.md} | 4 +-
...nts.md => 06_extend-sdk-stream-requirements.md} | 4 +-
documentation/docs/06_extend-setup.md | 107 ++++-
...rs.md => 06_extend-tutorial-data-processors.md} | 4 +-
...l-sinks.md => 06_extend-tutorial-data-sinks.md} | 4 +-
...urces.md => 06_extend-tutorial-data-sources.md} | 8 +-
documentation/docs/07_technicals-architecture.md | 57 ++-
...iguration.md => 07_technicals-configuration.md} | 2 +-
documentation/docs/dev-guide-archetype.md | 137 ------
documentation/docs/dev-guide-architecture.md | 56 ---
documentation/docs/dev-guide-configuration.md | 53 ---
documentation/docs/dev-guide-environment.md | 107 -----
documentation/docs/dev-guide-event-model.md | 136 ------
documentation/docs/dev-guide-output-strategies.md | 341 --------------
documentation/docs/dev-guide-ssl.md | 30 --
documentation/docs/dev-guide-static-properties.md | 259 -----------
.../docs/dev-guide-stream-requirements.md | 173 --------
.../docs/dev-guide-tutorial-processors.md | 494 ---------------------
documentation/docs/dev-guide-tutorial-sinks.md | 241 ----------
documentation/docs/dev-guide-tutorial-sources.md | 277 ------------
documentation/website/i18n/en.json | 73 ++-
documentation/website/sidebars.json | 15 +-
27 files changed, 396 insertions(+), 2357 deletions(-)
diff --git a/documentation/docs/03_use-pipeline-editor.md
b/documentation/docs/03_use-pipeline-editor.md
index b45df1c..e9ffac2 100644
--- a/documentation/docs/03_use-pipeline-editor.md
+++ b/documentation/docs/03_use-pipeline-editor.md
@@ -4,4 +4,17 @@ title: Pipeline Editor
sidebar_label: Pipeline Editor
---
-tbd
\ No newline at end of file
+## Pipeline Elements
+
+## Creating Pipelines
+
+### Configuring Pipeline Elements
+
+### Pipeline Element Options
+
+* Pipeline Element Recommendation
+
+### Pipeline Editor Options
+
+### Saving a pipeline
+
diff --git a/documentation/docs/dev-guide-ssl.md
b/documentation/docs/05_deploy-use-ssl.md
similarity index 79%
copy from documentation/docs/dev-guide-ssl.md
copy to documentation/docs/05_deploy-use-ssl.md
index a6f3453..d5762b8 100644
--- a/documentation/docs/dev-guide-ssl.md
+++ b/documentation/docs/05_deploy-use-ssl.md
@@ -1,16 +1,16 @@
---
-id: dev-guide-ssl
+id: deploy-use-ssl
title: Use SSL
sidebar_label: Use SSL
---
-In this tutorial we explain how you can use SSL Certificates to provide
transport layer security between your Browser and the Streampipes Backend.
+This page explains how SSL Certificates can be used to provide transport layer
security between your Browser and the Streampipes Backend.
## Prerequisites
You need a valid Certificate consisting of a Private and a Public Key. Both
Keys must be in PEM Format. Please note that your Private Key should never be
shared, otherwise the communication can not be considered secure.
## Edit docker-compose.yml
-In order to use SSL you have to open port 443 on the nginx Service. Incoming
insecure Traffic on Port 80 will automatically be reroutet to Port 443.
+In order to use SSL you have to open port 443 on the nginx Service. Incoming
insecure Traffic on Port 80 will be automatically rerouted to Port 443.
The Environment-Variable NGINX_SSL must be set to "true".
@@ -18,7 +18,7 @@ Finally you have to inject the Certificates into the
Docker-Container. In the ex
```yaml
[...]
nginx:
- image: streampipes/ui
+ image: apachestreampipes/ui
ports:
- "80:80"
- "443:443"
@@ -32,4 +32,4 @@ Finally you have to inject the Certificates into the
Docker-Container. In the ex
networks:
spnet:
[...]
-```
\ No newline at end of file
+```
diff --git a/documentation/docs/06_extend-archetypes.md
b/documentation/docs/06_extend-archetypes.md
index 2777865..df98143 100644
--- a/documentation/docs/06_extend-archetypes.md
+++ b/documentation/docs/06_extend-archetypes.md
@@ -4,4 +4,140 @@ title: Maven Archetypes
sidebar_label: Maven Archetypes
---
-tbd
\ No newline at end of file
+In this tutorial we explain how you can use the Maven archetypes to develop
your own StreamPipes processors and sinks.
+We use IntelliJ in this tutorial, but it works with any IDE of your choice.
+
+## Prerequisites
+You need to have Maven installed, further you need an up and running
StreamPipes installation on your development computer.
+To ease the configuration of environment variables, we use the IntelliJ [env
Plugin](https://plugins.jetbrains.com/plugin/7861-envfile).
+Install this in IntelliJ. The development also works without the plugin, then
you have to set the environment variables manually instead of using the env
configuration file.
+
+## Create Project
+To create a new project, we provide multiple Maven Archteypes.
+Currently, we have archetypes for the JVM and Flink wrappers, each for
processors and sinks.
+The commands required to create a new pipeline element project can be found
below. Make sure that you select a version compatible with your StreamPipes
installation.
+Copy the command into your terminal to create a new project.
+The project will be created in the current folder.
+First, the ``groupId`` of the resulting Maven artifact must be set.
+We use ``groupId``: ``org.example`` and ``artifactId``: ``ExampleProcessor``.
+You can keep the default values for the other settings, confirm them by
hitting enter.
+Now, a new folder with the name ``ExampleProcessor`` is generated.
+
+The current {sp.version} is 0.66.0
+
+```bash
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.streampipes
\
+ -DarchetypeArtifactId=streampipes-archetype-pe-processors-jvm \
+ -DarchetypeVersion={sp.version}
+```
+<details class="info">
+ <summary>Select: [Processors / Sinks] [JVM / Flink]</summary>
+
+## Processors JVM
+```bash
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.streampipes
\
+ -DarchetypeArtifactId=streampipes-archetype-pe-processors-jvm \
+ -DarchetypeVersion={sp.version}
+```
+
+## Processors Flink
+```bash
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.streampipes
\
+ -DarchetypeArtifactId=streampipes-archetype-pe-processors-flink \
+ -DarchetypeVersion={sp.version}
+```
+
+## Sinks JVM
+```bash
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.streampipes
\
+ -DarchetypeArtifactId=streampipes-archetype-pe-sinks-jvm \
+ -DarchetypeVersion={sp.version}
+```
+
+## Sinks Flink
+```bash
+mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.streampipes
\
+ -DarchetypeArtifactId=streampipes-archetype-pe-sinks-flink \
+ -DarchetypeVersion={sp.version}
+```
+</details>
+
+
+## Edit Processor
+Open the project in your IDE.
+If everything worked, the structure should look similar to the following image.
+The *config* package contains all the configuration parameters of your
processors / sinks.
+In the *main* package, it is defined which processors / sinks you want to
activate and the *pe.processor.example* package contains three classes with the
application logic.
+For details, have a look at the other parts of the Developer Guide, where
these classes are explained in more depth.
+
+<img src="/docs/img/archetype/project_structure.png" width="30%" alt="Project
Structure">
+
+Open the class *Example* and edit the ``onEvent`` method to print the incoming
event, log it to the console and send it to the next component without changing
it.
+
+```java
+@Override
+public void onEvent(Event event, SpOutputCollector collector) {
+ // Print the incoming event on the console
+ System.out.println(event);
+
+ // Hand the incoming event to the output collector without changing it.
+ collector.collect(event);
+}
+```
+
+## Start Processor
+Before the processor can be started, you need to edit the *env* file in the
*development* folder.
+Replace all local hosts in this file with the IP address or DNS name of your
computer.
+This is relevant to make the mapping between the services running in Docker
and your component running in the local IDE.
+After all updates are updated, this file is used by the envfile plugin to
provide configuration parameters to the pipeline element.
+Alternatively, environment variables can also be set on your host or IDE.
+Now start the project by clicking on **(Run -> Edit Configuration)**.
+Add a new configuration in the Configuration menu by clicking on the + sign
and select **Application**.
+Name the configuration *ExampleProcessor* and select the *Init* class as the
'Main class'.
+Then set *ExampleProcessor* in 'Use classpath of module'.
+
+
+As the last step, switch to the tab *EnvFile* and load the env file.
+Click on 'Enable EnvFile' to activate it and add the just edited env file by
clicking on the + sign.
+Save all the changes by clicking *Apply*.
+Now you can start the processor.
+
+<div class="my-carousel">
+ <img src="/docs/img/archetype/run_configuration.png" alt="Configuration
View">
+ <img src="/docs/img/archetype/run_env_configuration.png" alt="Environment
Configuration View">
+</div>
+
+To check if the service is up and running, open the browser on
*'localhost:6666'*. The machine-readable description of the processor should be
visible as shown below.
+
+<img src="/docs/img/archetype/endpoint.png" width="90%" alt="Project
Structure">
+
+
+<div class="admonition error">
+<div class="admonition-title">Common Problems</div>
+<p>
+If the service description is not shown on 'localhost:6666', you might have to
change the port address.
+This needs to be done in the configuration of your service, further explained
in the configurations part of the developer guide.
+
+If the service does not show up in the StreamPipes installation menu, click on
'MANAGE ENDPOINTS' and add 'http://<span></span>YOUR_IP_OR_DNS_NAME:6666'.
+Use the IP or DNS name you provided in the env file.
+After adding the endpoint, a new processor with the name *Example* should show
up.
+</p>
+</div>
+
+Now you can go to StreamPipes.
+Your new processor *'Example'* should now show up in the installation menu.
+Install it, then switch to the pipeline view and create a simple pipeline that
makes use of your newly created processor.
+In case you opened the StreamPipes installation for the first time, it should
have been automatically installed during the setup process.
+
+<img src="/docs/img/archetype/example_pipeline.png" width="80%" alt="Project
Structure">
+
+Start this pipeline.
+Now you should see logging messages in your console and, once you've created a
visualization, you can also see the resulting events of your component in
StreamPipes.
+
+Congratulations, you have just created your first processor!
+From here on you can start experimenting and implement your own algorithms.
diff --git a/documentation/docs/dev-guide-event-model.md
b/documentation/docs/06_extend-sdk-event-model.md
similarity index 98%
copy from documentation/docs/dev-guide-event-model.md
copy to documentation/docs/06_extend-sdk-event-model.md
index 1134bdd..8d85666 100644
--- a/documentation/docs/dev-guide-event-model.md
+++ b/documentation/docs/06_extend-sdk-event-model.md
@@ -1,7 +1,7 @@
---
-id: dev-guide-event-model
+id: extend-sdk-event-model
title: SDK Guide: Event Model
-sidebar_label: Event Model
+sidebar_label: SDK: Event Model
---
## Introduction
diff --git a/documentation/docs/dev-guide-output-strategies.md
b/documentation/docs/06_extend-sdk-output-strategies.md
similarity index 99%
copy from documentation/docs/dev-guide-output-strategies.md
copy to documentation/docs/06_extend-sdk-output-strategies.md
index ac17f6a..abb5db7 100644
--- a/documentation/docs/dev-guide-output-strategies.md
+++ b/documentation/docs/06_extend-sdk-output-strategies.md
@@ -1,7 +1,7 @@
---
-id: dev-guide-output-strategies
+id: extend-sdk-output-strategies
title: SDK Guide: Output Strategies
-sidebar_label: Output Strategies
+sidebar_label: SDK: Output Strategies
---
## Introduction
diff --git a/documentation/docs/dev-guide-static-properties.md
b/documentation/docs/06_extend-sdk-static-properties.md
similarity index 99%
copy from documentation/docs/dev-guide-static-properties.md
copy to documentation/docs/06_extend-sdk-static-properties.md
index de2e0c3..afbdfc4 100644
--- a/documentation/docs/dev-guide-static-properties.md
+++ b/documentation/docs/06_extend-sdk-static-properties.md
@@ -1,7 +1,7 @@
---
-id: dev-guide-static-properties
+id: extend-sdk-static-properties
title: SDK Guide: Static Properties
-sidebar_label: Static Properties
+sidebar_label: SDK: Static Properties
---
## Introduction
diff --git a/documentation/docs/dev-guide-stream-requirements.md
b/documentation/docs/06_extend-sdk-stream-requirements.md
similarity index 99%
copy from documentation/docs/dev-guide-stream-requirements.md
copy to documentation/docs/06_extend-sdk-stream-requirements.md
index 4fa7949..002d27c 100644
--- a/documentation/docs/dev-guide-stream-requirements.md
+++ b/documentation/docs/06_extend-sdk-stream-requirements.md
@@ -1,7 +1,7 @@
---
-id: dev-guide-stream-requirements
+id: extend-sdk-stream-requirements
title: SDK Guide: Stream Requirements
-sidebar_label: Stream Requirements
+sidebar_label: SDK: Stream Requirements
---
## Introduction
diff --git a/documentation/docs/06_extend-setup.md
b/documentation/docs/06_extend-setup.md
index c0a4d07..eabaf81 100644
--- a/documentation/docs/06_extend-setup.md
+++ b/documentation/docs/06_extend-setup.md
@@ -4,4 +4,109 @@ title: Development Setup
sidebar_label: Development Setup
---
-tbd
\ No newline at end of file
+In this section, we describe our recommended minimum setup for locally setting
up a development instance of StreamPipes needed to develop, run and test new
pipeline elements.
+
+## IDE & required dev tools
+StreamPipes does not have specific requirements on the IDE - so feel free to
choose the IDE of your choice.
+The only requirements in terms of development tools are that you have Java 8
and Maven installed.
+
+## Docker-based local StreamPipes instance
+In order to quickly test developed pipeline elements without needing to
install all services required by StreamPipes, we provide a CLI tool that allows
you to selectively start StreamPipes components.
+The CLI tool allows to switch to several templates (based on docker-compose)
depending on the role. For instance, if you are developing a pipeline element,
use the template ``pe-developer``. This will start backend and ui components in
a Docker container, while you can easily test your pipeline element in your IDE.
+
+For now, we refer to the Github Readme for instructions on how to use the CLI
tool:
[https://github.com/apache/incubator-streampipes-installer](https://github.com/apache/incubator-streampipes-installer)
+
+## Starter projects
+
+Now, once you've started the development instance, you are ready to develop
your very first pipeline element.
+Instead of starting from scratch, we recommend using our provided maven
archetypes:
+
+### Maven archetypes
+
+Create the Maven archetype as described in the [Getting
Started](/docs/dev-guide-archetype) guide.
+
+### Starting from scratch
+
+In order to develop a new pipeline element from scratch, you need to create a
new Maven project and import the following dependencies:
+
+<details class="info">
+<summary>pom.xml</summary>
+```
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-container-standalone</artifactId>
+ <version>0.64.0</version>
+</dependency>
+
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-sdk</artifactId>
+ <version>0.64.0</version>
+</dependency>
+
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-vocabulary</artifactId>
+ <version>0.64.0</version>
+</dependency>
+
+<!-- This dependency needs to be imported if you plan to develop a new data
processor or data sink using the Apache Flink wrapper -->
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-wrapper-flink</artifactId>
+ <version>0.64.0</version>
+</dependency>
+
+<!-- This dependency needs to be imported if you plan to develop a new data
processor or data sink which is running directly on the JVM -->
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-wrapper-standalone</artifactId>
+ <version>0.64.0</version>
+</dependency>
+
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-dataformat-json</artifactId>
+ <version>0.64.0</version>
+</dependency>
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-dataformat-smile</artifactId>
+ <version>0.64.0</version>
+</dependency>
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-dataformat-cbor</artifactId>
+ <version>0.64.0</version>
+</dependency>
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-dataformat-fst</artifactId>
+ <version>0.64.0</version>
+</dependency>
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-messaging-jms</artifactId>
+ <version>0.64.0</version>
+</dependency>
+<dependency>
+ <groupId>org.streampipes</groupId>
+ <artifactId>streampipes-messaging-kafka</artifactId>
+ <version>0.64.0</version>
+</dependency>
+```
+</details>
+
+The following three dependencies are mandatory:
+
+* `streampipes-container-standalone`, defines that we are going to create a
new pipeline element where the description will be accessible through an
embedded web server.
+* `streampipes-sdk` imports the SDK which provides many convencience functions
to create new pipeline elements.
+* `streampipes-vocabulary` imports various RDF vocabularies which are used by
the SDK to auto-generate the semantic description of pipeline elements.
+
+The following three dependencies might be optional depending on the pipeline
element type you plan to create:
+
+* `streampipes-wrapper-flink` should be used in case you plan to connect a
new data processor or data sink that uses Apache Flink for processing events at
runtime.
+* `streampipes-wrapper-standalone` should be used in case you plan to connect
a new data processor or data sink that does not use an external processing
engine. Events are directly processed in a single-host fashion.
+
+
+Finally, this dependency will provide abstract classes to define data sources
and streams.
diff --git a/documentation/docs/dev-guide-tutorial-processors.md
b/documentation/docs/06_extend-tutorial-data-processors.md
similarity index 99%
copy from documentation/docs/dev-guide-tutorial-processors.md
copy to documentation/docs/06_extend-tutorial-data-processors.md
index e346368..1512643 100644
--- a/documentation/docs/dev-guide-tutorial-processors.md
+++ b/documentation/docs/06_extend-tutorial-data-processors.md
@@ -1,5 +1,5 @@
---
-id: dev-guide-tutorial-processors
+id: extend-tutorial-data-processors
title: Tutorial: Data Processors
sidebar_label: Tutorial: Data Processors
---
@@ -496,4 +496,4 @@ To install the just created element, open the StreamPipes
UI and follow the manu
Congratulations! You've just created your first data processor for StreamPipes.
There are many more things to explore and data processors can be defined in
much more detail using multiple wrappers.
-Follow our [SDK guide](dev-guide-sdk-guide-processors) to see what's possible!
\ No newline at end of file
+Follow our [SDK guide](dev-guide-sdk-guide-processors) to see what's possible!
diff --git a/documentation/docs/dev-guide-tutorial-sinks.md
b/documentation/docs/06_extend-tutorial-data-sinks.md
similarity index 99%
copy from documentation/docs/dev-guide-tutorial-sinks.md
copy to documentation/docs/06_extend-tutorial-data-sinks.md
index e4cf3b5..cc9a85c 100644
--- a/documentation/docs/dev-guide-tutorial-sinks.md
+++ b/documentation/docs/06_extend-tutorial-data-sinks.md
@@ -1,5 +1,5 @@
---
-id: dev-guide-tutorial-sinks
+id: extend-tutorial-data-sinks
title: Tutorial: Data Sinks
sidebar_label: Tutorial: Data Sinks
---
@@ -243,4 +243,4 @@ To install the just created element, open the StreamPipes
UI and follow the manu
Congratulations! You've just created your first data sink for StreamPipes.
There are many more things to explore and data sinks can be defined in much
more detail using multiple wrappers.
-Follow our [SDK guide](dev-guide-sdk-guide-sinks) to see what's possible!
\ No newline at end of file
+Follow our [SDK guide](dev-guide-sdk-guide-sinks) to see what's possible!
diff --git a/documentation/docs/dev-guide-tutorial-sources.md
b/documentation/docs/06_extend-tutorial-data-sources.md
similarity index 97%
copy from documentation/docs/dev-guide-tutorial-sources.md
copy to documentation/docs/06_extend-tutorial-data-sources.md
index 27a3081..41d965c 100644
--- a/documentation/docs/dev-guide-tutorial-sources.md
+++ b/documentation/docs/06_extend-tutorial-data-sources.md
@@ -1,5 +1,5 @@
---
-id: dev-guide-tutorial-sources
+id: extend-tutorial-data-sources
title: Tutorial: Data Sources
sidebar_label: Tutorial: Data Sources
---
@@ -116,9 +116,9 @@ These four _event properties_ compose our _event schema_.
An event property must
* **Runtime Name**. The runtime name indicates the key of the property at
runtime, e.g., if our JSON message contains a structure such as `{"plateNumber"
: "KA-F 123"}`, the runtime name must be `plateNumber`.
* **Runtime Type**. An event property must have a primitive type (we will
later see how to model more complex properties such as lists and nested
properties).
-The type must be an instance of `XMLSchema` primitives, however, the SDK
provides convenience methods to provide the property type.
+ The type must be an instance of `XMLSchema` primitives, however, the SDK
provides convenience methods to provide the property type.
* **Domain Property**. The domain property indicates the semantics of the
event property. For instance, the `latitude` property is linked to the
`http://www.w3.org/2003/01/geo/wgs84_pos#lat` property of the WGS84 vocabulary.
-The domain property should be an URI as part of an existing or domain-specific
vocabulary. The SDK provides convenience methods for popuplar vocabularies
(e.g., Schema.org, Dolce or WGS84).
+ The domain property should be an URI as part of an existing or
domain-specific vocabulary. The SDK provides convenience methods for popuplar
vocabularies (e.g., Schema.org, Dolce or WGS84).
In order to complete the minimum required specification of an event stream, we
need to provide information on the transport format and protocol of the data
stream at runtime.
@@ -279,4 +279,4 @@ To install the just created element, open the StreamPipes
UI and follow the manu
Congratulations! You've just created your first pipeline element for
StreamPipes.
There are many more things to explore and data sources can be defined in much
more detail.
-Follow our [SDK guide](dev-guide-source-sdk) to see what's possible!
\ No newline at end of file
+Follow our [SDK guide](dev-guide-source-sdk) to see what's possible!
diff --git a/documentation/docs/07_technicals-architecture.md
b/documentation/docs/07_technicals-architecture.md
index 0022841..16963b8 100644
--- a/documentation/docs/07_technicals-architecture.md
+++ b/documentation/docs/07_technicals-architecture.md
@@ -4,4 +4,59 @@ title: Architecture
sidebar_label: Architecture
---
-tbd
\ No newline at end of file
+
+The following picture illustrates the high-level architecture of StreamPipes:
+
+<img src="/docs/img/architecture/high-level-architecture.png" alt="High Level
Architecture of StreamPipes">
+
+Users mainly interact (besides other UI components) with the _Pipeline Editor_
to create stream processing pipelines based on data streams, data processors
and data sinks.
+These reusable pipeline elements are provided by self-contained _pipeline
element containers_, each of them having a semantic description that specifies
their characteristics (e.g., input, output and required user input for data
processors).
+Each pipeline element container has a REST endpoint that provides these
characteristics as a JSON-LD document.
+
+Pipeline element containers are built using one of several provided _wrappers_.
+Wrappers abstract from the underlying runtime stream processing framework.
+Currently, the StreamPipes framework provides wrappers for Apache Flink, Esper
and algorithms running directly on the JVM.
+
+The _pipeline manager_ manages the definition and execution of pipelines.
+When creating pipelines, the manager continuously matches the pipeline against
its semantic description and provides user guidance in form of recommendations.
+Once a pipeline is started, the pipeline manager invokes the corresponding
pipeline element containers.
+The container prepares the actual execution logic and submits the program to
the underlying execution engine, e.g., the program is deployed in the Apache
Flink cluster.
+
+Pipeline elements exchange data using one or more message brokers and
protocols (e.g., Kafka or MQTT).
+StreamPipes does not rely on a specific broker or message format, but
negotiates suitable brokers based on the capabilities of connected pipeline
elements.
+
+Thus, StreamPipes provides a higher-level abstraction of existing stream
processing technology by leveraging domain experts to create streaming
analytics pipelines in a self-service manner.
+
+## Semantic description
+Pipeline elements in StreamPipes are meant to be reusable:
+
+* Data processors and data sink are generic (or domain-specific) elements that
express their requirements and are able to operate on any stream that satisfies
these requirements.
+* Data processors and data sinks can be manually configured by offering
possible configuration parameters which users can individually define when
creating pipelines.
+* Data streams can be connected to any data processor or data sink that
matches the capabilities of the stream.
+
+When users create pipelines by connecting a data stream with a data processor
(or further processors), the pipeline manager _matches_ the input stream of a
data processor against its requirements.
+This matching is performed based on the _semantic description of each element.
+The semantic description (technically an RDF graph serialized as JSON-LD) can
be best understood by seeing it as an envelope around a pipeline element.
+It only provides metadata information, while we don't rely on any RDF at
runtime for exchanging events between pipeline elements.
+While RDF-based metadata ensures good understanding of stream capabilities,
lightweight event formats at runtime (such as JSON or Thrift) ensure fast
processing of events.
+
+Let's look at an example stream that produces a continuous stream of vehicle
positions as illustrated below:
+
+<img src="/docs/img/architecture/semantic-description-stream.png"
alt="Semantic description of data streams">
+
+While the runtime layer produces plain JSON by submitting actual values of the
position and the vehicle's plate number, the description layer describes
various characteristics of the stream:
+For instance, it defines the event schema (including, besides the data type
and the runtime name of each property also a more fine-grained meaning of the
property), quality aspects (e.g., the measurement unit of a property or the
frequency) and the grounding (e.g., the format used at runtime and the
communication protocol used for transmitting events).
+
+The same accounts for data processors and data sinks:
+
+<img src="/docs/img/architecture/semantic-description-processor.png"
alt="Semantic description of data processor">
+
+Data processors (and, with some differences, data sinks) are annotated by
providing metadata information on their required input and output.
+For instance, we can define minimum schema requirements (such as geospatial
coordinates that need to be provided by any stream that is connected to a
processor), but also required (minimum or maximum) quality levels and supported
transport protocols and formats.
+In addition, required configuration parameters users can define during the
pipeline definition process are provided by the semantic description.
+
+Once new pipeline elements are imported into StreamPipes, we store all
information provided by the description layer in a central repository and use
this information to guide useres through the pipeline definition process.
+
+Don't worry - you will never be required to model RDF by yourself.
+Our SDK provides convenience methods that help creating the description
automatically.
+
diff --git a/documentation/docs/dev-guide-configuration.md
b/documentation/docs/07_technicals-configuration.md
similarity index 99%
copy from documentation/docs/dev-guide-configuration.md
copy to documentation/docs/07_technicals-configuration.md
index 1b558de..4e574cc 100644
--- a/documentation/docs/dev-guide-configuration.md
+++ b/documentation/docs/07_technicals-configuration.md
@@ -1,5 +1,5 @@
---
-id: dev-guide-configuration
+id: technicals-configuration
title: Configuration
sidebar_label: Configuration
---
diff --git a/documentation/docs/dev-guide-archetype.md
b/documentation/docs/dev-guide-archetype.md
index 408af78..75eef47 100644
--- a/documentation/docs/dev-guide-archetype.md
+++ b/documentation/docs/dev-guide-archetype.md
@@ -4,140 +4,3 @@ title: Start Developing
sidebar_label: Start Developing
---
-In this tutorial we explain how you can use the Maven archetypes to develop
your own StreamPipes processors and sinks.
-We use IntelliJ in this tutorial, but it works with any IDE of your choice.
-
-## Prerequisites
-You need to have Maven installed, further you need an up and running
StreamPipes installation on your development computer.
-To ease the configuration of environment variables, we use the IntelliJ [env
Plugin](https://plugins.jetbrains.com/plugin/7861-envfile).
-Install this in IntelliJ. The development also works without the plugin, then
you have to set the environment variables manually instead of using the env
configuration file.
-
-## Create Project
-To create a new project, we provide multiple Maven Archteypes.
-Currently, we have archetypes for the JVM and Flink wrappers, each for
processors and sinks.
-The commands required to create a new pipeline element project can be found
below. Make sure that you select a version compatible with your StreamPipes
installation.
-Copy the command into your terminal to create a new project.
-The project will be created in the current folder.
-First, the ``groupId`` of the resulting Maven artifact must be set.
-We use ``groupId``: ``org.example`` and ``artifactId``: ``ExampleProcessor``.
-You can keep the default values for the other settings, confirm them by
hitting enter.
-Now, a new folder with the name ``ExampleProcessor`` is generated.
-
-The current {sp.version} is 0.66.0
-
-```bash
-mvn archetype:generate \
- -DarchetypeGroupId=org.apache.streampipes
\
- -DarchetypeArtifactId=streampipes-archetype-pe-processors-jvm \
- -DarchetypeVersion={sp.version}
-```
-<details class="info">
- <summary>Select: [Processors / Sinks] [JVM / Flink]</summary>
-
-## Processors JVM
-```bash
-mvn archetype:generate \
- -DarchetypeGroupId=org.apache.streampipes
\
- -DarchetypeArtifactId=streampipes-archetype-pe-processors-jvm \
- -DarchetypeVersion={sp.version}
-```
-
-## Processors Flink
-```bash
-mvn archetype:generate \
- -DarchetypeGroupId=org.apache.streampipes
\
- -DarchetypeArtifactId=streampipes-archetype-pe-processors-flink \
- -DarchetypeVersion={sp.version}
-```
-
-## Sinks JVM
-```bash
-mvn archetype:generate \
- -DarchetypeGroupId=org.apache.streampipes
\
- -DarchetypeArtifactId=streampipes-archetype-pe-sinks-jvm \
- -DarchetypeVersion={sp.version}
-```
-
-## Sinks Flink
-```bash
-mvn archetype:generate \
- -DarchetypeGroupId=org.apache.streampipes
\
- -DarchetypeArtifactId=streampipes-archetype-pe-sinks-flink \
- -DarchetypeVersion={sp.version}
-```
-</details>
-
-
-## Edit Processor
-Open the project in your IDE.
-If everything worked, the structure should look similar to the following image.
-The *config* package contains all the configuration parameters of your
processors / sinks.
-In the *main* package, it is defined which processors / sinks you want to
activate and the *pe.processor.example* package contains three classes with the
application logic.
-For details, have a look at the other parts of the Developer Guide, where
these classes are explained in more depth.
-
-<img src="/docs/img/archetype/project_structure.png" width="30%" alt="Project
Structure">
-
-Open the class *Example* and edit the ``onEvent`` method to print the incoming
event, log it to the console and send it to the next component without changing
it.
-
-```java
-@Override
-public void onEvent(Event event, SpOutputCollector collector) {
- // Print the incoming event on the console
- System.out.println(event);
-
- // Hand the incoming event to the output collector without changing it.
- collector.collect(event);
-}
-```
-
-## Start Processor
-Before the processor can be started, you need to edit the *env* file in the
*development* folder.
-Replace all local hosts in this file with the IP address or DNS name of your
computer.
-This is relevant to make the mapping between the services running in Docker
and your component running in the local IDE.
-After all updates are updated, this file is used by the envfile plugin to
provide configuration parameters to the pipeline element.
-Alternatively, environment variables can also be set on your host or IDE.
-Now start the project by clicking on **(Run -> Edit Configuration)**.
-Add a new configuration in the Configuration menu by clicking on the + sign
and select **Application**.
-Name the configuration *ExampleProcessor* and select the *Init* class as the
'Main class'.
-Then set *ExampleProcessor* in 'Use classpath of module'.
-
-
-As the last step, switch to the tab *EnvFile* and load the env file.
-Click on 'Enable EnvFile' to activate it and add the just edited env file by
clicking on the + sign.
-Save all the changes by clicking *Apply*.
-Now you can start the processor.
-
-<div class="my-carousel">
- <img src="/docs/img/archetype/run_configuration.png" alt="Configuration
View">
- <img src="/docs/img/archetype/run_env_configuration.png" alt="Environment
Configuration View">
-</div>
-
-To check if the service is up and running, open the browser on
*'localhost:6666'*. The machine-readable description of the processor should be
visible as shown below.
-
-<img src="/docs/img/archetype/endpoint.png" width="90%" alt="Project
Structure">
-
-
-<div class="admonition error">
-<div class="admonition-title">Common Problems</div>
-<p>
-If the service description is not shown on 'localhost:6666', you might have to
change the port address.
-This needs to be done in the configuration of your service, further explained
in the configurations part of the developer guide.
-
-If the service does not show up in the StreamPipes installation menu, click on
'MANAGE ENDPOINTS' and add 'http://<span></span>YOUR_IP_OR_DNS_NAME:6666'.
-Use the IP or DNS name you provided in the env file.
-After adding the endpoint, a new processor with the name *Example* should show
up.
-</p>
-</div>
-
-Now you can go to StreamPipes.
-Your new processor *'Example'* should now show up in the installation menu.
-Install it, then switch to the pipeline view and create a simple pipeline that
makes use of your newly created processor.
-In case you opened the StreamPipes installation for the first time, it should
have been automatically installed during the setup process.
-
-<img src="/docs/img/archetype/example_pipeline.png" width="80%" alt="Project
Structure">
-
-Start this pipeline.
-Now you should see logging messages in your console and, once you've created a
visualization, you can also see the resulting events of your component in
StreamPipes.
-
-Congratulations, you have just created your first processor!
-From here on you can start experimenting and implement your own algorithms.
diff --git a/documentation/docs/dev-guide-architecture.md
b/documentation/docs/dev-guide-architecture.md
index 39490d9..d85caad 100644
--- a/documentation/docs/dev-guide-architecture.md
+++ b/documentation/docs/dev-guide-architecture.md
@@ -3,59 +3,3 @@ id: dev-guide-architecture
title: Architecture
sidebar_label: Architecture
---
-
-The following picture illustrates the high-level architecture of StreamPipes:
-
-<img src="/docs/img/architecture/high-level-architecture.png" alt="High Level
Architecture of StreamPipes">
-
-Users mainly interact (besides other UI components) with the _Pipeline Editor_
to create stream processing pipelines based on data streams, data processors
and data sinks.
-These reusable pipeline elements are provided by self-contained _pipeline
element containers_, each of them having a semantic description that specifies
their characteristics (e.g., input, output and required user input for data
processors).
-Each pipeline element container has a REST endpoint that provides these
characteristics as a JSON-LD document.
-
-Pipeline element containers are built using one of several provided _wrappers_.
-Wrappers abstract from the underlying runtime stream processing framework.
-Currently, the StreamPipes framework provides wrappers for Apache Flink, Esper
and algorithms running directly on the JVM.
-
-The _pipeline manager_ manages the definition and execution of pipelines.
-When creating pipelines, the manager continuously matches the pipeline against
its semantic description and provides user guidance in form of recommendations.
-Once a pipeline is started, the pipeline manager invokes the corresponding
pipeline element containers.
-The container prepares the actual execution logic and submits the program to
the underlying execution engine, e.g., the program is deployed in the Apache
Flink cluster.
-
-Pipeline elements exchange data using one or more message brokers and
protocols (e.g., Kafka or MQTT).
-StreamPipes does not rely on a specific broker or message format, but
negotiates suitable brokers based on the capabilities of connected pipeline
elements.
-
-Thus, StreamPipes provides a higher-level abstraction of existing stream
processing technology by leveraging domain experts to create streaming
analytics pipelines in a self-service manner.
-
-## Semantic description
-Pipeline elements in StreamPipes are meant to be reusable:
-
-* Data processors and data sink are generic (or domain-specific) elements that
express their requirements and are able to operate on any stream that satisfies
these requirements.
-* Data processors and data sinks can be manually configured by offering
possible configuration parameters which users can individually define when
creating pipelines.
-* Data streams can be connected to any data processor or data sink that
matches the capabilities of the stream.
-
-When users create pipelines by connecting a data stream with a data processor
(or further processors), the pipeline manager _matches_ the input stream of a
data processor against its requirements.
-This matching is performed based on the _semantic description of each element.
-The semantic description (technically an RDF graph serialized as JSON-LD) can
be best understood by seeing it as an envelope around a pipeline element.
-It only provides metadata information, while we don't rely on any RDF at
runtime for exchanging events between pipeline elements.
-While RDF-based metadata ensures good understanding of stream capabilities,
lightweight event formats at runtime (such as JSON or Thrift) ensure fast
processing of events.
-
-Let's look at an example stream that produces a continuous stream of vehicle
positions as illustrated below:
-
-<img src="/docs/img/architecture/semantic-description-stream.png"
alt="Semantic description of data streams">
-
-While the runtime layer produces plain JSON by submitting actual values of the
position and the vehicle's plate number, the description layer describes
various characteristics of the stream:
-For instance, it defines the event schema (including, besides the data type
and the runtime name of each property also a more fine-grained meaning of the
property), quality aspects (e.g., the measurement unit of a property or the
frequency) and the grounding (e.g., the format used at runtime and the
communication protocol used for transmitting events).
-
-The same accounts for data processors and data sinks:
-
-<img src="/docs/img/architecture/semantic-description-processor.png"
alt="Semantic description of data processor">
-
-Data processors (and, with some differences, data sinks) are annotated by
providing metadata information on their required input and output.
-For instance, we can define minimum schema requirements (such as geospatial
coordinates that need to be provided by any stream that is connected to a
processor), but also required (minimum or maximum) quality levels and supported
transport protocols and formats.
-In addition, required configuration parameters users can define during the
pipeline definition process are provided by the semantic description.
-
-Once new pipeline elements are imported into StreamPipes, we store all
information provided by the description layer in a central repository and use
this information to guide useres through the pipeline definition process.
-
-Don't worry - you will never be required to model RDF by yourself.
-Our SDK provides convenience methods that help creating the description
automatically.
-
diff --git a/documentation/docs/dev-guide-configuration.md
b/documentation/docs/dev-guide-configuration.md
index 1b558de..b1b7e37 100644
--- a/documentation/docs/dev-guide-configuration.md
+++ b/documentation/docs/dev-guide-configuration.md
@@ -3,56 +3,3 @@ id: dev-guide-configuration
title: Configuration
sidebar_label: Configuration
---
-
-On this page we explain how the StreamPipes configuration works.
-StreamPipes allows the individual services (pipeline element containers and
third-party services) to store configuration parameters in a distributed
key-value store.
-This has the advantage that individual services do not need to store any
configurations on the local file system, enabling us to run containers anywhere.
-As a key-value store we use [Consul](https://www.consul.io/), which is an
essential service for all our services.
-
-<img src="/docs/img/configuration/consul.png" width="50%" alt="Semantic
description of data processor">
-
-
-## Edit Configurations
-All services in StreamPipes can have configuration parameters.
-You can either change them in the consul user interface (which is by default
running on port 8500) or directly in the StreamPipes Configurations Page.
-Once a new pipeline element container is started, it is registered in Consul
and the parameters can be edited in the configuration page, as shown below.
-To store changes in Consul, the update button must be clicked.
-
-<div class="my-carousel">
- <img src="/docs/img/configuration/configuration_1.png" alt="Configuration
View">
-</div>
-
-## Configuration for Developers
-We provide a Configurations API for the use of configuration parameters in
your services.
-Each processing element project has a “config” package
[[Example]](https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-sinks-internal-jvm/src/main/java/org/streampipes/sinks/internal/jvm/config).
-This package usually contains two classes.
-One containing unique keys for the configuration values and one containing the
getter and setter methods to access these values.
-For the naming of configuration keys, we recommend to use “SP” as a prefix.
-As we explain later, it is possible to set default configurations as
environment variables, this prefix makes them unique on your server.
-A configuration entry needs a unique config key. For this key, a value can be
specified containing the configuration, like for example the port number of the
service.
-For each configuration, a description explaining the parameter can be
provided, further the data type must be specified and whether it is a password
or not.
-Below, the schema of a configuration item is shown on the left and an example
of a port configuration on the right.
-
-<img src="/docs/img/configuration/config_key.png" width="80%" alt="Semantic
description of data processor">
-
-As a developer, you can add as many new configurations to services as you
wish, but there are some that are required for all processing element
containers.
-Those are **the host**, **the port**, and **the name** of the service.
-
-## Default Values
-You can provide default values for the configurations, which are used when a
configuration is read for the first time.
-The first option is to register a configuration parameter in the Config class.
-This is a fallback value, which is used if nothing else is defined.
-Since this value is static, we offer a second option.
-It is possible to provide a default value by setting an environment variable.
-In this case, the convention is that the key of a configuration parameter must
be used as the environment variable.
-Now, this value is used instead of the value defined in the Config class.
-During development, the configuration values often need to be changed for
debugging purposes, therefore we provide an .env file in all processing element
projects and archetypes.
-This file can be used by your IDE to set the environment variables. (e.g.,
[Intellij Plugin](https://plugins.jetbrains.com/plugin/7861-envfile))
-When you need to change the variable at runtime, you can do this in the
StreamPipes configurations as explained before.
-Those changes take effect immediately without the need of a container restart.
-
-<div class="admonition warning">
-<div class="admonition-title">Installed pipeline elements</div>
-<p>Be cautious, when the configuration is used in the semantic description of
a processing element which is already installed in StreamPipes, you have to
reload this element in StreamPipes (my elements -> reload).
- In addition, changes might affect already running pipelines.</p>
-</div>
diff --git a/documentation/docs/dev-guide-environment.md
b/documentation/docs/dev-guide-environment.md
index 5f2c9d6..ac4a9f2 100644
--- a/documentation/docs/dev-guide-environment.md
+++ b/documentation/docs/dev-guide-environment.md
@@ -3,110 +3,3 @@ id: dev-guide-development-environment
title: Development Environment
sidebar_label: Development Environment
---
-
-In this section, we describe our recommended minimum setup for locally setting
up a development instance of StreamPipes needed to develop, run and test new
pipeline elements.
-
-## IDE & required dev tools
-StreamPipes does not have specific requirements on the IDE - so feel free to
choose the IDE of your choice.
-The only requirements in terms of development tools are that you have Java 8
and Maven installed.
-
-## Docker-based local StreamPipes instance
-In order to quickly test developed pipeline elements without needing to
install all services required by StreamPipes, we provide a CLI tool that allows
you to selectively start StreamPipes components.
-The CLI tool allows to switch to several templates (based on docker-compose)
depending on the role. For instance, if you are developing a pipeline element,
use the template ``pe-developer``. This will start backend and ui components in
a Docker container, while you can easily test your pipeline element in your IDE.
-
-For now, we refer to the Github Readme for instructions on how to use the CLI
tool:
[https://github.com/apache/incubator-streampipes-installer](https://github.com/apache/incubator-streampipes-installer)
-
-## Starter projects
-
-Now, once you've started the development instance, you are ready to develop
your very first pipeline element.
-Instead of starting from scratch, we recommend using our provided maven
archetypes:
-
-### Maven archetypes
-
-Create the Maven archetype as described in the [Getting
Started](/docs/dev-guide-archetype) guide.
-
-### Starting from scratch
-
-In order to develop a new pipeline element from scratch, you need to create a
new Maven project and import the following dependencies:
-
-<details class="info">
-<summary>pom.xml</summary>
-```
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-container-standalone</artifactId>
- <version>0.64.0</version>
-</dependency>
-
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-sdk</artifactId>
- <version>0.64.0</version>
-</dependency>
-
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-vocabulary</artifactId>
- <version>0.64.0</version>
-</dependency>
-
-<!-- This dependency needs to be imported if you plan to develop a new data
processor or data sink using the Apache Flink wrapper -->
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-wrapper-flink</artifactId>
- <version>0.64.0</version>
-</dependency>
-
-<!-- This dependency needs to be imported if you plan to develop a new data
processor or data sink which is running directly on the JVM -->
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-wrapper-standalone</artifactId>
- <version>0.64.0</version>
-</dependency>
-
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-dataformat-json</artifactId>
- <version>0.64.0</version>
-</dependency>
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-dataformat-smile</artifactId>
- <version>0.64.0</version>
-</dependency>
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-dataformat-cbor</artifactId>
- <version>0.64.0</version>
-</dependency>
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-dataformat-fst</artifactId>
- <version>0.64.0</version>
-</dependency>
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-messaging-jms</artifactId>
- <version>0.64.0</version>
-</dependency>
-<dependency>
- <groupId>org.streampipes</groupId>
- <artifactId>streampipes-messaging-kafka</artifactId>
- <version>0.64.0</version>
-</dependency>
-```
-</details>
-
-The following three dependencies are mandatory:
-
-* `streampipes-container-standalone`, defines that we are going to create a
new pipeline element where the description will be accessible through an
embedded web server.
-* `streampipes-sdk` imports the SDK which provides many convencience functions
to create new pipeline elements.
-* `streampipes-vocabulary` imports various RDF vocabularies which are used by
the SDK to auto-generate the semantic description of pipeline elements.
-
-The following three dependencies might be optional depending on the pipeline
element type you plan to create:
-
-* `streampipes-wrapper-flink` should be used in case you plan to connect a
new data processor or data sink that uses Apache Flink for processing events at
runtime.
-* `streampipes-wrapper-standalone` should be used in case you plan to connect
a new data processor or data sink that does not use an external processing
engine. Events are directly processed in a single-host fashion.
-
-
-Finally, this dependency will provide abstract classes to define data sources
and streams.
diff --git a/documentation/docs/dev-guide-event-model.md
b/documentation/docs/dev-guide-event-model.md
index 1134bdd..e67a9d2 100644
--- a/documentation/docs/dev-guide-event-model.md
+++ b/documentation/docs/dev-guide-event-model.md
@@ -3,139 +3,3 @@ id: dev-guide-event-model
title: SDK Guide: Event Model
sidebar_label: Event Model
---
-
-## Introduction
-
-The 0.61.0 release of StreamPipes introduces a new event model that replaces
the ``Map`` representation of events at runtime. This guide explains the usage
of the new event model to manipulate runtime events for data processors and
data sink.
-
-## Reference
-
-This guide assumes that you are already familiar with the basic setup of [data
processors](dev-guide-processor-sdk.md) and [data sinks](dev-guide-sink-sdk.md).
-
-### Property Selectors
-
-In most cases, fields that are subject to be transformed by pipeline elements
are provided by the assigned ``MappingProperty`` (see the guide on [static
properties](dev-guide-static-properties.md)).
-
-Mapping properties return a ``PropertySelector`` that identifies a field based
on (i) the **streamIndex** and (ii) the runtime name of the field.
-Let's assume we have an event with the following structure:
-
-```json
-{
- "timestamp" : 1234556,
- "temperature" : 37.0,
- "deviceId" : "sensor1",
- "running" : true,
- "location" : {"latitude" : 34.4, "longitude" : -47},
- "lastValues" : [45, 22, 21]
-}
-```
-
-In addition, we assume that a data processor exists (with one input node) that
converts the temperature value (measured in degrees celsius) to a degree
fahrenheit value.
-In this case, a mapping property (selected by the pipeline developer in the
StreamPipes UI) would link to the ``temperature`` field of the event.
-
-The mapping property value will be the ``PropertySelector`` of the temperature
value, which looks as follows:
-
-```
-s0::temperature
-```
-
-``s0`` identifies the stream (in this case, only one input streams exist, but
as data processors might require more than one input stream, a stream
identifier is required), while the appendix identifies the runtime name.
-
-Note: If you add a new field to an input event, you don't need to provide the
selector, you can just assign the runtime name as defined by the [output
strategy](dev-guide-output-strategies.md).
-
-### Reading Fields
-
-You can get a field from an event by providing the corresponding selector:
-
-```java
-
-@Override
- public void onEvent(Event event, SpOutputCollector out) {
-
- PrimitiveField temperatureField =
event.getFieldBySelector(PROPERTY_SELECTOR).getAsPrimitive();
- }
-
-```
-
-Similarly, if your mapping property links to a nested property, use
-
-```java
-
-@Override
- public void onEvent(Event event, SpOutputCollector out) {
-
- NestedField nestedField =
event.getFieldBySelector(PROPERTY_SELECTOR).getAsNested();
- }
-
-```
-
-and for a list-based field:
-
-```java
-
-@Override
- public void onEvent(Event event, SpOutputCollector out) {
-
- ListField listField =
event.getFieldBySelector(PROPERTY_SELECTOR).getAsList();
- }
-
-```
-
-### Parsing Fields
-
-#### Primitive Fields
-
-A ``PrimitiveField`` contains convenience methods to directly cast a field to
the target datatype:
-
-```java
-
-// parse the value as a float datatype
-Float temperatureValue =
event.getFieldBySelector(temperatureSelector).getAsPrimitive().getAsFloat();
-
-// or do the same with a double datatype
-Double temperatureValue =
event.getFieldBySelector(temperatureSelector).getAsPrimitive().getAsDouble();
-
-// extracting a string
-String deviceId =
event.getFieldBySelector(deviceIdSelector).getAsPrimitive().getAsString();
-
-// this also works for extracting fields from nested fields:
-Double latitude =
event.getFieldBySelector(latitudeSelector).getAsPrimitive().getAsDouble();
-
-// extracting boolean values
-Boolean running =
event.getFieldBySelector(runningSelector).getAsPrimitive().getAsBoolean();
-```
-
-In rare cases, you might want to receive a field directly based on the runtime
name as follows:
-
-```java
-Double temperature =
event.getFieldByRuntimeName("temperature").getAsPrimitive().getAsDouble();
-```
-
-#### List Fields
-
-Lists can also be retrieved by providing the corresponding selector and can
automatically be parsed to a list of primitive datatypes:
-
-```java
-
-List<Integer> lastValues =
event.getFieldBySelector(lastValueSelector).getAsList().parseAsSimpleType(Integer.class);
-
-```
-
-(coming soon: parsing complex lists)
-
-
-### Adding/Updating Fields
-
-Primitive fields can easily be added to an event by providing the runtime name
and the object:
-
-```java
-
- // add a primitive field with runtime name "city" and value "Karlsruhe"
- event.addField("city", "Karlsruhe");
-
- // remove the field "temperature" from the event
- event.removeFieldBySelector(temperatureSelector);
-
- // add a new field
- event.addField("fahrenheit", 48);
-```
diff --git a/documentation/docs/dev-guide-output-strategies.md
b/documentation/docs/dev-guide-output-strategies.md
index ac17f6a..bdd206f 100644
--- a/documentation/docs/dev-guide-output-strategies.md
+++ b/documentation/docs/dev-guide-output-strategies.md
@@ -3,344 +3,3 @@ id: dev-guide-output-strategies
title: SDK Guide: Output Strategies
sidebar_label: Output Strategies
---
-
-## Introduction
-In StreamPipes, output strategies determine the output of a data processor.
-As the exact input schema of a processor is usually not yet known at
development time (as processors can be connected with any stream that matches
their requirements), output strategies are a concept to define how an input
data stream is transformed to an output data stream.
-
-The following reference describes how output strategies can be defined using
the SDK.
-
-<div class="admonition tip">
-<div class="admonition-title">Code on Github</div>
-<p>For all examples, the code can be found on <a
href="https://www.github.com/apache/incubator-streampipes-examples/tree/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/streampipes/pe/examples/jvm/outputstrategy/";>Github</a>.</p>
-</div>
-
-## Reference
-
-The methods described below to create static properties are available in the
``ProcessingElementBuilder`` class and are usually used in the ``declareModel``
method of the controller class.
-
-As follows, we will use the following example event to explain how output
strategies define the output of a data processor:
-
-```json
-{
- "timestamp" : 1234556,
- "temperature" : 37.0,
- "deviceId" : "1"
-
-}
-```
-
-### Keep Output
-
-A ``KeepOutputStrategy`` declares that the output event schema will be equal
to the input event schema.
-In other terms, the processor does not change the schema, but might change the
values of event properties.
-
-A keep output strategy can be defined as follows:
-
-```java
-
-@Override
- public DataProcessorDescription declareModel() {
- return
ProcessingElementBuilder.create("org.streampipes.examples.outputstrategy" +
- ".keep", "Keep output example example", "")
- .requiredStream(StreamRequirementsBuilder.
- create()
- .requiredProperty(EpRequirements.anyProperty())
- .build())
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
-
- // declaring a keep output strategy
- .outputStrategy(OutputStrategies.keep())
-
- .build();
- }
-
-```
-
-According to the example above, the expected output event schema of the
example input event would be:
-
-```json
-{
- "timestamp" : 1234556,
- "temperature" : 37.0,
- "deviceId" : "1"
-
-}
-```
-
-Data processors that perform filter operations (e.g., filtering temperature
values that are above a given threshold) are a common example for using keep
output strategies.
-
-
-### Fixed Output
-
-A ``FixedOutputStrategy`` declares that the data processor itself provides the
event schema. The output schema does not depend on the input event.
-
-Fixed output strategies need to provide the event schema they produce at
development time:
-
-```java
-
- @Override
- public DataProcessorDescription declareModel() {
- return
ProcessingElementBuilder.create("org.streampipes.examples.outputstrategy" +
- ".fixed", "Fixed output example", "")
- .requiredStream(StreamRequirementsBuilder.
- create()
- .requiredProperty(EpRequirements.anyProperty())
- .build())
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
-
- // the fixed output strategy provides the schema
-
.outputStrategy(OutputStrategies.fixed(EpProperties.timestampProperty("timestamp"),
- EpProperties.doubleEp(Labels.from("avg", "Average value",
""), "avg", SO.Number)))
-
- .build();
- }
-
-```
-
-In this example, we declare that the output schema always consists of two
fields (``timestamp`` and ``avg``).
-
-Therefore, an output event should look like:
-
-```json
-{
- "timestamp" : 1234556,
- "avg" : 36.0
-}
-```
-
-
-### Append Output
-
-An ``AppendOutputStrategy`` appends additional fields to a schema of an
incoming event stream. For instance, data processors that perform enrichment
operations usually make use of append output strategies.
-
-Similar to the fixed output strategy, the additional fields must be provided
at development time in the controller method as follows:
-
-```java
- @Override
- public DataProcessorDescription declareModel() {
- return
ProcessingElementBuilder.create("org.streampipes.examples.outputstrategy" +
- ".append", "Append output example", "")
-
- // boilerplate code not relevant here, see above
-
- // declaring an append output
-
.outputStrategy(OutputStrategies.append(EpProperties.integerEp(Labels.from("avg",
- "The average value", ""), "avg", SO.Number)))
-
- .build();
- }
-```
-
-In this case, the output event would have an additional field ``avg``:
-
-```json
-{
- "timestamp" : 1234556,
- "temperature" : 37.0,
- "deviceId" : "1",
- "avg" : 123.0
-
-}
-```
-
-### Custom Output
-
-In some cases, pipeline developers using the StreamPipes UI should be able to
manually select fields from an input event schema. For such use cases, a
``CustomOutputStrategy`` can be used:
-
-```java
-
-@Override
- public DataProcessorDescription declareModel() {
- return
ProcessingElementBuilder.create("org.streampipes.examples.outputstrategy" +
- ".custom", "Custom output example", "")
-
- // boilerplate code not relevant here, see above
-
- // declaring a custom output
- .outputStrategy(OutputStrategies.custom())
-
- .build();
- }
-
-```
-
-If a data processor defines a custom output strategy, the customization dialog
in the pipeline editor will show a dialog to let users select the fields to
keep:
-
-<img src="/docs/img/dev-guide-output-strategies/os-custom.png" width="80%"
alt="Number Parameter">
-
-Taking our example, and assuming that the user selects both the ``timestamp``
and the ``temperature`` the expected output event should look like this:
-
-```json
-{
- "timestamp" : 1234556,
- "temperature" : 37.0
-}
-```
-
-How do we know which fields were selected once the data processor is invoked?
Use the proper method from the extractor in the ``onInvocation`` method:
-
-```java
-@Override
- public ConfiguredEventProcessor<DummyParameters>
onInvocation(DataProcessorInvocation graph, ProcessingElementParameterExtractor
extractor) {
-
- List<String> outputSelectors = extractor.outputKeySelectors();
-
- return new ConfiguredEventProcessor<>(new DummyParameters(graph),
DummyEngine::new);
- }
-```
-
-### Transform Output
-
-A ``TransformOutputStrategy`` declares that one or more fields of an incoming
event stream are transformed. Transformations can be applied to the datatype of
the property, the runtime name of the property, or any other scheam-related
declaration such as measurement units.
-
-#### Static Transform Operations
-
-Static transform operations do not depend on any user input (at pipeline
development time) in order to know how to transform a field of an incoming
event schema.
-
-Let's say our data processor transforms strings (that are actually a number)
to a number datatype. In this case, we can use a static transform output
strategy:
-
-```java
-
- @Override
- public DataProcessorDescription declareModel() {
- return
ProcessingElementBuilder.create("org.streampipes.examples.outputstrategy" +
- ".transform", "Transform output example example", "")
- .requiredStream(StreamRequirementsBuilder.
- create()
-
.requiredPropertyWithUnaryMapping(EpRequirements.stringReq(), Labels.from
- ("str", "The date property as a string", ""),
PropertyScope.NONE)
- .build())
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
-
- // static transform operation
- .outputStrategy(OutputStrategies.transform(TransformOperations
- .staticDatatypeTransformation("str", Datatypes.Long)))
-
- .build();
- }
-
-```
-
-Note the mapping property that we use to determine which field of the input
event should be transformed.
-
-The expected output event would look like this:
-
-```json
-{
- "timestamp" : 1234556,
- "temperature" : 37.0,
- "deviceId" : 1
-}
-```
-
-#### Dynamic Transform Operations
-
-Sometimes, user input depends on the exact transform output. Let's take a
field renaming processor as an example, which lets the user rename a field from
an input event schema to another field name.
-For such use cases, we can use a ``DynamicTransformOperation``:
-
-```java
-
- @Override
- public DataProcessorDescription declareModel() {
- return
ProcessingElementBuilder.create("org.streampipes.examples.outputstrategy" +
- ".transform", "Transform output example example", "")
- .requiredStream(StreamRequirementsBuilder.
- create()
-
.requiredPropertyWithUnaryMapping(EpRequirements.stringReq(), Labels.from
- ("str", "The date property as a string", ""),
PropertyScope.NONE)
- .build())
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
-
- // the text input to enter the new runtime name
- .requiredTextparameter(Labels.from("new-runtime-name", "New
Runtime Name", ""))
-
- // static transform operation
- .outputStrategy(OutputStrategies.transform(TransformOperations
- .dynamicRuntimeNameTransformation("str",
"new-runtime-name")))
-
- .build();
- }
-
-```
-
-For dynamic transform operations, an additional identifier that links to
another static property can be assigned and later be fetched in the
``onInvocation`` method.
-
-Assuming we want to rename the field ``temperature`` to ``temp``, the
resulting output event should look like this:
-
-```json
-{
- "timestamp" : 1234556,
- "temp" : 37.0,
- "deviceId" : 1
-}
-```
-
-### Custom Transform Output
-
-Finally, in some cases the output schema cannot be described at pipeline
development time. For these (usually rare) cases, a ``CustomTransformOutput``
strategy can be used.
-
-In this case, a callback function will be invoked in the controller class just
after a user has filled in any static properties and clicks on ``Save`` in the
pipeline editor.
-
-To define a custom transform output, we need to implement an interface in the
controller class:
-
-```java
-public class CustomTransformOutputController extends
- StandaloneEventProcessingDeclarer<DummyParameters> implements
- ResolvesContainerProvidedOutputStrategy<DataProcessorInvocation,
ProcessingElementParameterExtractor> {
-
-
-@Override
- public EventSchema resolveOutputStrategy(DataProcessorInvocation
processingElement, ProcessingElementParameterExtractor parameterExtractor)
throws SpRuntimeException {
-
- }
-}
-```
-
-In addition, the output strategy must be declared in the ``declareModel``
method:
-
-```java
-
-@Override
- public DataProcessorDescription declareModel() {
- return
ProcessingElementBuilder.create("org.streampipes.examples.outputstrategy" +
- ".customtransform", "Custom transform output example example", "")
- .requiredStream(StreamRequirementsBuilder.
- create()
-
.requiredPropertyWithUnaryMapping(EpRequirements.stringReq(), Labels.from
- ("str", "The date property as a string", ""),
PropertyScope.NONE)
- .build())
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
-
- // declare a custom transform output
- .outputStrategy(OutputStrategies.customTransformation())
-
- .build();
- }
-
-```
-
-Once a new pipeline using this data processor is created and the configuration
is saved, the ``resolveOutputStrategy`` method will be called, so that an event
schema can be provided based on the given configuration. An extractor instance
(see the guide on static properties) is available to extract the selected
static properties and the connected event stream.
-
-```java
-@Override
- public EventSchema resolveOutputStrategy(DataProcessorInvocation
processingElement, ProcessingElementParameterExtractor parameterExtractor)
throws SpRuntimeException {
- return new EventSchema(Arrays
- .asList(EpProperties
- .stringEp(Labels.from("runtime", "I was added at runtime",
""), "runtime", SO.Text)));
- }
-```
-
-In this example, the output event schema should look like this:
-
-```json
-{
- "runtime" : "Hello world!"
-}
-```
-
diff --git a/documentation/docs/dev-guide-ssl.md
b/documentation/docs/dev-guide-ssl.md
index a6f3453..bab974b 100644
--- a/documentation/docs/dev-guide-ssl.md
+++ b/documentation/docs/dev-guide-ssl.md
@@ -3,33 +3,3 @@ id: dev-guide-ssl
title: Use SSL
sidebar_label: Use SSL
---
-
-In this tutorial we explain how you can use SSL Certificates to provide
transport layer security between your Browser and the Streampipes Backend.
-
-## Prerequisites
-You need a valid Certificate consisting of a Private and a Public Key. Both
Keys must be in PEM Format. Please note that your Private Key should never be
shared, otherwise the communication can not be considered secure.
-
-## Edit docker-compose.yml
-In order to use SSL you have to open port 443 on the nginx Service. Incoming
insecure Traffic on Port 80 will automatically be reroutet to Port 443.
-
-The Environment-Variable NGINX_SSL must be set to "true".
-
-Finally you have to inject the Certificates into the Docker-Container. In the
example below, the Certificates are placed in the directory /etc/ssl/private/
on the host machine. Please change the path according to the place where the
Certificates are located on your machine. The path after the colon should not
be changed!
-```yaml
-[...]
- nginx:
- image: streampipes/ui
- ports:
- - "80:80"
- - "443:443"
- environment:
- - NGINX_SSL=true
- volumes:
- - /etc/ssl/private/private.pem:/etc/nginx/ssl/ssl.pem
- - /etc/ssl/private/public.pem:/etc/nginx/ssl/cert.pem
- depends_on:
- - backend
- networks:
- spnet:
-[...]
-```
\ No newline at end of file
diff --git a/documentation/docs/dev-guide-static-properties.md
b/documentation/docs/dev-guide-static-properties.md
index de2e0c3..2f607c6 100644
--- a/documentation/docs/dev-guide-static-properties.md
+++ b/documentation/docs/dev-guide-static-properties.md
@@ -3,262 +3,3 @@ id: dev-guide-static-properties
title: SDK Guide: Static Properties
sidebar_label: Static Properties
---
-
-## Introduction
-Static properties represent user-faced parameters that are provided by
pipeline developers.
-Processing elements can specify required static properties, which will render
different UI views in the pipeline editor.
-
-The following reference describes how static properties can be defined using
the SDK.
-
-<div class="admonition tip">
-<div class="admonition-title">Code on Github</div>
-<p>For all examples, the code can be found on <a
href="https://github.com/apache/incubator-streampipes-examples/tree/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/streampipes/pe/examples/jvm/staticproperty";>Github</a>.</p>
-</div>
-
-## Reference
-
-The methods described below to create static properties are available in the
``ProcessingElementBuilder`` and ``DataSinkBuilder`` classes and are usually
used in the ``declareModel`` method of the controller class.
-
-### Mapping property
-
-In StreamPipes, processing elements usually operate on fields of an event
stream. For instance, a filter processor operates on a specific field from an
input stream (e.g., a field measuring the temperature).
-Typically, pipeline developers should select the exact field where the
operations is applied upon by themselves.
-As this field is not yet known at pipeline element development time (as it is
defined by the pipeline developer in the pipeline editor), mapping properties
serve to map a stream requirement to a specific field from the actual input
event stream.
-
-### Unary mapping property
-
-A unary mapping property maps a stream requirement to an actual field of an
event stream. Therefore, the ``StreamRequirementsBuilder`` provides the
opportunity to directly add a mapping property based along with a property
requirement:
-
-```java
-.requiredStream(StreamRequirementsBuilder.
- create()
- .requiredPropertyWithUnaryMapping(EpRequirements.numberReq(),
- Labels.from("mp-key", "My Mapping", ""),
- PropertyScope.NONE)
- .build())
-```
-
-This leads to a selection dialog in the pipeline element customization which
provides the user with a selection of all event properties (fields) from the
input stream that match the specified property requirement:
-
-<img src="/docs/img/dev-guide-static-properties/sp-mapping-unary.png"
width="80%" alt="Text">
-
-At invocation time, the value can be extracted in the ``onInvocation`` method
as follows:
-
-```java
-// Extract the mapping property value
-String mappingPropertySelector = extractor.mappingPropertyValue("mp-key");
-```
-
-Note that this method returns a ``PropertySelector``, which can be used by the
event model to extract the actual value of this field.
-
-### N-ary mapping property
-
-N-ary mapping properties work similar to unary mapping properties, but allow
the mapping of one requirement to multiple event properties matching the
requirement:
-
-```java
-.requiredStream(StreamRequirementsBuilder.
- create()
- .requiredPropertyWithNaryMapping(EpRequirements.numberReq(),
- Labels.from("mp-key", "My Mapping", ""),
- PropertyScope.NONE)
- .build())
-```
-
-This renders the following selection, where users can select more than one
matching event property:
-
-<img src="/docs/img/dev-guide-static-properties/sp-mapping-nary.png"
width="80%" alt="Text">
-
-The following snippet returns a list containing the property selectors of all
event properties that have been selected:
-
-```java
-// Extract the mapping property value
-List<String> mappingPropertySelectors =
extractor.mappingPropertyValues("mp-key");
-```
-
-### Free-Text Parameters
-
-A free-text parameter requires the pipeline developer to enter a single value
- which can be a string or another primitive data type.
-The input of free-text parameters can be restricted to specific value ranges
or can be linked to the value set of a connected input data stream.
-
-#### Text Parameters
-
-A text parameter lets the user enter a string value. The following code line
in the controller class
-
-```java
-.requiredTextParameter(Labels.from(SP_KEY, "Example Name", "Example
Description"))
-```
-
-leads to the following input dialog in the pipeline editor:
-
-<img src="/docs/img/dev-guide-static-properties/sp-text-parameter.png"
width="80%" alt="Text">
-
-Users can enter any value that will be converted to a string datatype. To
receive the entered value in the ``onInvocation`` method, use the following
method from the ``ParameterExtractor``
-
-```java
-String textParameter = extractor.singleValueParameter(SP_KEY, String.class);
-```
-
-#### Number parameters
-
-A number parameter lets the user enter a number value, either a floating-point
number or an integer:
-
-```java
-// create an integer parameter
-.requiredIntegerParameter(Labels.from(SP_KEY, "Integer Parameter", "Example
Description"))
-
-// create a float parameter
-.requiredFloatParameter(Labels.from("float-key", "Float Parameter", "Example
Description"))
-
-```
-
-leads to the following input dialog in the pipeline editor only accepting
integer values:
-
-<img src="/docs/img/dev-guide-static-properties/sp-number-parameter.png"
width="80%" alt="Number Parameter">
-
-The pipeline editor performs type validation and ensures that only numbers can
be added by the user. To receive the entered value in the ``onInvocation``
method, use the following method from the ``ParameterExtractor``
-
-```java
-// Extract the integer parameter value
-Integer integerParameter = extractor.singleValueParameter(SP_KEY,
Integer.class);
-
-// Extract the float parameter value
-Float floatParameter = extractor.singleValueParameter("float-key",
Float.class);
-
-```
-
-#### Numbers with value specification
-
-You can also specify the value range of a number-based free text parameter:
-
-```java
-// create an integer parameter with value range
-.requiredIntegerParameter(Labels.from(SP_KEY, "Integer Parameter", "Example
Description"), 0, 100, 1)
-
-```
-
-which renders the following input field:
-
-<img
src="/docs/img/dev-guide-static-properties/sp-number-parameter-with-range.png"
width="80%" alt="Number Parameter">
-
-Receive the entered value in the same way as a standard number parameter.
-
-#### Free-text parameters linked to an event property
-
-
-### Single-Value Selections
-
-Single-value selections let the user select from a pre-defined list of options.
-A single-value selection requires to select exactly one option.
-
-```java
-.requiredSingleValueSelection(Labels.from("id", "Example Name", "Example
Description"),
- Options.from("Option A", "Option B", "Option C"))
-
-```
-
-Single-value selections will be rendered as a set of radio buttons in the
pipeline editor:
-
-<img src="/docs/img/dev-guide-static-properties/sp-single-selection.png"
width="80%" alt="Number Parameter">
-
-To extract the selected value, use the following method from the parameter
extractor:
-
-```java
-// Extract the selected value
-String selectedSingleValue = extractor.selectedSingleValue("id", String.class);
-```
-
-<div class="admonition tip">
-<div class="admonition-title">Declaring options</div>
-<p>Sometimes, you may want to use an internal name that differs from the
display name of an option.
-For that, you can use the method Options.from(Tuple2<String, String>) and the
extractor method selectedSingleValueInternalName.</p>
-</div>
-
-
-
-### Multi-Value Selections
-
-Multi-value selections let the user select from a pre-defined list of options,
where multiple or no option might be selected.
-
-```java
-.requiredMultiValueSelection(Labels.from("id", "Example Name", "Example
Description"),
- Options.from("Option A", "Option B", "Option C"))
-
-```
-
-Multi-value selections will be rendered as a set of checkboxes in the pipeline
editor:
-
-<img src="/docs/img/dev-guide-static-properties/sp-multi-selection.png"
width="80%" alt="Number Parameter">
-
-To extract the selected value, use the following method from the parameter
extractor:
-
-```java
-// Extract the selected value
-List<String> selectedMultiValue = extractor.selectedMultiValues("id",
String.class);
-```
-
-### Domain Concepts
-
-(coming soon...)
-
-### Collections
-
-You can also define collections based on other static properties.
-
-```java
-// create a collection parameter
-.requiredParameterAsCollection(Labels.from("collection", "Example Name",
"Example " +
- "Description"), StaticProperties.stringFreeTextProperty(Labels
- .from("text-property","Text","")))
-```
-
-While the items of the collection can be provided in the same way as the
underlying static property, the UI provides buttons to add and remove items to
the collections.
-
-<img src="/docs/img/dev-guide-static-properties/sp-collection.png" width="80%"
alt="Number Parameter">
-
-To extract the selected values from the collection, use the following method
from the parameter extractor:
-
-```java
-// Extract the text parameter value
-List<String> textParameters =
extractor.singleValueParameterFromCollection("collection", String.class);
-```
-
-### Runtime-resolvable selections
-
-In some cases, the options of selection parameters are not static, but depend
on other values or might change at runtime. In this case, you can use
runtime-resolvable selections.
-
-First, let your controller class implement
``ResolvesContainerProvidedOptions``:
-
-```java
-public class RuntimeResolvableSingleValue extends
- StandaloneEventProcessingDeclarer<DummyParameters> implements
ResolvesContainerProvidedOptions { ... }
-```
-
-Next, define the parameter in the ``declareModel`` method:
-
-```java
-// create a single value selection parameter that is resolved at runtime
- .requiredSingleValueSelectionFromContainer(Labels.from("id", "Example
Name", "Example " +
- "Description"))
-```
-
-Finally, implement the method ``resolveOptions``, which will be called at
runtime once the processor is used:
-
-```java
- @Override
- public List<RuntimeOptions> resolveOptions(String requestId, EventProperty
linkedEventProperty) {
- return Arrays.asList(new RuntimeOptions("I was defined at runtime", ""));
- }
-```
-
-The UI will render a single-value parameter based on the options provided at
runtime:
-
-<img
src="/docs/img/dev-guide-static-properties/sp-single-selection-remote.png"
width="80%" alt="Number Parameter">
-
-The parameter extraction does not differ from the extraction of static
single-value parameters.
-
-<div class="admonition info">
-<div class="admonition-title">Multi-value selections</div>
-<p>Although this example shows the usage of runtime-resolvable selections
using single value selections, the same also works for multi-value
selections!</p>
-</div>
-
-
diff --git a/documentation/docs/dev-guide-stream-requirements.md
b/documentation/docs/dev-guide-stream-requirements.md
index 4fa7949..0fa2398 100644
--- a/documentation/docs/dev-guide-stream-requirements.md
+++ b/documentation/docs/dev-guide-stream-requirements.md
@@ -3,176 +3,3 @@ id: dev-guide-stream-requirements
title: SDK Guide: Stream Requirements
sidebar_label: Stream Requirements
---
-
-## Introduction
-
-Data processors and data sinks can define ``StreamRequirements``. Stream
requirements allow pipeline elements to express requirements on an incoming
event stream that are needed for the element to work properly.
-Once users create pipelines in the StreamPipes Pipeline Editor, these
requirements are verified against the connected event stream.
-By using this feature, StreamPipes ensures that only pipeline elements can be
connected that are syntactically and semantically valid.
-
-This guide covers the creation of stream requirements. Before reading this
section, we recommend that you make yourself familiar with the SDK guide on
[data processors](dev-guide-processor-sdk.md) and [data
sinks](dev-guide-sink-sdk.md).
-
-<div class="admonition tip">
-<div class="admonition-title">Code on Github</div>
-<p>For all examples, the code can be found on <a
href="https://www.github.com/apache/incubator-streampipes-examples/tree/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/streampipes/pe/examples/jvm/requirements/";>Github</a>.</p>
-</div>
-
-## The StreamRequirementsBuilder
-
-Stream requirements can be defined in the ``Controller`` class of the pipeline
element. Start with a method body like this:
-
-```java
-
-@Override
- public DataProcessorDescription declareModel() {
- return ProcessingElementBuilder.create(ID, PIPELINE_ELEMENT_NAME,
DESCRIPTION)
- .requiredStream(StreamRequirementsBuilder.
- create()
-
- .build())
-
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
- .outputStrategy(OutputStrategies.keep())
-
- .build();
- }
-```
-
-The ``StreamRequirementsBuilder`` class provides methods to add stream
requirements to a pipeline element.
-
-## Requirements on primitive fields
-
-As a very first example, let's assume we would like to create a data processor
that filters numerical values that are above a given threshold.
-Consequently, any data stream that is connected to the filter processor needs
to provide a numerical value.
-
-The stream requirement would be assigned as follows:
-
-```java
-@Override
- public DataProcessorDescription declareModel() {
- return ProcessingElementBuilder.create(ID, PIPELINE_ELEMENT_NAME,
DESCRIPTION)
- .requiredStream(StreamRequirementsBuilder
- .create()
- .requiredProperty(EpRequirements.numberReq())
- .build())
-
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
- .outputStrategy(OutputStrategies.keep())
-
- .build();
- }
-```
-
-Note the line starting with ``requiredProperty``, which requires any stream to
provide a datatype of type ``number``.
-
-In many cases, you'll want to let the user select a specific field from a data
stream from all available fields that match the specified requirement. For
that, you simply use the method ``requiredPropertyWithUnaryMapping`` as follows:
-
-```java
-@Override
- public DataProcessorDescription declareModel() {
- return ProcessingElementBuilder.create(ID, PIPELINE_ELEMENT_NAME,
DESCRIPTION)
- .requiredStream(StreamRequirementsBuilder
- .create()
-
.requiredPropertyWithUnaryMapping(EpRequirements.numberReq(),
- Labels.from("number-mapping", "The value that should be
filtered", ""), PropertyScope.NONE)
- .build())
-
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
- .outputStrategy(OutputStrategies.keep())
-
- .build();
- }
-```
-
-See also the developer guide on [static
properties](dev-guide-static-properties.md) to better understand the usage of
``MappingProperties``.
-
-Requirements on primitive fields can be specified for all common datatypes:
-
-```java
- @Override
- public DataProcessorDescription declareModel() {
- return
ProcessingElementBuilder.create("org.streampipes.examples.requirements" +
- ".simple", "Simple requirements specification examples", "")
- .requiredStream(StreamRequirementsBuilder.
- create()
- .requiredProperty(EpRequirements.numberReq()) // any number
- .requiredProperty(EpRequirements.doubleReq()) // any field
of type double
- .requiredProperty(EpRequirements.booleanReq()) // any
field of type boolean
- .requiredProperty(EpRequirements.integerReq()) // any
field of type integer
- .requiredProperty(EpRequirements.stringReq()) // any field
of type string
-
- .requiredProperty(EpRequirements.anyProperty()) // any
field allowed (no restriction)
- .requiredProperty(EpRequirements.timestampReq()) // any
timestamp field
- .build())
-
-
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
- .outputStrategy(OutputStrategies.keep())
-
- .build();
-```
-
-### Specifying semantics
-
-For some algorithms, only specifying the datatype is not sufficient. Let's
consider a geofencing algorithm that detects the precense some geospatial
coordinate (e.g., from a vehicle) within a given location.
-
-You could specify something like this:
-
-```java
- StreamRequirementsBuilder
- .create()
- .requiredPropertyWithUnaryMapping(EpRequirements.doubleEp(),
Labels.from("mapping-latitude", "Latitude", ""), PropertyScope.NONE)
- .requiredPropertyWithUnaryMapping(EpRequirements.doubleEp(),
Labels.from("mapping-longitude", "Longitude", ""), PropertyScope.NONE)
- .build()
-```
-
-However, this would allow users to create strange pipelines as users could
connect any stream containing a double value to our geofencing algorithm.
-To avoid such situations, you can also specify requirements based on the
semantics of a field:
-
-```java
- StreamRequirementsBuilder
- .create()
-
.requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(SO.Latitude),
Labels.from("mapping-latitude", "Latitude", ""), PropertyScope.NONE)
-
.requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(SO.Longitude),
Labels.from("mapping-longitude", "Longitude", ""), PropertyScope.NONE)
- .build()
-```
-
-Note that in this case, we make use of Schema.org's ``Latitude`` concept
([https://schema.org/latitude](https://schema.org/latitude)). StreamPipes
already includes popular vocabularies for specifying semantics. You are also
free to use your own vocabularies.
-
-
-## Requirements on lists
-
-Similarly to primitive requirements, you can define processors that require
data streams with list fields, see the following examples:
-
-```java
-@Override
- public DataProcessorDescription declareModel() {
- return
ProcessingElementBuilder.create("org.streampipes.examples.requirements" +
- ".list", "List requirements specification examples", "")
- .requiredStream(StreamRequirementsBuilder.
- create()
-
.requiredProperty(EpRequirements.listRequirement(Datatypes.Integer))
-
.requiredProperty(EpRequirements.listRequirement(Datatypes.Double))
-
.requiredProperty(EpRequirements.listRequirement(Datatypes.Boolean))
-
.requiredProperty(EpRequirements.listRequirement(Datatypes.String))
- .build())
-
-
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
- .outputStrategy(OutputStrategies.keep())
-
- .build();
- }
-```
-
-## Requirements on nested properties
-
-(coming soon, see the Javadoc for now)
-
-
-
diff --git a/documentation/docs/dev-guide-tutorial-processors.md
b/documentation/docs/dev-guide-tutorial-processors.md
index e346368..5e9e6dc 100644
--- a/documentation/docs/dev-guide-tutorial-processors.md
+++ b/documentation/docs/dev-guide-tutorial-processors.md
@@ -3,497 +3,3 @@ id: dev-guide-tutorial-processors
title: Tutorial: Data Processors
sidebar_label: Tutorial: Data Processors
---
-
-In this tutorial, we will add a new data processor using the Apache Flink
wrapper.
-
-From an architectural point of view, we will create a self-contained service
that includes the description of the data processor and a Flink-compatible
implementation.
-Once a pipeline is started that uses this data processor, the implementation
is submitted to an Apache Flink cluster.
-
-## Objective
-
-We are going to create a new data processor that realized a simple geofencing
algorithm - we detect vehicles that enter a specified radius around a
user-defined location.
-This pipeline element will be a generic element that works with any event
stream that provides geospatial coordinates in form of a latitude/longitude
pair.
-
-The algorithm outputs every location event once the position has entered the
geofence.
-
-
-<div class="admonition note">
-<div class="admonition-title">Note</div>
-<p>The implementation in this tutorial is pretty simple - our processor will
fire an event every time the GPS location is inside the geofence.
- In a real-world application, you would probably want to define a
pattern that recognizes the _first_ event a vehicle enters the geofence.<br/>
- This can be easily done using a CEP library, e.g., Apache Flink CEP.</p>
-</div>
-
-
-## Project setup
-
-To create new projects from scratch, several Maven archetypes exist to start
developing.
-Enter the following command to create a new project based on the StreamPipes
``Processors-Flink`` archetype:
-
-```
-mvn archetype:generate -DarchetypeGroupId=org.streampipes \
--DarchetypeArtifactId=streampipes-archetype-pe-processors-flink
-DarchetypeVersion=0.64.0 \
--DgroupId=org.streampipes.tutorial -DartifactId=geofence-tutorial
-DclassNamePrefix=Geofencing -DpackageName=geofencing
-```
-
-Once you've imported the generated project, the project structure should look
as follows:
-
-<img src="/docs/img/tutorial-processors/project-structure-processor.PNG"
alt="Project Structure Data Processor">
-
-
-<div class="admonition tip">
-<div class="admonition-title">Tip</div>
-<p>Besides the basic project skeleton, the sample project also includes an
example Dockerfile you can use to package your application into a Docker
container.</p>
-</div>
-
-Now you're ready to create your first data processor for StreamPipes!
-
-## Adding data processor requirements
-
-First, we will add a new stream requirement.
-Open the class `GeofencingController` which should look as follows:
-
-```java
-package org.streampipes.tutorial.pe.processor.geofencing;
-
-import org.streampipes.tutorial.config.Config;
-
-import org.streampipes.model.DataProcessorType;
-import org.streampipes.model.graph.DataProcessorDescription;
-import org.streampipes.model.graph.DataProcessorInvocation;
-import org.streampipes.sdk.builder.ProcessingElementBuilder;
-import org.streampipes.sdk.builder.StreamRequirementsBuilder;
-import org.streampipes.sdk.extractor.ProcessingElementParameterExtractor;
-import org.streampipes.sdk.helpers.EpRequirements;
-import org.streampipes.sdk.helpers.Labels;
-import org.streampipes.sdk.helpers.OutputStrategies;
-import org.streampipes.sdk.helpers.SupportedFormats;
-import org.streampipes.sdk.helpers.SupportedProtocols;
-import org.streampipes.wrapper.flink.FlinkDataProcessorDeclarer;
-import org.streampipes.wrapper.flink.FlinkDataProcessorRuntime;
-
-public class GeofencingController extends
-
FlinkDataProcessorDeclarer<GeofencingParameters> {
-
- private static final String EXAMPLE_KEY = "example-key";
-
- @Override
- public DataProcessorDescription declareModel() {
- return
ProcessingElementBuilder.create("org.streampipes.tutorial-geofencing")
-
.category(DataProcessorType.ENRICH)
- .withAssets(Assets.DOCUMENTATION, Assets.ICON)
- .withLocales(Locales.EN)
-
.requiredStream(StreamRequirementsBuilder
- .create()
-
.requiredProperty(EpRequirements.anyProperty())
- .build())
-
.outputStrategy(OutputStrategies.keep())
-
.requiredTextParameter(Labels.from(EXAMPLE_KEY, "Example Text Parameter",
"Example " +
- "Text Parameter Description"))
- .build();
- }
-
- @Override
- public FlinkDataProcessorRuntime<GeofencingParameters>
getRuntime(DataProcessorInvocation
- graph, ProcessingElementParameterExtractor
extractor) {
-
- String exampleString =
extractor.singleValueParameter(EXAMPLE_KEY, String.class);
-
- GeofencingParameters params = new GeofencingParameters(graph,
exampleString);
-
- return new GeofencingProgram(params,
Config.INSTANCE.getDebug());
- }
-
-}
-
-```
-
-In this class, we need to implement two methods: The `declareModel` method is
used to define abstract stream requirements such as event properties that must
be present in any input stream that is later connected to the element using the
StreamPipes UI.
-The second method, `getRuntime` is used to create and deploy the parameterized
Flink program once a pipeline using this element is started.
-
-Similar to data sources, the SDK provides a builder class to generate the
description for data processors.
-Delete the content within the ``declareModel`` method and add the following
lines to the `declareModel` method:
-
-```java
-return ProcessingElementBuilder.create("org.streampipes.tutorial.geofencing",
"Geofencing", "A simple geofencing data processor " +
- "using the Apache Flink wrapper")
-```
-
-This creates a new data processor with the ID, title and description assigned
to the element builder.
-Next, we add some _stream requirements_ to the description. As we'd like to
develop a generic pipeline element that works with any event that provides a
lat/lng pair, we define two stream requirements as stated below:
-
-```java
-.requiredStream(StreamRequirementsBuilder
- .create()
-
.requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(Geo.lat),
- Labels.from("latitude-field", "Latitude", "The event " +
- "property containing the latitude value"),
PropertyScope.MEASUREMENT_PROPERTY)
-
.requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(Geo.lng),
- Labels.from("longitude-field", "Longitude", "The event " +
- "property containing the longitude value"),
PropertyScope.MEASUREMENT_PROPERTY)
- .build())
-```
-
-The first line, `.requiredStream()` defines that we want a data processor with
exactly one input stream. Adding more stream requirements would create elements
with multiple input connectors in StreamPipes.
-Stream requirements can be assigned by using the `StreamRequirementsBuilder`
class.
-In our example, we define two requirements, so-called _domain property
requirements_. In contrast to _data type requirements_ where we'd expect an
event property with a field of a specific data type (e.g., float), domain
property requirements expect a specific domain property, e.g., from a
vocabulary such as the WGS84 Geo vocab.
-
-Once a pipeline is deployed, we are interested in the actual field (and its
field name) that contains the latitude and longitude values.
-In some cases, there might me more than one field that satisfies a property
requirement and we would like users to select the property the geofencing
component should operate on.
-Therefore, our example uses the method `requiredPropertyWithUnaryMapping`,
which will map a requirement to a real event property of an input stream and
let the user choose the appropriate field in the StreamPipes UI when pipelines
are defined.
-
-Finally, the `PropertyScope` indicates that the required property is a
measurement value (in contrast to a dimension value). This allows us later to
provide improved user guidance in the pipeline editor.
-
-Besides requirements, users should be able to define the center coordinate of
the Geofence and the size of the fence defined as a radius around the center in
meters.
-The radius can be defined by adding a simple required text field to the
description:
-
-```java
-.requiredIntegerParameter("radius", "Geofence Size", "The size of the circular
geofence in meters.", 0, 1000, 1)
-```
-
-Similar to mapping properties, text parameters have an internalId (radius), a
label and a description.
-In addition, we can assign a _value specification_ to the parameter indicating
the value range we support.
-Our example supports a radius value between 0 and 1000 with a granularity of 1.
-In the StreamPipes UI, a required text parameter is rendered as a text input
field, in case we provide an optional value specification, a slider input is
automatically generated.
-
-Such user-defined parameters are called _static properties_. There are many
different types of static properties (see the [Processor
SDK](dev-guide-sdk-guide-processors#docsNav) for an overview).
-
-One example are _DomainProperties_ we use for defining the center of the
geofence.
-Our data processor requires a lat/lng pair that marks the center of the
geofence.
-However, letting users directly input latitude and longitude coordinates
wouldn't be very user-friendly.
-Therefore, we can link required text parameters to _ontology concepts_. By
understanding the required input, the StreamPipes UI is able to determine which
user interface works best for a specific concept.
-
-Add the following line to the `declareModel` method:
-
-```java
-.requiredOntologyConcept(Labels.from("location", "Geofence Center", "Provide
the coordinate of the " +
- "geofence center"), OntologyProperties.mandatory(Geo.lat),
OntologyProperties.mandatory(Geo.lng))
-
-```
-
-We've now defined that we would like to receive an instance that provides a
latitude and a longitude coordinate.
-Users can input these values either manually, or they can look up _domain
knowledge_, i.e., knowledge stored isolated from the stream definition.
-
-Finally, we need to define technical requirements of the data processor,
called _groundings_.
-StreamPipes supports various runtime data formats (e.g., JSON or Thrift) and
various protocols (e.g., Kafka or JMS).
-Each component defines supports formats and protocol separately.
-For our example, we'd like to support JSON-based messages and Kafka as input
and output broker protocol, so append the following:
-
-```java
-.supportedProtocols(SupportedProtocols.kafka())
-.supportedFormats(SupportedFormats.jsonFormat())
-.build();
-```
-
-Now we need to define the output of our Geofencing pipeline element.
-As explained in the first section, the element should fire every time some
geo-located entity arrives within the defined geofence.
-Therefore, the processor outputs the same schema as it receives as an input.
-Although we don't know the exact input right now as it depends on the stream
users connect in StreamPipes when creating pipelines, we can define an _output
strategy_ as follows:
-
-```java
-.outputStrategy(OutputStrategies.keep())
-```
-
-This defines a _KeepOutputStrategy_, i.e., the input event schema is not
modified by the processor.
-There are many more output strategies you can define depending on the
functionality you desire, e.g., _AppendOutput_ for defining a processor that
enriches events or _CustomOutput_ in case you would like users to select the
output by themselves.
-
-That's it! We've now defined input requirements, required user input,
technical requirements concerned with data format and protocol and an output
strategy.
-In the next section, you will learn how to extract these parameters once the
pipeline element is invoked after a pipeline was created.
-
-## Pipeline element invocation
-
-Once users start a pipeline that uses our geofencing component, the
_getRuntime_ method in our class is called. The class `DataSinkInovcation`
includes a graph that contains information on the configuration parameters a
users has selected in the pipeline editor and information on the acutal streams
that are connected to the pipeline element.
-
-Before we explain in more detail how to extract these values from the
processor invocation, we need to adapt a little helper class.
-Open the file ```GeofencingParameters``` and modify it as follows:
-
-```java
-public class GeofencingParameters extends EventProcessorBindingParams {
-
- private String latitudeFieldName;
- private String longitudeFieldName;
-
- private Float centerLatitude;
- private Float centerLongitude;
-
- private Integer radius;
-
- public GeofencingParameters(DataProcessorInvocation graph, String
latitudeFieldName, String longitudeFieldName,
- Float centerLatitude, Float centerLongitude,
Integer radius) {
- super(graph);
- this.latitudeFieldName = latitudeFieldName;
- this.longitudeFieldName = longitudeFieldName;
- this.centerLatitude = centerLatitude;
- this.centerLongitude = centerLongitude;
- this.radius = radius;
- }
-
- public String getLatitudeFieldName() {
- return latitudeFieldName;
- }
-
- public String getLongitudeFieldName() {
- return longitudeFieldName;
- }
-
- public Float getCenterLatitude() {
- return centerLatitude;
- }
-
- public Float getCenterLongitude() {
- return centerLongitude;
- }
-
- public Integer getRadius() {
- return radius;
- }
-```
-
-This simple Pojo class will later serve to store user-defined parameters in a
single object.
-
-Now we go back to the controller class and extract these values from the
invocation object.
-
-The ``ProcessingElementParameterExtractor`` provides convenience methods to
extract the relevant information from the `DataProcessorInvocation` object.
-
-Next, we are interested in the fields of the input event stream that contains
the latitude and longitude value we would like to compute against the geofence
center location as follows:
-
-```java
-String latitudeFieldName = extractor.mappingPropertyValue("latitude-field");
-String longitudeFieldName = extractor.mappingPropertyValue("longitude-field");
-```
-
-We use the same `internalId` we've used to define the mapping property
requirements in the `declareModel` method.
-
-Next, for extracting the geofence center coordinates, we use the following
statements:
-
-```java
-Float centerLatitude = extractor.supportedOntologyPropertyValue("location",
Geo.lat, Float.class);
-Float centerLongitude = extractor.supportedOntologyPropertyValue("location",
Geo.lng, Float.class);
-```
-
-The radius value can be extracted as follows:
-
-```java
-Integer radius = extractor.singleValueParameter("radius", Integer.class);
-```
-
-Now we can create a new instance of our previously created parameter class:
-
-```java
-GeofencingParameters params = new
GeofencingParameters(dataProcessorInvocation, latitudeFieldName,
- longitudeFieldName, centerLatitude, centerLongitude, radius);
-```
-
-Finally, return an instance of the class ```GeofencingProgram```:
-
-```java
-return new GeofencingProgram(params, true);
-```
-
-<div class="admonition tip">
-<div class="admonition-title">Info</div>
-<p>The line above uses the Flink MiniCluster to start the Flink program for
debugging purposes.
- Before you build the project and use it in a real environment, replace
the line as follows, which triggers cluster execution:
- <code>return new GeofencingProgram(params, new
FlinkDeploymentConfig(Config.JAR_FILE, Config.INSTANCE.getFlinkHost(),
Config.INSTANCE.getFlinkPort())</code></p>
-</div>
-
-
-Great! That's all we need to describe a data processor for usage in
StreamPipes. Your controller class should look as follows:
-
-```java
-import org.streampipes.model.graph.DataProcessorDescription;
-import org.streampipes.model.graph.DataProcessorInvocation;
-import org.streampipes.model.schema.PropertyScope;
-import org.streampipes.sdk.builder.ProcessingElementBuilder;
-import org.streampipes.sdk.builder.StreamRequirementsBuilder;
-import org.streampipes.sdk.extractor.ProcessingElementParameterExtractor;
-import org.streampipes.sdk.helpers.EpRequirements;
-import org.streampipes.sdk.helpers.Labels;
-import org.streampipes.sdk.helpers.OntologyProperties;
-import org.streampipes.sdk.helpers.OutputStrategies;
-import org.streampipes.sdk.helpers.SupportedFormats;
-import org.streampipes.sdk.helpers.SupportedProtocols;
-import org.streampipes.vocabulary.Geo;
-import org.streampipes.wrapper.flink.FlinkDataProcessorDeclarer;
-import org.streampipes.wrapper.flink.FlinkDataProcessorRuntime;
-
-public class GeofencingController extends
FlinkDataProcessorDeclarer<GeofencingParameters> {
-
- @Override
- protected FlinkDataProcessorRuntime<GeofencingParameters>
getRuntime(DataProcessorInvocation dataProcessorInvocation) {
- ProcessingElementParameterExtractor extractor =
ProcessingElementParameterExtractor.from(dataProcessorInvocation);
-
- String latitudeFieldName =
extractor.mappingPropertyValue("latitude-field");
- String longitudeFieldName =
extractor.mappingPropertyValue("longitude-field");
-
- Float centerLatitude =
extractor.supportedOntologyPropertyValue("location", Geo.lat, Float.class);
- Float centerLongitude =
extractor.supportedOntologyPropertyValue("location", Geo.lng, Float.class);
-
- Integer radius = extractor.singleValueParameter("radius", Integer.class);
-
- GeofencingParameters params = new
GeofencingParameters(dataProcessorInvocation, latitudeFieldName,
- longitudeFieldName, centerLatitude, centerLongitude, radius);
-
- return new GeofencingProgram(params);
- }
-
- @Override
- public DataProcessorDescription declareModel() {
- return ProcessingElementBuilder.create("geofencing-flink", "Geofencing",
"A simple geofencing data processor " +
- "using the Apache Flink wrapper")
- .requiredStream(StreamRequirementsBuilder
- .create()
-
.requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(Geo.lat),
- Labels.from("latitude-field", "Latitude", "The
event " +
- "property containing the latitude value"),
PropertyScope.MEASUREMENT_PROPERTY)
-
.requiredPropertyWithUnaryMapping(EpRequirements.domainPropertyReq(Geo.lng),
- Labels.from("longitude-field", "Longitude", "The
event " +
- "property containing the longitude
value"), PropertyScope.MEASUREMENT_PROPERTY)
- .build())
- .requiredIntegerParameter("radius", "Geofence Size", "The size of
the circular geofence in meters.",
- 0, 1000, 1)
- .requiredOntologyConcept(Labels.from("location", "Geofence
Center", "Provide the coordinate of the " +
- "geofence center"), OntologyProperties.mandatory(Geo.lat),
OntologyProperties.mandatory(Geo.lng))
- .supportedProtocols(SupportedProtocols.kafka())
- .supportedFormats(SupportedFormats.jsonFormat())
- .outputStrategy(OutputStrategies.keep())
- .build();
- }
-}
-
-```
-
-## Adding an implementation
-
-Everything we need to do now is to add an implementation which does not differ
from writing an Apache Flink topology.
-
-Open the class `GeofencingProcessor.java` and add the following piece of code,
which realizes the Geofencing functionality:
-
-```java
-public class GeofencingProcessor implements FlatMapFunction<Map<String,
Object>, Map<String, Object>> {
-
- private String latitudeFieldName;
- private String longitudeFieldName;
-
- private Float centerLatitude;
- private Float centerLongitude;
-
- private Integer radius;
-
- public GeofencingProcessor(String latitudeFieldName, String
longitudeFieldName, Float centerLatitude, Float centerLongitude, Integer
radius) {
- this.latitudeFieldName = latitudeFieldName;
- this.longitudeFieldName = longitudeFieldName;
- this.centerLatitude = centerLatitude;
- this.centerLongitude = centerLongitude;
- this.radius = radius;
- }
-
- @Override
- public void flatMap(Event in, Collector<Event> out) throws Exception {
- Float latitude =
in.getFieldBySelector(latitudeFieldName).getAsPrimitive().getAsFloat();
- Float longitude =
in.getFieldBySelector(longitudeFieldName).getAsPrimitive().getAsFloat();
-
- Float distance = distFrom(latitude, longitude, centerLatitude,
centerLongitude);
-
- if (distance <= radius) {
- out.collect(in);
- }
- }
-
- public static Float distFrom(float lat1, float lng1, float lat2, float lng2)
{
- double earthRadius = 6371000;
- double dLat = Math.toRadians(lat2-lat1);
- double dLng = Math.toRadians(lng2-lng1);
- double a = Math.sin(dLat/2) * Math.sin(dLat/2) +
- Math.cos(Math.toRadians(lat1)) * Math.cos(Math.toRadians(lat2)) *
- Math.sin(dLng/2) * Math.sin(dLng/2);
- double c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
- return (float) (earthRadius * c);
- }
-
-}
-```
-
-We won't go into details here as this isn't StreamPipes-related code, but in
general the class extracts latitude and longitude fields from the input event
(which is provided as a map data type) and calculates the distance between the
geofence center and these coordinates.
-If the distance is below the given radius, the event is forwarded to the next
operator.
-
-Finally, we need to connect this program to the Flink topology. StreamPipes
automatically adds things like the Kafka consumer and producer, so that you
only need to invoke the actual geofencing processor.
-Open the file `GeofencingProgram` and append the following line inside the
`getApplicationLogic()` method:
-
-```java
-return dataStreams[0].flatMap(new
GeofencingProcessor(params.getLatitudeFieldName(),
params.getLongitudeFieldName(),
- params.getCenterLatitude(), params.getCenterLongitude(),
params.getRadius()));
-```
-
-## Preparing the container
-The final step is to define the deployment type of our new data source. In
this tutorial, we will create a so-called `StandaloneModelSubmitter`.
-This client will start an embedded web server that provides the description of
our data source and automatically submits the program to the registered Apache
Flink cluster.
-
-Go to the class `Init` that extends `StandaloneModelSubmitter` and should look
as follows:
-```java
-package org.streampipes.tutorial.main;
-
-import org.streampipes.container.init.DeclarersSingleton;
-import org.streampipes.container.standalone.init.StandaloneModelSubmitter;
-
-import org.streampipes.tutorial.config.Config;
-import org.streampipes.tutorial.pe.processor.geofencing.GeofencingController;
-
-public class Init extends StandaloneModelSubmitter {
-
- public static void main(String[] args) throws Exception {
- DeclarersSingleton.getInstance()
- .add(new GeofencingController());
-
- new Init().init(Config.INSTANCE);
-
- }
-}
-```
-
-<div class="admonition info">
-<div class="admonition-title">Info</div>
-<p>In the example above, we make use of a class `Config`.
- This class contains both mandatory and additional configuration
parameters required by a pipeline element container.
- These values are stored in the Consul-based key-value store of your
StreamPipes installation.
- The SDK guide contains a detailed manual on managing container
configurations.
-</p>
-</div>
-
-## Starting the container
-<div class="admonition tip">
-<div class="admonition-title">Tip</div>
-<p>By default, the container registers itself using the hostname later used by
the Docker container, leading to a 404 error when you try to access an RDF
description.
- For local development, we provide an environment file in the
``development`` folder. You can add your hostname here, which will override
settings from the Config class.
- For instance, use the IntelliJ ``EnvFile`` plugin to automatically
provide the environment variables upon start.
-</p>
-</div>
-
-
-<div class="admonition tip">
-<div class="admonition-title">Tip</div>
-<p> The default port of all pipeline element containers as defined in the
`Config` file is port 8090.
- If you'd like to run mutliple containers at the same time on your
development machine, change the port in the environment file.
-</p>
-</div>
-
-Now we are ready to start our container!
-
-Execute the main method in the class `Main` we've just created, open a web
browser and navigate to http://localhost:8090 (or the port you have assigned in
the environment file).
-
-You should see something as follows:
-
-<img src="/docs/img/tutorial-processors/pe-overview-flink.PNG" alt="Pipeline
Element Container Overview (Flink)">
-
-Click on the link of the data source to see the RDF description of the
pipeline element.
-
-<img src="/docs/img/tutorial-processors/pe-rdf-geofencing.PNG" alt="Geofencing
RDF description">
-
-The container automatically registers itself in the Consul installation of
StreamPipes.
-To install the just created element, open the StreamPipes UI and follow the
manual provided in the [user guide](user-guide-installation).
-
-## Read more
-
-Congratulations! You've just created your first data processor for StreamPipes.
-There are many more things to explore and data processors can be defined in
much more detail using multiple wrappers.
-Follow our [SDK guide](dev-guide-sdk-guide-processors) to see what's possible!
\ No newline at end of file
diff --git a/documentation/docs/dev-guide-tutorial-sinks.md
b/documentation/docs/dev-guide-tutorial-sinks.md
index e4cf3b5..641b62f 100644
--- a/documentation/docs/dev-guide-tutorial-sinks.md
+++ b/documentation/docs/dev-guide-tutorial-sinks.md
@@ -3,244 +3,3 @@ id: dev-guide-tutorial-sinks
title: Tutorial: Data Sinks
sidebar_label: Tutorial: Data Sinks
---
-
-In this tutorial, we will add a new data sink using the standalone wrapper.
-
-From an architectural point of view, we will create a self-contained service
that includes the description of the data sink and a corresponding
implementation.
-
-## Objective
-
-We are going to create a new data sink that calls an external HTTP endpoint to
forward data to an external service.
-
-For each incoming event, an external service is invoked using an HTTP POST
request. In this example, we'll call an endpoint provided by
[RequestBin](https://requestbin.com/).
-To setup your own endpoint, go to
[https://requestbin.com/](https://requestbin.com/) and click "Create a request
bin". Copy the URL of the newly created endpoint.
-
-
-## Project setup
-
-We'll create a new project using the provided sinks-standalone-jvm Maven
archetype.
-Enter the following command to create a new project based on the StreamPipes
``Sinks-JVM`` archetype:
-
-```
-mvn archetype:generate -DarchetypeGroupId=org.streampipes \
--DarchetypeArtifactId=streampipes-archetype-pe-sinks-jvm
-DarchetypeVersion=0.64.0 \
--DgroupId=org.streampipes.tutorial -DartifactId=sink-tutorial
-DclassNamePrefix=Rest -DpackageName=geofencing
-```
-
-Once you've imported the generated project, the project structure should look
as follows:
-
-<img src="/docs/img/tutorial-sinks/project-structure-sinks.png" alt="Project
Structure Data Sink">
-
-
-<div class="admonition tip">
-<div class="admonition-title">Tip</div>
-<p>Besides the basic project skeleton, the sample project also includes an
example Dockerfile you can use to package your application into a Docker
container.</p>
-</div>
-
-Now you're ready to create your first data sink for StreamPipes!
-
-## Adding data sink requirements
-
-First, we will add a new stream requirement.
-Open the class `RestController` which should look as follows:
-
-```java
-package org.streampipes.tutorial.pe.sink.rest;
-
-import org.streampipes.model.DataSinkType;
-import org.streampipes.model.graph.DataSinkDescription;
-import org.streampipes.model.graph.DataSinkInvocation;
-import org.streampipes.sdk.builder.DataSinkBuilder;
-import org.streampipes.sdk.builder.StreamRequirementsBuilder;
-import org.streampipes.sdk.extractor.DataSinkParameterExtractor;
-import org.streampipes.sdk.helpers.EpRequirements;
-import org.streampipes.sdk.helpers.Labels;
-import org.streampipes.sdk.helpers.SupportedFormats;
-import org.streampipes.sdk.helpers.SupportedProtocols;
-import org.streampipes.wrapper.standalone.ConfiguredEventSink;
-import org.streampipes.wrapper.standalone.declarer.StandaloneEventSinkDeclarer;
-import org.streampipes.sdk.helpers.*;
-import org.streampipes.sdk.utils.Assets;
-
-public class RestController extends
StandaloneEventSinkDeclarer<RestParameters> {
-
- private static final String EXAMPLE_KEY = "example-key";
-
- @Override
- public DataSinkDescription declareModel() {
- return
DataSinkBuilder.create("org.streampipes.tutorial.pe.sink.rest")
-
.category(DataSinkType.NOTIFICATION)
-
.withAssets(Assets.DOCUMENTATION, Assets.ICON)
- .withLocales(Locales.EN)
-
.requiredStream(StreamRequirementsBuilder
- .create()
-
.requiredPropertyWithNaryMapping(EpRequirements.anyProperty(), Labels.withId(
- "fields-to-send"),
PropertyScope.NONE)
- .build())
- .build();
- }
-
- @Override
- public ConfiguredEventSink<RestParameters>
onInvocation(DataSinkInvocation graph, DataSinkParameterExtractor extractor) {
-
- List<String> fieldsToSend =
extractor.mappingPropertyValues("fields-to-send");
-
- RestParameters params = new RestParameters(graph, fieldsToSend);
-
- return new ConfiguredEventSink<>(params, Rest::new);
- }
-
-}
-
-```
-
-In this class, we need to implement two methods: The `declareModel` method is
used to define abstract stream requirements such as event properties that must
be present in any input stream that is later connected to the element using the
StreamPipes UI.
-The second method, `onInvocation` is used to create and deploy program once a
pipeline using this sink is started.
-
-The ``declareModel`` method describes the properties of our data sink:
-* ``category`` defines a category for this sink.
-* ``withAssets`` denotes that we will provide an external documentation file
and an icon, which can be found in the ``resources`` folder
-* ``withLocales`` defines that we will provide an external language file, also
available in the ``resources`` folder
-* ``requiredStream`` defines requirements any input stream connected to this
sink must provide. In this case, we do not have any specific requirements, we
just forward all incoming events to the REST sink. However, we want to let the
user display a list of available fields from the connected input event, where
users can select a subset. This is defined by defining a Mapping from the empty
requirement. This will later on render a selection dialog in the pipeline
editor.
-
-The ``onInvocation`` method is called when a pipeline containing the sink is
started. Once a pipeline is started, we would like to extract user-defined
parameters.
-In this example, we simply extract the fields selected by users that should be
forwarded to the REST sink. Finally, we return a new configured event sink
containing the parameters.
-
-## Pipeline element invocation
-
-Once users start a pipeline that uses our geofencing component, the
_getRuntime_ method in our class is called. The class `DataSinkInovcation`
includes a graph that contains information on the configuration parameters a
users has selected in the pipeline editor and information on the acutal streams
that are connected to the pipeline element.
-
-Before we explain in more detail how to extract these values from the
processor invocation, we need to adapt a little helper class.
-Open the file ```RestParameters``` and modify it as follows:
-
-```java
-public class RestParameters extends EventSinkBindingParams {
-
- private List<String> fieldsToSend;
-
- public RestParameters(DataSinkInvocation graph, List<String> fieldsToSend) {
- super(graph);
- this.fieldsToSend = fieldsToSend;
- }
-
- public List<String> getFieldsToSend() {
- return fieldsToSend;
- }
-}
-```
-
-This file will later provide information on the configured pipeline element.
-
-## Adding an implementation
-
-Now open the class ``Rest`` to add the proper implementation (i.e., the Rest
call executed for every incoming event).
-
-Our final class should look as follows:
-
-```java
-private static Logger LOG =
LoggerFactory.getLogger(Rest.class.getCanonicalName());
-
- private static final String REST_ENDPOINT_URI = YOUR_REQUEST_BIN_URL;
- private List<String> fieldsToSend;
- private SpDataFormatDefinition dataFormatDefinition;
-
- public Rest() {
- this.dataFormatDefinition = new JsonDataFormatDefinition();
- }
-
- @Override
- public void onInvocation(RestParameters parameters, EventSinkRuntimeContext
runtimeContext) {
- this.fieldsToSend = parameters.getFieldsToSend();
- }
-
- @Override
- public void onEvent(Event event) {
- Map<String, Object> outEventMap = event.getSubset(fieldsToSend).getRaw();
- try {
- String json = new String(dataFormatDefinition.fromMap(outEventMap));
- Request.Post(REST_ENDPOINT_URI).body(new StringEntity(json,
Charsets.UTF_8)).execute();
- } catch (SpRuntimeException e) {
- LOG.error("Could not parse incoming event");
- } catch (IOException e) {
- LOG.error("Could not reach endpoint at {}", REST_ENDPOINT_URI);
- }
- }
-
- @Override
- public void onDetach() {
-
- }
-```
-The only class variable you need to change right now is the REST_ENDPOINT_URL.
Change this url to the URL provided by your request bin.
-We'll ignore the other class variables and the constructor for now. Important
are three methods, ``onInvocation``, ``onEvent`` and ``onDetach``.
-
-The ``onInvocation`` method is called once a pipeline containing our REST data
sink is started. The ``onEvent`` method is called for each incoming event.
Finally, ``onDetach`` is called when a pipeline is stopped.
-
-In the ``onInvocation`` method, we can extract the selected fields to be
forwarded to the REST endpoint.
-In the ``ònEvent`` method, we use a helper method to get a subset of the
incoming event.
-Finally, we convert the resulting ``Map`` to a JSON string and call the
endpoint.
-
-
-## Preparing the container
-The final step is to define the deployment type of our new data source. In
this tutorial, we will create a so-called `StandaloneModelSubmitter`.
-This client will start an embedded web server that provides the description of
our data source and automatically starts the program in an embedded container.
-
-Go to the class `Init` that extends `StandaloneModelSubmitter` and should look
as follows:
-```java
-
-public static void main(String[] args) throws Exception {
- DeclarersSingleton.getInstance()
- .add(new RestController());
-
- DeclarersSingleton.getInstance().setPort(Config.INSTANCE.getPort());
- DeclarersSingleton.getInstance().setHostName(Config.INSTANCE.getHost());
-
- DeclarersSingleton.getInstance().registerDataFormats(new
JsonDataFormatFactory(),
- new CborDataFormatFactory(),
- new SmileDataFormatFactory(),
- new FstDataFormatFactory());
-
- DeclarersSingleton.getInstance().registerProtocols(new
SpKafkaProtocolFactory(),
- new SpJmsProtocolFactory());
-
- new Init().init(Config.INSTANCE);
-```
-
-<div class="admonition info">
-<div class="admonition-title">Info</div>
-<p>In the example above, we make use of a class `Config`.
- This class contains both mandatory and additional configuration
parameters required by a pipeline element container.
- These values are stored in the Consul-based key-value store of your
StreamPipes installation.
- The SDK guide contains a detailed manual on managing container
configurations.
-</p>
-</div>
-
-## Starting the container
-<div class="admonition tip">
-<div class="admonition-title">Tip</div>
-<p>By default, the container registers itself using the hostname later used by
the Docker container, leading to a 404 error when you try to access an RDF
description.
- For local development, we provide an environment file in the
``development`` folder. You can add your hostname here, which will override
settings from the Config class.
- For instance, use the IntelliJ ``EnvFile`` plugin to automatically
provide the environment variables upon start.
-</p>
-</div>
-
-
-<div class="admonition tip">
-<div class="admonition-title">Tip</div>
-<p> The default port of all pipeline element containers as defined in the
`Config` file is port 8090.
- If you'd like to run mutliple containers at the same time on your
development machine, change the port in the environment file.
-</p>
-</div>
-
-Now we are ready to start our container!
-
-Execute the main method in the class `Main` we've just created.
-
-The container automatically registers itself in the Consul installation of
StreamPipes.
-To install the just created element, open the StreamPipes UI and follow the
manual provided in the [user guide](user-guide-installation).
-
-## Read more
-
-Congratulations! You've just created your first data sink for StreamPipes.
-There are many more things to explore and data sinks can be defined in much
more detail using multiple wrappers.
-Follow our [SDK guide](dev-guide-sdk-guide-sinks) to see what's possible!
\ No newline at end of file
diff --git a/documentation/docs/dev-guide-tutorial-sources.md
b/documentation/docs/dev-guide-tutorial-sources.md
index 27a3081..7257aa3 100644
--- a/documentation/docs/dev-guide-tutorial-sources.md
+++ b/documentation/docs/dev-guide-tutorial-sources.md
@@ -3,280 +3,3 @@ id: dev-guide-tutorial-sources
title: Tutorial: Data Sources
sidebar_label: Tutorial: Data Sources
---
-
-In this tutorial, we will add a new data source consisting of a single data
stream. The source will be provided as a standalone component (i.e., the
description will be accessible through an integrated web server).
-
-## Objective
-
-We are going to create a new data stream that is produced by a GPS sensor
installed in a delivery vehicle.
-The sensor produces a continuous stream of events that contain the current
timestamp, the current lat/lng position of the vehicle and the plate number of
the vehicle.
-Events are published in a JSON format as follows:
-```json
-{
- "timestamp" : 145838399,
- "latitude" : 37.04,
- "longitude" : 17.04,
- "plateNumber" : "KA-AB 123"
-}
-```
-
-These events are published to a Kafka broker using the topic
`org.streampipes.tutorial.vehicle`.
-
-In the following section, we show how to describe this stream in a form that
allows you to import and use it in StreamPipes.
-
-## Project setup
-
-Instead of creating a new project from scratch, we recommend to use the Maven
archetype to create a new project skeleton.
-Enter the following command in a command line of your choice (Apache Maven
needs to be installed):
-
-```
-mvn archetype:generate \
--DarchetypeGroupId=org.streampipes
-DarchetypeArtifactId=streampipes-archetype-pe-sources \
--DarchetypeVersion=0.65.0 -DgroupId=my.groupId \
--DartifactId=my-source -DclassNamePrefix=MySource -DpackageName=mypackagename
-```
-
-Configure the variables ``artifactId`` (which will be the Maven artifactId),
``classNamePrefix`` (which will be the class name of your data stream) and
``packageName``.
-
-For this tutorial, use ``Vehicle`` as ``classNamePrefix``.
-
-Your project will look as follows:
-
-<img src="/docs/img/tutorial-sources/project-structure.PNG" alt="Project
Structure">
-
-That's it, go to the next section to learn how to create your first data
stream!
-
-<div class="admonition tip">
-<div class="admonition-title">Tip</div>
-<p>Besides the basic project skeleton, the sample project also includes an
example Dockerfile you can use to package your application into a Docker
container.
-</p>
-</div>
-
-## Adding a data stream description
-
-Now we will add a new data stream definition.
-First, open the class `VehicleStream` which should look as follows:
-
-```java
-
-package my.groupId.pe.mypackagename;
-
-import org.streampipes.model.SpDataStream;
-import org.streampipes.model.graph.DataSourceDescription;
-import org.streampipes.sdk.builder.DataStreamBuilder;
-import org.streampipes.sdk.helpers.EpProperties;
-import org.streampipes.sdk.helpers.Formats;
-import org.streampipes.sdk.helpers.Protocols;
-import org.streampipes.sources.AbstractAdapterIncludedStream;
-
-
-public class MySourceStream extends AbstractAdapterIncludedStream {
-
- @Override
- public SpDataStream declareModel(DataSourceDescription sep) {
- return DataStreamBuilder.create("my.groupId-mypackagename", "MySource", "")
- .property(EpProperties.timestampProperty("timestamp"))
-
- // configure your stream here
-
- .format(Formats.jsonFormat())
- .protocol(Protocols.kafka("localhost", 9092,
"TOPIC_SHOULD_BE_CHANGED"))
- .build();
- }
-
- @Override
- public void executeStream() {
-
- }
-}
-```
-
-This class extends the class ``AbstractAdapterIncludedStream``, which
indicates that this source continuously produces data (configured in the
``executeStream()`` method.
-In contrast, the class `AbstractAlreadyExistingStream` indicates that we only
want to describe an already existing stream (e.g., a stream that already sends
data to an existing Kafka broker).
-
-Next, we will add the definition of the data stream. Add the following code
inside of the `declareModel` method:
-```java
-return DataStreamBuilder.create("org.streampipes.tutorial.vehicle.position",
"Vehicle Position", "An event stream " +
- "that produces current vehicle positions")
-```
-
-This line creates a new instance of the SDK's `DataStreamBuilder` by providing
three basic parameters:
-The first parameter must be a unique identifier of your data stream.
-The second and third parameters indicate a label and a description of your
stream.
-These values will later be used in the StreamPipes UI to display stream
details in a human-readable manner.
-
-Next, we will add the properties as stated above to the stream definition by
adding the following lines:
-```java
-.property(EpProperties.timestampProperty("timestamp"))
-.property(EpProperties.stringEp(Labels.from("plate-number", "Plate Number",
"Denotes the plate number of the vehicle"), "plateNumber",
"http://my.company/plateNumber";))
-.property(EpProperties.doubleEp(Labels.from("latitude", "Latitude", "Denotes
the latitude value of the vehicle's position"), "latitude", Geo.lat))
-.property(EpProperties.doubleEp(Labels.from("longitude", "Longitude", "Denotes
the longitude value of the vehicle's position"), "longitude", Geo.lng))
-```
-These four _event properties_ compose our _event schema_. An event property
must, at least, provide the following attributes:
-
-* **Runtime Name**. The runtime name indicates the key of the property at
runtime, e.g., if our JSON message contains a structure such as `{"plateNumber"
: "KA-F 123"}`, the runtime name must be `plateNumber`.
-* **Runtime Type**. An event property must have a primitive type (we will
later see how to model more complex properties such as lists and nested
properties).
-The type must be an instance of `XMLSchema` primitives, however, the SDK
provides convenience methods to provide the property type.
-* **Domain Property**. The domain property indicates the semantics of the
event property. For instance, the `latitude` property is linked to the
`http://www.w3.org/2003/01/geo/wgs84_pos#lat` property of the WGS84 vocabulary.
-The domain property should be an URI as part of an existing or domain-specific
vocabulary. The SDK provides convenience methods for popuplar vocabularies
(e.g., Schema.org, Dolce or WGS84).
-
-In order to complete the minimum required specification of an event stream, we
need to provide information on the transport format and protocol of the data
stream at runtime.
-
-This can be achieved by extending the builder with the respective properties
(which should already have been auto-generated):
-```java
-.format(Formats.jsonFormat())
-.protocol(Protocols.kafka("localhost", 9092, "TOPIC_SHOULD_BE_CHANGED"))
-.build();
-```
-
-Set ``org.streampipes.tutorial.vehicle`` as your new topic by replacing the
term ``TOPIC_SHOULD_BE_CHANGED`.
-
-In this example, we defined that the data stream consists of events in a JSON
format and that Kafka is used as a message broker to transmit events.
-The last build() method call triggers the construction of the RDF-based data
stream definition.
-
-That's it! In the next section, we will connect the data stream to a source
and inspect the generated RDF description.
-
-## Creating some dummy data
-
-Let's assume our stream should produce some random values that are sent to
StreamPipes. We'll add a very simple data simulator to the ``executeStream``
method as follows:
-
-```java
-@Override
- public void executeStream() {
-
- SpKafkaProducer producer = new SpKafkaProducer("localhost:9092",
"TOPIC_SHOULD_BE_CHANGED");
- Random random = new Random();
- Runnable runnable = new Runnable() {
- @Override
- public void run() {
- for (;;) {
- JsonObject jsonObject = new JsonObject();
- jsonObject.addProperty("timestamp", System.currentTimeMillis());
- jsonObject.addProperty("plateNumber", "KA-FZ 1");
- jsonObject.addProperty("latitude", random.nextDouble());
- jsonObject.addProperty("longitude", random.nextDouble());
-
- producer.publish(jsonObject.toString());
-
- try {
- Thread.sleep(1000);
- } catch (InterruptedException e) {
- e.printStackTrace();
- }
-
- }
- }
- };
-
- new Thread(runnable).start();
- }
-```
-
-Change the topic and the URL of your Kafka broker (as stated in the
controller).
-
-## Adding a source description
-
-A data source can be seen like a container for a set of data streams. Usually,
a data source includes events that are logically or physically connected.
-For instance, in our example we would add other streams produced by vehicle
sensors (such as fuel consumption) to the same data source description.
-
-Open the class `DataSource` which should look as follows:
-```java
-
-package my.groupId.pe.mypackagename;
-
-import org.streampipes.container.declarer.DataStreamDeclarer;
-import org.streampipes.container.declarer.SemanticEventProducerDeclarer;
-import org.streampipes.model.graph.DataSourceDescription;
-import org.streampipes.sdk.builder.DataSourceBuilder;
-
-import java.util.Arrays;
-import java.util.List;
-
-
-public class DataSource implements SemanticEventProducerDeclarer {
-
- public DataSourceDescription declareModel() {
- return DataSourceBuilder.create("my.groupId.mypackagename.source",
"MySource " +
- "Source", "")
- .build();
- }
-
- public List<DataStreamDeclarer> getEventStreams() {
- return Arrays.asList(new MySourceStream());
- }
-}
-```
-First, we need to define the source. Similar to data streams, a source
consists of an id, a human-readable name and a description.
-Replace the content defined in the `declareModel` method with the following
code:
-```java
-return DataSourceBuilder.create("org.streampipes.tutorial.source.vehicle",
"Vehicle Source", "A data source that " +
- "holds event streams produced by vehicles.")
- .build();
-```
-
-## Preparing the container
-
-The final step is to define the deployment type of our new data source. In
this tutorial, we will create a so-called `StandaloneModelSubmitter`.
-This client will start an embedded web server that provides the description of
our data source.
-
-Go to the class `Init` that implements `StandaloneModelSubmitter`, which
should look as follows:
-```java
-package my.groupId.main;
-
-import org.streampipes.container.init.DeclarersSingleton;
-import org.streampipes.container.standalone.init.StandaloneModelSubmitter;
-import my.groupId.config.Config;
-import my.groupId.pe.mypackagename.DataSource;
-
-public class Init extends StandaloneModelSubmitter {
-
- public static void main(String[] args) throws Exception {
- DeclarersSingleton.getInstance()
- .add(new DataSource());
-
- new Init().init(Config.INSTANCE);
-
- }
-}
-```
-This code adds the `VehicleSource`. Finally, the `init` method is called
-which triggers the generation of the corresponding RDF description and startup
of the web server.
-
-<div class="admonition info">
-<div class="admonition-title">Info</div>
-<p>In the example above, we make use of a class `Config`.
- This class contains both mandatory and additional configuration
parameters required by a pipeline element container.
- These values are stored in the Consul-based key-value store of your
StreamPipes installation.
- The SDK guide contains a detailed manual on managing container
configurations.</p>
-</div>
-
-## Starting the container
-
-<div class="admonition tip">
-<div class="admonition-title">Tip</div>
-<p>By default, the container registers itself using the hostname later used by
the Docker container, leading to a 404 error when you try to access an RDF
description.
- For local development, we provide an environment file in the
``development`` folder. You can add your hostname here, which will override
settings from the Config class.
- For instance, use the IntelliJ ``EnvFile`` plugin to automatically
provide the environment variables upon start.
-</p>
-</div>
-
-Now we are ready to start our first container!
-
-Execute the main method in the class `Main` we've just created, open a web
browser and navigate to http://localhost:8090, or change the port according to
the value of the ``SP_PORT`` variable in the env file.
-
-You should see something as follows:
-
-<img src="/docs/img/tutorial-sources/pe-overview.PNG" alt="Pipeline Element
Container Overview">
-
-Click on the link of the data source to see the RDF description of the
pipeline element.
-
-<img src="/docs/img/tutorial-sources/pe-rdf.PNG" alt="Pipeline Element RDF
description">
-
-The container automatically registers itself in the Consul installation of
StreamPipes.
-To install the just created element, open the StreamPipes UI and follow the
manual provided in the [user guide](user-guide-introduction).
-
-## Read more
-
-Congratulations! You've just created your first pipeline element for
StreamPipes.
-There are many more things to explore and data sources can be defined in much
more detail.
-Follow our [SDK guide](dev-guide-source-sdk) to see what's possible!
\ No newline at end of file
diff --git a/documentation/website/i18n/en.json
b/documentation/website/i18n/en.json
index d315a4f..e7972c6 100644
--- a/documentation/website/i18n/en.json
+++ b/documentation/website/i18n/en.json
@@ -38,7 +38,7 @@
"sidebar_label": "Configurations"
},
"use-connect": {
- "title": "Connecting Data",
+ "title": "StreamPipes Connect",
"sidebar_label": "StreamPipes Connect"
},
"use-dashboard": {
@@ -65,18 +65,54 @@
"title": "Kubernetes Deployment",
"sidebar_label": "Kubernetes Deployment"
},
+ "deploy-use-ssl": {
+ "title": "Use SSL",
+ "sidebar_label": "Use SSL"
+ },
"extend-archetypes": {
"title": "Maven Archetypes",
"sidebar_label": "Maven Archetypes"
},
+ "extend-sdk-event-model": {
+ "title": "SDK Guide: Event Model",
+ "sidebar_label": "SDK: Event Model"
+ },
+ "extend-sdk-output-strategies": {
+ "title": "SDK Guide: Output Strategies",
+ "sidebar_label": "SDK: Output Strategies"
+ },
+ "extend-sdk-static-properties": {
+ "title": "SDK Guide: Static Properties",
+ "sidebar_label": "SDK: Static Properties"
+ },
+ "extend-sdk-stream-requirements": {
+ "title": "SDK Guide: Stream Requirements",
+ "sidebar_label": "SDK: Stream Requirements"
+ },
"extend-setup": {
"title": "Development Setup",
"sidebar_label": "Development Setup"
},
+ "extend-tutorial-data-processors": {
+ "title": "Tutorial: Data Processors",
+ "sidebar_label": "Tutorial: Data Processors"
+ },
+ "extend-tutorial-data-sinks": {
+ "title": "Tutorial: Data Sinks",
+ "sidebar_label": "Tutorial: Data Sinks"
+ },
+ "extend-tutorial-data-sources": {
+ "title": "Tutorial: Data Sources",
+ "sidebar_label": "Tutorial: Data Sources"
+ },
"technicals-architecture": {
"title": "Architecture",
"sidebar_label": "Architecture"
},
+ "technicals-configuration": {
+ "title": "Configuration",
+ "sidebar_label": "Configuration"
+ },
"technicals-messaging": {
"title": "Messaging",
"sidebar_label": "Messaging"
@@ -106,20 +142,16 @@
"sidebar_label": "Start Developing"
},
"dev-guide-architecture": {
- "title": "Architecture",
- "sidebar_label": "Architecture"
+ "title": "dev-guide-architecture"
},
"dev-guide-configuration": {
- "title": "Configuration",
- "sidebar_label": "Configuration"
+ "title": "dev-guide-configuration"
},
- "dev-guide-development-environment": {
- "title": "Development Environment",
- "sidebar_label": "Development Environment"
+ "dev-guide-environment": {
+ "title": "dev-guide-environment"
},
"dev-guide-event-model": {
- "title": "SDK Guide: Event Model",
- "sidebar_label": "Event Model"
+ "title": "dev-guide-event-model"
},
"dev-guide-introduction": {
"title": "Introduction",
@@ -130,8 +162,7 @@
"sidebar_label": "Migration Guide"
},
"dev-guide-output-strategies": {
- "title": "SDK Guide: Output Strategies",
- "sidebar_label": "Output Strategies"
+ "title": "dev-guide-output-strategies"
},
"dev-guide-sdk-guide-processors": {
"title": "SDK Guide: Data Processors",
@@ -146,28 +177,22 @@
"sidebar_label": "SDK Guide: Data Sources"
},
"dev-guide-ssl": {
- "title": "Use SSL",
- "sidebar_label": "Use SSL"
+ "title": "dev-guide-ssl"
},
"dev-guide-static-properties": {
- "title": "SDK Guide: Static Properties",
- "sidebar_label": "Static Properties"
+ "title": "dev-guide-static-properties"
},
"dev-guide-stream-requirements": {
- "title": "SDK Guide: Stream Requirements",
- "sidebar_label": "Stream Requirements"
+ "title": "dev-guide-stream-requirements"
},
"dev-guide-tutorial-processors": {
- "title": "Tutorial: Data Processors",
- "sidebar_label": "Tutorial: Data Processors"
+ "title": "dev-guide-tutorial-processors"
},
"dev-guide-tutorial-sinks": {
- "title": "Tutorial: Data Sinks",
- "sidebar_label": "Tutorial: Data Sinks"
+ "title": "dev-guide-tutorial-sinks"
},
"dev-guide-tutorial-sources": {
- "title": "Tutorial: Data Sources",
- "sidebar_label": "Tutorial: Data Sources"
+ "title": "dev-guide-tutorial-sources"
},
"faq-common-problems": {
"title": "Common Problems",
diff --git a/documentation/website/sidebars.json
b/documentation/website/sidebars.json
index 625a0f6..ec4bd74 100644
--- a/documentation/website/sidebars.json
+++ b/documentation/website/sidebars.json
@@ -157,17 +157,26 @@
],
"⚡ Deploy StreamPipes": [
"deploy-docker",
- "deploy-kubernetes"
+ "deploy-kubernetes",
+ "deploy-use-ssl"
],
"\uD83D\uDCBB Extend StreamPipes": [
"extend-setup",
- "extend-archetypes"
+ "extend-archetypes",
+ "extend-tutorial-data-sources",
+ "extend-tutorial-data-processors",
+ "extend-tutorial-data-sinks",
+ "extend-sdk-event-model",
+ "extend-sdk-stream-requirements",
+ "extend-sdk-static-properties",
+ "extend-sdk-output-strategies"
],
"\uD83D\uDD27 Technicals": [
"technicals-architecture",
"technicals-user-guidance",
"technicals-runtime-wrappers",
- "technicals-messaging"
+ "technicals-messaging",
+ "technical-configuration"
],
"\uD83D\uDEA6 Monitoring/Debugging": [
"debugging-monitoring",
