This is an automated email from the ASF dual-hosted git repository. riemer pushed a commit to branch extend-docs-sections in repository https://gitbox.apache.org/repos/asf/streampipes-website.git
commit a9b1082f6f8e8c56e3f646d1299b825a6a888933 Author: Dominik Riemer <[email protected]> AuthorDate: Wed Nov 15 11:47:32 2023 +0100 Add documentation pages for env variables and UI customization --- docs/05_deploy-choosing-the-right-flavor.md | 47 ++++ docs/05_deploy-environment-variables.md | 87 ++++++- docs/05_deploy-kubernetes.md | 252 ++++++++++++++++++++- docs/06_extend-customize-ui.md | 226 ++++++++++++++++++ docs/07_technicals-messaging.md | 23 ++ docs/07_technicals-runtime-wrappers.md | 32 ++- website-v2/sidebars.json | 10 +- .../version-0.92.0-sidebars.json | 1 - 8 files changed, 669 insertions(+), 9 deletions(-) diff --git a/docs/05_deploy-choosing-the-right-flavor.md b/docs/05_deploy-choosing-the-right-flavor.md new file mode 100644 index 000000000..a140bf46e --- /dev/null +++ b/docs/05_deploy-choosing-the-right-flavor.md @@ -0,0 +1,47 @@ +--- +id: choosing-the-right-flavor +title: Choosing the right flavor +sidebar_label: Service selection options +--- + + +## Introduction + +StreamPipes comes with many different options to customize a deployment. This section introduces the various options you can choose from when installing StreamPipes. + +You can choose between various **deployment modes**, choose from two different core packages and several extension packages, wich are described below. + +## Deployment Mode + +For the deployment model, you choose between a standard multi-container `Docker-Compose` installation and the `Kubernetes` installation. +we provide several `Docker-Compose` files for the various options shown here and a `helm chart`. +See [Docker Deployment](05_deploy-docker.md) and [Kubernetes Deployment](05_deploy-kubernetes.md) for more details. + +### Running StreamPipes in a non-containerized environment + +Of course, it is also possible to launch StreamPipes in a non-containerized environment. +You will need to build your own executable binaries by running `mvn package`. +In addition, it is required to install the required 3rd party services (see [Architecture](07_technicals-architecture.md)) and configure the environment variables as described in [Environment Variables](05_deploy-environment-variables.md). + +## Core Service + +We provide two different pre-packaged versions of core services. The default `streampipes-service-core` is a packaged JAR file which includes client libraries for the various messaging systems StreamPipes supports at the cost of a larger file size. +In case you plan to run StreamPipes on less resource-intensive hardware, we recommend to switch to the `streampipes-service-core-minimal` package, which only includes support for MQTT and NATS, but has a smaller file size and slightly improved startup performance. + +## Extension Services + +Similar to the core, we provide several pre-packaged extension services which differ mainly by their file size, number of supported adapters and pipeline elements and messaging systems. + +The following packages exist: + +* `streampipes-extensions-all-jvm` is the largest package and includes all official StreamPipes adapters and pipeline elements. It also includes support for all messaging systems Streampipes currently supports. +* `streampipes-extensions-all-iiot` is a subset of the aforementioned package and excludes adapters and pipeline elements which are often not relevant for IIoT use cases. For instance, the package excludes text mining-related pipeline elements. +* `streampipes-extensions-iiot-minimal` is a subset of the aforementioned package and includes only support for the lightweight messaging systems MQTT and NATS. + +Generally said, in cases where you plan to deploy StreamPipes on a resource-limited edge device, we recommend a combination of the `streampipes-service-core-minimal` and `streampipes-extensions-iiot-minimal` package. This could, for instance, be a device with less than 4GB memory. +In other cases, it depends on the use case and if you need all adapters and pipeline elements or are ok with the IIoT-related extensions. + +## Messaging System + +StreamPipes can be configured to use different messaging systems for exchanging events between adapters and pipeline elements. +The section [Messaging](07_technicals-messaging.md) includes detailed information on the configuration of messaging systems. diff --git a/docs/05_deploy-environment-variables.md b/docs/05_deploy-environment-variables.md index 26b925f62..c4066fb70 100644 --- a/docs/05_deploy-environment-variables.md +++ b/docs/05_deploy-environment-variables.md @@ -1,5 +1,88 @@ --- - +id: deploy-environment-variables +title: Environment Variables +sidebar_label: Environment Variables --- ---- \ No newline at end of file +## Introduction + +A StreamPipes installation can be configured in many ways by providing environment variables. +The following lists describe available environment variables along with a description. + +## StreamPipes Core Service + +### Internal + +| Env Variable Name | Default Value | Description | +|--------------------------------|---------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------| +| SP_DEBUG | false | Should only be set for local development to reroute traffic to localhost | +| SP_INITIAL_ADMIN_EMAIL | [email protected] | Installation-time variable for defining the default user name | +| SP_INITIAL_ADMIN_PASSWORD | admin | Installation-time variable for defining the default user password | +| SP_INITIAL_SERVICE_USER | sp-service-client | Installation-time variable for defining the initial service user (must be same to the configured user in the extension service) | +| SP_INITIAL_SERVICE_USER_SECRET | my-apache-streampipes-secret-key-change-me | Installation-time variable for defining the initial service secret (minimum 35 chars) | +| SP_JWT_SECRET | Empty for Docker, Auto-generated for K8s | JWT secret, base64-encoded, minimum 256 bits | +| SP_JWT_SIGNING_MODE | HMAC | HMAC or RSA, RSA can be used to authenticate Core-Extensions communication | +| SP_JWT_PRIVATE_KEY_LOC | Empty | Required id SP_JWT_SIGNING_MODE=RSA, path to the private key, can be generated in the UI (Settings->Security->Generate Key Pair) | +| SP_ENCRYPTION_PASSCODE | eGgemyGBoILAu3xckolp for Docker, Auto-generated for K8s | Encryption passcode for `SecretStaticProperties` | +| SP_PRIORITIZED_PROTOCOL | kafka | Messaging layer for data exchange between extensions | + + +### Third-party services + +| Env Variable Name | Default Value | Description | +|------------------------|---------------|---------------------------------------------------------------------------| +| SP_COUCHDB_HOST | couchdb | The hostname or IP of the CouchDB database | +| SP_COUCHDB_PROTOCOL | http | The protocol (http or https) of the CouchDB database | +| SP_COUCHDB_PORT | 5984 | The port of the CouchDB database | +| SP_COUCHDB_USER | admin | The user of the CouchDB database (must have permissions to add databases) | +| SP_COUCHDB_PASSWORD | admin | The password of the CouchDB user | +| SP_TS_STORAGE_HOST | influxdb | The hostname of the timeseries storage (currently InfluxDB) | +| SP_TS_STORAGE_PORT | 8086 | The port of the timeseries storage | +| SP_TS_STORAGE_PROTOCOL | http | The protocol of the timeseries storage (http or https) | +| SP_TS_STORAGE_BUCKET | sp | The InfluxDB storage bucket name | +| SP_TS_STORAGE_ORG | sp | The InfluxDB storage org | +| SP_TS_STORAGE_TOKEN | sp-admin | The InfluxDB storage token | + +The InfluxDB itself can be configured by providing the variables `DOCKER_INFLUXDB_INIT_PASSWORD` and `DOCKER_INFLUXDB_INIT_ADMIN_TOKEN`. See the `docker-compose` file for details. + +## StreamPipes Extensions Service + +### Internal + +| Env Variable Name | Default Value | Description | +|--------------------------------|--------------------------------------------|--------------------------------------------------------------------------------------------------------------------| +| SP_CLIENT_USER | Empty | Service account for communication with Core | +| SP_CLIENT_SECRET | Empty | Service secret for communication with Core | +| SP_EXT_AUTH_MODE | sp-service-client | When set to AUTH: all interfaces are only accessible with authentication (requires SP_JET_PRIVATE_KEY_LOC in Core) | +| SP_JWT_PUBLIC_KEY_LOC | my-apache-streampipes-secret-key-change-me | Path to the public key of the corresponding SP_JWT_PRIVATE_KEY defined in Core | + +### Third-party services + +The following variables are only required for extensions which require access to the internal time-series storage (the `Data Lake Sink`). + +| Env Variable Name | Default Value | Description | +|------------------------|---------------|---------------------------------------------------------------------------| +| SP_TS_STORAGE_HOST | influxdb | The hostname of the timeseries storage (currently InfluxDB) | +| SP_TS_STORAGE_PORT | 8086 | The port of the timeseries storage | +| SP_TS_STORAGE_PROTOCOL | http | The protocol of the timeseries storage (http or https) | +| SP_TS_STORAGE_BUCKET | sp | The InfluxDB storage bucket name | +| SP_TS_STORAGE_ORG | sp | The InfluxDB storage org | +| SP_TS_STORAGE_TOKEN | sp-admin | The InfluxDB storage token | + + +## Recommended variables + +For a standard deployment, it is recommended to customize the following variables: + +* Initiales Admin-Passwort (SP_INITIAL_ADMIN_PASSWORD, Core) +* Initiales Client Secret (SP_INITIAL_SERVICE_USER_SECRET, Core) +* Client Secret Extensions (SP_CLIENT_USER, Extensions) +* Encryption Passcode (SP_ENCRYPTION_PASSCODE, Core) +* CouchDB-Password (SP_COUCHDB_PASSWORD, Core + Extensions + CouchDB) +* InfluxDB Storage Password (DOCKER_INFLUXDB_INIT_PASSWORD, InfluxDB) +* InfluxDB Storage Token (SP_TS_STORAGE_TOKEN (Core, Extensions) + * DOCKER_INFLUXDB_INIT_ADMIN_TOKEN (InfluxDB service) + +## Auto-generation of variables in K8s setups + +See the [Kubernetes Guide](05_deploy-kubernetes.md) for an overview of auto-generated variables. diff --git a/docs/05_deploy-kubernetes.md b/docs/05_deploy-kubernetes.md index 327a5685d..478e1a3c9 100644 --- a/docs/05_deploy-kubernetes.md +++ b/docs/05_deploy-kubernetes.md @@ -18,7 +18,7 @@ helm install streampipes ./ ``` After a while, all containers should successfully started, indicated by the `Running` status. -The `values.yaml` file contains several configuration options to customize your StreamPipes installation. See the file for all configuration options. +The `values.yaml` file contains several configuration options to customize your StreamPipes installation. See the section below for all configuration options. ## Ingress @@ -28,6 +28,256 @@ The helm chart provides several options to configure an Ingress or to define an You can override the `storageClassName` variable to configure StreamPipes for dynamic volume provisioning. +## Parameters + +Here is an overview of the supported parameters to configure StreamPipes. + +### Common parameters + +| Parameter Name | Description | Value | +|--------------------------------------------------|---------------------------------------------------------|-----------------------------------------| +| deployment | Deployment type (lite or full) | lite | +| preferredBroker | Preferred broker for deployment | "nats" | +| monitoringSystem | Enable monitoring system (true/false) | false | +| pullPolicy | Image pull policy | "Always" | +| restartPolicy | Restart policy for the container | Always | +| persistentVolumeReclaimPolicy | Reclaim policy for persistent volumes | "Delete" | +| persistentVolumeAccessModes | Access mode for persistent volumes | "ReadWriteOnce" | +| initialDelaySeconds | Initial delay for liveness and readiness probes | 60 | +| periodSeconds | Interval between liveness and readiness probes | 30 | +| failureThreshold | Number of consecutive failures for readiness probes | 30 | +| hostPath | Host path for the application | "" | + +### StreamPipes common parameters + +| Parameter Name | Description | Value | +|-------------------------------------------------|---------------------------------------------------------|------------------------------------------| +| streampipes.version | StreamPipes version | "0.93.0-SNAPSHOT" | +| streampipes.registry | StreamPipes registry URL | "apachestreampipes" | +| streampipes.auth.secretName | The secret name for storing secrets | "sp-secrets" | +| streampipes.auth.users.admin.user | The initial admin user | "[email protected]" | +| streampipes.auth.users.admin.password | The initial admin password (leave empty for autogen) | "admin" | +| streampipes.auth.users.service.user | The initial service account user | "sp-service-client" | +| streampipes.auth.users.service.secret | The initial service account secret | empty (auto-generated) | +| streampipes.auth.encryption.passcode | Passcode for value encryption | empty (auto-generated) | +| streampipes.core.appName | StreamPipes backend application name | "backend" | +| streampipes.core.port | StreamPipes backend port | 8030 | +| streampipes.core.persistence.storageClassName | Storage class name for backend PVs | "hostpath" | +| streampipes.core.persistence.storageSize | Size of the backend PV | "1Gi" | +| streampipes.core.persistence.claimName | Name of the backend PersistentVolumeClaim | "backend-pvc" | +| streampipes.core.persistence.pvName | Name of the backend PersistentVolume | "backend-pv" | +| streampipes.core.service.name | Name of the backend service | "backend" | +| streampipes.core.service.port | TargetPort of the StreamPipes backend service | 8030 | +| streampipes.ui.appName | StreamPipes UI application name | "ui" | +| streampipes.ui.resolverActive | Flag for enabling DNS resolver for Nginx proxy | true | +| streampipes.ui.port | StreamPipes UI port | 8088 | +| streampipes.ui.resolver | DNS resolver for Nginx proxy | "kube-dns.kube-system.svc.cluster.local" | +| streampipes.ui.service.name | Name of the UI service | "ui" | +| streampipes.ui.service.type | Type of the UI service | "ClusterIP" | +| streampipes.ui.service.nodePort | Node port for the UI service | 8088 | +| streampipes.ui.service.port | TargetPort of the StreamPipes UI service | 8088 | +| streampipes.ingress.active | Flag for enabling Ingress for StreamPipes | false | +| streampipes.ingress.annotations | Annotations for Ingress | {} | +| streampipes.ingress.host | Hostname for Ingress | "" | +| streampipes.ingressroute.active | Flag for enabling IngressRoute for StreamPipes | true | +| streampipes.ingressroute.annotations | Annotations for IngressRoute | {} | +| streampipes.ingressroute.entryPoints | Entry points for IngressRoute | ["web", "websecure"] | +| streampipes.ingressroute.host | Hostname for IngressRoute | "" | +| streampipes.ingressroute.certResolverActive | Flag for enabling certificate resolver for IngressRoute | true | +| streampipes.ingressroute.certResolver | Certificate resolver for IngressRoute | "" | + + +### Extensions common parameters + +| Parameter Name | Description | Value | +|-------------------------------------------------|---------------------------------------------------------|------------------------------------------| +| extensions.iiot.appName | IIoT extensions application name | extensions-all-iiot | +| extensions.iiot.port | Port for the IIoT extensions application | 8090 | +| extensions.iiot.service.name | Name of the IIoT extensions service | extensions-all-iiot | +| extensions.iiot.service.port | TargetPort of the IIoT extensions service | 8090 | + + +### External common parameters + +#### Consul common parameters + +| Parameter Name | Description | Value | +|-------------------------------------------------|----------------------------------------------------------|------------------------------------------| +| external.consul.appName | Consul application name | "consul" | +| external.consul.version | Consul version | 1.14.3 | +| external.consul.webPort | Port number for the Consul web interface | 8500 | +| external.consul.dnsPort | Port number for the Consul DNS interface | 8600 | +| external.consul.persistence.storageClassName | Storage class name for Consul PVs | "hostpath" | +| external.consul.persistence.storageSize | Size of the Consul PV | "1Gi" | +| external.consul.persistence.claimName | Name of the Consul PersistentVolumeClaim | "consul-pvc" | +| external.consul.persistence.pvName | Name of the Consul PersistentVolume | "consul-pv" | +| external.consul.service.name | Name of the Consul service | "consul" | +| external.consul.service.webPort | TargetPort of the Consul service for web interface | 8500 | +| external.consul.service.dnsPort | TargetPort of the Consul service for DNS interface | 8600 | + +#### Couchdb common parameters + +| Parameter Name | Description | Value | +|-------------------------------------------------|----------------------------------------------------------|------------------------------------------| +| external.couchdb.appName | CouchDB application name | "couchdb" | +| external.couchdb.version | CouchDB version | 3.3.1 | +| external.couchdb.user | CouchDB admin username | "admin" | +| external.couchdb.password | CouchDB admin password | empty (auto-generated) | +| external.couchdb.port | Port for the CouchDB service | 5984 | +| external.couchdb.service.name | Name of the CouchDB service | "couchdb" | +| external.couchdb.service.port | TargetPort of the CouchDB service | 5984 | +| external.couchdb.persistence.storageClassName | Storage class name for CouchDB PVs | "hostpath" | +| external.couchdb.persistence.storageSize | Size of the CouchDB PV | "1Gi" | +| external.couchdb.persistence.claimName | Name of the CouchDB PersistentVolumeClaim | "couchdb-pvc" | +| external.couchdb.persistence.pvName | Name of the CouchDB PersistentVolume | "couchdb-pv" | + +#### Influxdb common parameters + +| Parameter Name | Description | Value | +|-------------------------------------------------|----------------------------------------------------------|------------------------------------------| +| external.influxdb.appName | InfluxDB application name | "influxdb" | +| external.influxdb.version | InfluxDB version | 2.6 | +| external.influxdb.username | InfluxDB admin username | "admin" | +| external.influxdb.password | InfluxDB admin password | empty (auto-generated) | +| external.influxdb.adminToken | InfluxDB admin token | empty (auto-generated) | +| external.influxdb.initOrg | InfluxDB initial organization | "sp" | +| external.influxdb.initBucket | InfluxDB initial bucket | "sp" | +| external.influxdb.initMode | InfluxDB initialization mode | "setup" | +| external.influxdb.apiPort | Port number for the InfluxDB service (API) | 8083 | +| external.influxdb.httpPort | Port number for the InfluxDB service (HTTP) | 8086 | +| external.influxdb.grpcPort | Port number for the InfluxDB service (gRPC) | 8090 | +| external.influxdb.service.name | Name of the InfluxDB service | "influxdb" | +| external.influxdb.service.apiPort | TargetPort of the InfluxDB service for API | 8083 | +| external.influxdb.service.httpPort | TargetPort of the InfluxDB service for HTTP | 8086 | +| external.influxdb.service.grpcPort | TargetPort of the InfluxDB service for gRPC | 8090 | +| external.influxdb.persistence.storageClassName | Storage class name for InfluxDB PVs | "hostpath" | +| external.influxdb.persistence.storageSize | Size of the InfluxDB PV | "1Gi" | +| external.influxdb.persistence.storageSizeV1 | Size of the InfluxDB PV for v1 databases | "1Gi" | +| external.influxdb.persistence.claimName | Name of the InfluxDBv2 PersistentVolumeClaim | "influxdb2-pvc" | +| external.influxdb.persistence.claimNameV1 | Name of the InfluxDBv1 PersistentVolumeClaim | "influxdb-pvc" | +| external.influxdb.persistence.pvName | Name of the InfluxDBv2 PersistentVolume | "influxdb2-pv" | +| external.influxdb.persistence.pvNameV1 | Name of the InfluxDBv1 PersistentVolume | "influxdb-pv" | + + +#### Nats common parameters + +| Parameter Name | Description | Value | +|-------------------------------------------------|----------------------------------------------------------|------------------------------------------| +| external.nats.appName | NATS application name | "nats" | +| external.nats.port | Port for the NATS service | 4222 | +| external.nats.version | NATS version | | +| external.nats.service.type | Type of the NATS service | "NodePort" | +| external.nats.service.externalTrafficPolicy | External traffic policy for the NATS service | "Local" | +| external.nats.service.name | Name of the NATS service | "nats" | +| external.nats.service.port | TargetPort of the NATS service | 4222 | + + +#### Kafka common parameters + +| Parameter Name | Description | Value | +|-------------------------------------------------|----------------------------------------------------------|------------------------------------------| +| external.kafka.appName | Kafka application name | "kafka" | +| external.kafka.version | Kafka version | 2.2.0 | +| external.kafka.port | Port for the Kafka service | 9092 | +| external.kafka.external.hostname | Name which will be advertised to external clients. Clients which use (default) port 9094 | "localhost" +| external.kafka.service.name | Name of the Kafka service | "kafka" | +| external.kafka.service.port | TargetPort of the Kafka service | 9092 | +| external.kafka.service.portOutside | Port for Kafka client outside of the cluster | 9094 | +| external.kafka.persistence.storageClassName | Storage class name for Kafka PVs | "hostpath" | +| external.kafka.persistence.storageSize | Size of the Kafka PV | "1Gi" | +| external.kafka.persistence.claimName | Name of the Kafka PersistentVolumeClaim | "kafka-pvc" | +| external.kafka.persistence.pvName | Name of the Kafka PersistentVolume | "kafka-pv" | +| + +#### Zookeeper common parameters + +| Parameter Name | Description | Value | +|-------------------------------------------------|----------------------------------------------------------|------------------------------------------| +| external.zookeeper.appName | ZooKeeper application name | "zookeeper" | +| external.zookeeper.version | ZooKeeper version | 3.4.13 | +| external.zookeeper.port | Port for the ZooKeeper service | 2181 | +| external.zookeeper.service.name | Name of the ZooKeeper service | "zookeeper" | +| external.zookeeper.service.port | TargetPort of the ZooKeeper service | 2181 | +| external.zookeeper.persistence.storageClassName | Storage class name for ZooKeeper PVs | "hostpath" | +| external.zookeeper.persistence.storageSize | Size of the ZooKeeper PV | "1Gi" | +| external.zookeeper.persistence.claimName | Name of the ZooKeeper PersistentVolumeClaim | "zookeeper-pvc" | +| external.zookeeper.persistence.pvName | Name of the ZooKeeper PersistentVolume | "zookeeper-pv" | + + +#### Pulsar common parameters + +| Parameter Name | Description | Value | +|-------------------------------------------------|----------------------------------------------------------|------------------------------------------| +| external.pulsar.appName | pulsar application name | "pulsar" | +| external.pulsar.version | pulsar version | 3.0.0 | +| external.pulsar.port | Port for the pulsar service | 6650 | +| external.pulsar.service.name | Name of the pulsar service | "pulsar" | +| external.pulsar.service.port | TargetPort of the pulsar service | 6650 | +| external.pulsar.persistence.storageClassName | Storage class name for pulsar PVs | "hostpath" | +| external.pulsar.persistence.storageSize | Size of the pulsar PV | "1Gi" | +| external.pulsar.persistence.claimName | Name of the pulsar PersistentVolumeClaim | "pulsar-pvc" | +| external.pulsar.persistence.pvName | Name of the pulsar PersistentVolume | "pulsar-pv" | + +### Monitoring common parameters + +#### Monitoring - Prometheus + +| Parameter Name | Description | Value | +|-------------------------------------------------|----------------------------------------------------------|------------------------------------------| +| prometheus.appName | Prometheus application name | "prometheus" | +| prometheus.version | Prometheus version | 2.45.0 | +| prometheus.port | Prometheus port | 9090 | +| prometheus.service.name | Prometheus service name | "prometheus" | +| prometheus.service.port | Prometheus service port | 9090 | +| prometheus.persistence.storageClassName | Prometheus storage class name | "hostpath" | +| prometheus.persistence.storageSize | Prometheus storage size | "2Gi" | +| prometheus.persistence.claimName | Prometheus PVC claim name | "prometheus-pvc" | +| prometheus.persistence.pvName | Prometheus PV name | "prometheus-pv" | +| prometheus.persistence.tokenStorageSize | Prometheus token storage size | "16Ki" | +| prometheus.config.scrapeInterval | Prometheus scrape interval | 10s | +| prometheus.config.evaluationInterval | Prometheus evaluation interval | 15s | +| prometheus.config.backendJobName | Prometheus backend job name | "backend" | +| prometheus.config.extensionsName | Prometheus extensions job name | "extensions-all-iiot" | +| prometheus.config.tokenFileName | Prometheus token file name | "token" | +| prometheus.config.tokenFileDir | Prometheus token file directory | "/opt/data" + +#### Monitoring - Grafana + +| Parameter Name | Description | Value | +|-------------------------------------------------|----------------------------------------------------------|------------------------------------------| +| grafana.appName | Grafana application name | "grafana" | +| grafana.version | Grafana version | 10.1.2 | +| grafana.port | Grafana port | 3000 | +| grafana.service.name | Grafana service name | "grafana" | +| grafana.service.port | Grafana service port | 3000 | +| grafana.persistence.storageClassName | Grafana storage class name | "hostpath" | +| grafana.persistence.storageSize | Grafana storage size | "1Gi" | +| grafana.persistence.claimName | Grafana PVC claim name | "grafana-pvc" | +| grafana.persistence.pvName | Grafana PV name | "grafana-pv" | + + +## Auto-generation of parameters. + +The helm chart includes a `secrets.yaml` file which auto-generates several settings as follows: + +```yaml + +apiVersion: v1 +kind: Secret +metadata: + name: sp-secrets + namespace: {{ .Release.Namespace | quote }} +type: Opaque +data: + sp-initial-admin-password: {{ ternary (randAlphaNum 10) .Values.streampipes.auth.users.admin.password (empty .Values.streampipes.auth.users.admin.password) | b64enc | quote }} + sp-initial-client-secret: {{ ternary (randAlphaNum 35) .Values.streampipes.auth.users.service.secret (empty .Values.streampipes.auth.users.service.secret) | b64enc | quote }} + sp-encryption-passcode: {{ ternary (randAlphaNum 20) .Values.streampipes.auth.encryption.passcode (empty .Values.streampipes.auth.encryption.passcode) | b64enc | quote }} + sp-couchdb-password: {{ ternary (randAlphaNum 20) .Values.external.couchdb.password (empty .Values.external.couchdb.password) | b64enc | quote }} + sp-ts-storage-password: {{ ternary (randAlphaNum 20) .Values.external.influxdb.password (empty .Values.external.influxdb.password) | b64enc | quote }} + sp-ts-storage-token: {{ ternary (randAlphaNum 20) .Values.external.influxdb.adminToken (empty .Values.external.influxdb.adminToken) | b64enc | quote }} + +``` + ## Deleting the current helm chart deployment: ```bash diff --git a/docs/06_extend-customize-ui.md b/docs/06_extend-customize-ui.md new file mode 100644 index 000000000..c09823ffb --- /dev/null +++ b/docs/06_extend-customize-ui.md @@ -0,0 +1,226 @@ +--- +id: extend-customize-ui +title: UI customization +sidebar_label: UI customization +--- + + +## Custom theme + +It is possible to use a custom theme with individual styles, logos and images instead of the default StreamPipes theme. + +In this section, we describe the necessary steps to build and deploy a custom theme. + + +### Prerequisite: Learn how to run and build the UI + +To use a custom theme, it is required to build the UI with the custom settings. +In general, the UI can be found in the `ui` folder of the source code. + +Perform the following steps to build the UI; + +```bash + +# Install all necessary packages +npm install + +# Start the UI for development purposes +npm run start + +# Build the StreamPipes UI +npm run build + +``` + +## Customizable assets + +The following assets can be provided in a customized theme: + +* **Logo** This is the main logo image, which is shown e.g., on the login page. +* **Navigation Logo** This is the logo which appears in the top navigation bar after successful login +* **Favicon** The favicon is shown in the browser navbar. It is also used as the loading animation in StreamPipes. +* **String constants** Customizable strings, e.g., when you want to use another application name than **Apache StreamPipes**. +* **Theme variables** An scss file which defines custom colors and layouts. + +## Customize constants + +To customize constants, you can create a custom file `app.constants.ts` and modify the content based on the template below: + +```javascript + +import {Injectable} from '@angular/core'; + +@Injectable() +export class AppConstants { + + public readonly APP_NAME = "Apache StreamPipes"; + public readonly APP_TITLE = 'Apache StreamPipes'; + public readonly EMAIL = "[email protected]"; +} + + +``` + +## Customize theme + +To customize the theme, we provide a file named `variables.scss` which can be overridden with default color and style settings. + +See the example below: + +```scss + +/*! + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +$sp-color-primary: rgb(57, 181, 74); +$sp-color-primary-600: #06c12a; + +$sp-color-accent: #1b1464; + +$sp-color-accent-light-blue: rgb(59, 92, 149); +$sp-color-accent-light: rgb(156, 156, 156); +$sp-color-accent-light-transparent: rgba(156, 156, 156, 0.4); + +$sp-color-accent-dark: #83a3de; + +$sp-color-adapter: #7f007f; +$sp-color-stream: #ffeb3b; +$sp-color-processor: #009688; +$sp-color-sink: #3f51b5; + +$sp-color-error: #b71c1c; + +body { + --color-data-view: rgb(122, 206, 227); + --color-dashboard: rgb(76, 115, 164); + --color-adapter: rgb(182, 140, 97); + --color-data-source: #ffeb3b; + --color-pipeline: rgb(102, 185, 114); + --color-measurement: rgb(39, 164, 155); + --color-file: rgb(163, 98, 190); + + --button-border-radius: 5px; + --iconbar-width: 35px; + --navbar-icon-border-radius: 0; + --navbar-icon-padding: 0; +} + +:root { + --color-loading-bar: #{$sp-color-accent}; +} + +.dark-mode { + --color-primary: #{$sp-color-primary}; + --color-accent: #{$sp-color-accent-dark}; + --color-bg-outer: var(--color-bg-1); + --color-bg-page-container: var(--color-bg-0); + --color-bg-main-panel-header: var(--color-bg-0); + --color-bg-main-panel-content: var(--color-bg-0); + --color-bg-navbar-icon: inherit; + --color-bg-navbar-icon-selected: inherit; + --color-bg-0: #121212; + --color-bg-1: #282828; + --color-bg-2: #404040; + --color-bg-3: #424242; + --color-bg-4: #5f5f5f; + --color-bg-dialog: rgb(66, 66, 66); + --color-shadow: #c4c4c4; + --color-pe: #404040; + --color-default-text: rgba(255, 255, 255, 0.87); + --color-warn: #b36161; + + --color-tab-border: #cccccc; + + --color-navigation-bg: var(--color-primary); + --color-navigation-link-text: var(--color-bg-0); + --color-navigation-text: #121212; + --color-navigation-selected: #{$sp-color-primary}; + --color-navigation-hover: #{$sp-color-primary-600}; + --color-navigation-bg-selected: var(--color-bg-1); + --color-navigation-divider: #{$sp-color-primary}; + + --content-box-color: #404040; + --canvas-color: linear-gradient( + 90deg, + rgba(50, 50, 50, 0.5) 10%, + transparent 0% + ), + linear-gradient(rgba(50, 50, 50, 0.5) 10%, transparent 0%); +} + +.light-mode { + --color-primary: #{$sp-color-primary}; + --color-accent: #{$sp-color-accent}; + --color-bg-outer: var(--color-bg-1); + --color-bg-page-container: var(--color-bg-0); + --color-bg-main-panel-header: var(--color-bg-0); + --color-bg-main-panel-content: var(--color-bg-0); + --color-bg-navbar-icon: inherit; + --color-bg-navbar-icon-selected: inherit; + --color-bg-0: #ffffff; + --color-bg-1: #fafafa; + --color-bg-2: #f1f1f1; + --color-bg-3: rgb(224, 224, 224); + --color-bg-4: rgb(212, 212, 212); + --color-bg-dialog: #ffffff; + --color-shadow: #555; + --color-pe: #ffffff; + --color-default-text: #121212; + --color-warn: #b71c1c; + + --color-tab-border: #cccccc; + + --color-navigation-bg: var(--color-primary); + --color-navigation-link-text: var(--color-bg-0); + --color-navigation-text: #ffffff; + --color-navigation-selected: #{$sp-color-primary}; + --color-navigation-hover: #{$sp-color-primary-600}; + --color-navigation-bg-selected: var(--color-bg-1); + --color-navigation-divider: var(--color-primary); + + --content-box-color: rgb(156, 156, 156); + --canvas-color: linear-gradient( + 90deg, + rgba(208, 208, 208, 0.5) 10%, + transparent 0% + ), + linear-gradient(rgba(208, 208, 208, 0.5) 10%, transparent 0%); +} + +``` +## Run a customized build + +To create a new UI build with customized themes, use the following command: + +````bash + +UI_LOC=PATH_TO_FOLDER_WITH_CUSTOM_TEMPLATES \\ +THEME_LOC=$UI_LOC/_variables.scss \\ +LOGO_HEADER_LOC=$UI_LOC/img/logo.png \\ +FAVICON_LOC=$UI_LOC/img/favicon.png \\ +LOGO_NAV_LOC=$UI_LOC/img/logo-navigation.png \\ +CONSTANTS_FILE=$UI_LOC/app.constants.ts \\ +npm run build + +```` + +First, we create a helper environment variable that links to a folder which includes custom logos, the theme file and constants. +Next, we set the variables above to override default logos and stylings. +Finally, the usual build process is executed. + +Once finished, you've successfully customized an Apache StreamPipes instance! diff --git a/docs/07_technicals-messaging.md b/docs/07_technicals-messaging.md index 9ce44b9f4..d5308a6d8 100644 --- a/docs/07_technicals-messaging.md +++ b/docs/07_technicals-messaging.md @@ -40,3 +40,26 @@ To select a protocol when multiple protocols are supported by two pipeline eleme StreamPipes ensures that only pipeline elements which have a commonly supported protocol can be connected. Note that you might need to change the installation files. For the `Docker-Compose` based installation, we provide various compose file for different messaging setups. For the `Kubernetes` installation, we provide variables which can be set in the helm chart's `values.yaml` file. + +### Configure broker addresses + +By default, StreamPipes assumes that the messaging system is started from its own environment, e.g., the system configured in the selected `Docker-Compose` file. + +Besides that, it is also possible to let StreamPipes connect to an externally provided messaging system. For this purpose, various environment variables exist. + +* `SP_PRIORITIZED_PROTOCOL` to set the prioritized protocol to either `kafka`, `mqtt`, `nats` or `pulsar` + +* `SP_KAFKA_HOST`, `SP_KAFKA_PORT` to configure Kafka access +* `SP_MQTT_HOST`, `SP_MQTT_PORT` to configure MQTT access +* `SP_NATS_HOST`, `SP_NATS_PORT` to configure NATS access +* `SP_PULSAR_URL` to configure Pulsar access + + +Most settings can also be set in the UI under `Settings->Messaging`. + +:::warning Installation-time configurations +Although it is currently possible to change messaging settings in the user interface, we do not support dynamic modification of messaging systems. +Choosing a proper system is considered an installation-time setting which should not be changed afterwards. +Already existing Adapters and pipeline elements are not properly updated after changes of the messaging layer. +::: + diff --git a/docs/07_technicals-runtime-wrappers.md b/docs/07_technicals-runtime-wrappers.md index 8b757caa6..9cebadfd2 100644 --- a/docs/07_technicals-runtime-wrappers.md +++ b/docs/07_technicals-runtime-wrappers.md @@ -4,4 +4,34 @@ title: Runtime Wrappers sidebar_label: Runtime Wrappers --- -tbd \ No newline at end of file +## Overview + +In general, StreamPipes has an exchangeable runtime layer, e.g., the actual processing of incoming events can be delegated to a third-party stream processing system such as Kafka Streams or Apache Flink. + +The default runtime wrapper is the StreamPipes Native Wrapper, called the `StandaloneWrapper`. + +Although not recommended for production, we invite interested developers to check out our experimental wrappers: + +* Kafka Streams runtime wrapper at [https://github.com/apache/streampipes/tree/dev/streampipes-wrapper-kafka-streams](https://github.com/apache/streampipes/tree/dev/streampipes-wrapper-kafka-streams) +* Apache Flink runtime wrapper at [https://github.com/apache/streampipes/tree/dev/streampipes-wrapper-flink](https://github.com/apache/streampipes/tree/dev/streampipes-wrapper-flink) + +## Assigning a runtime wrapper to an extension service + +Runtime wrappers can be assigned in the `Service Definition` of the `Init` class of an extension service: + +```java + + @Override + public SpServiceDefinition provideServiceDefinition(){ + return SpServiceDefinitionBuilder.create("org.apache.streampipes.extensions.all.jvm", + "StreamPipes Extensions (JVM)", + "",8090) + ... + .registerRuntimeProvider(new StandaloneStreamPipesRuntimeProvider()) + ... + .build(); + } + +``` + +Please let us know through our communication channels if you are interested in this feature and if you are willing to contribute! diff --git a/website-v2/sidebars.json b/website-v2/sidebars.json index a5d0c2400..ce4d65d10 100644 --- a/website-v2/sidebars.json +++ b/website-v2/sidebars.json @@ -176,10 +176,12 @@ } ], "⚡ Deploy StreamPipes": [ + "choosing-the-right-flavor", "deploy-docker", "deploy-kubernetes", "deploy-use-ssl", - "deploy-security" + "deploy-security", + "deploy-environment-variables" ], "💻 Customize StreamPipes": [ "extend-setup", @@ -196,11 +198,11 @@ "extend-sdk-stream-requirements", "extend-sdk-static-properties", "extend-sdk-output-strategies", - "extend-sdk-migration-service-discovery" + "extend-sdk-migration-service-discovery", + "extend-customize-ui" ], "🔧 Technicals": [ "technicals-architecture", - "technicals-user-guidance", "technicals-runtime-wrappers", "technicals-messaging" ], @@ -214,4 +216,4 @@ "faq-common-problems" ] } -} \ No newline at end of file +} diff --git a/website-v2/versioned_sidebars/version-0.92.0-sidebars.json b/website-v2/versioned_sidebars/version-0.92.0-sidebars.json index cc356ddb3..657aa113c 100644 --- a/website-v2/versioned_sidebars/version-0.92.0-sidebars.json +++ b/website-v2/versioned_sidebars/version-0.92.0-sidebars.json @@ -197,7 +197,6 @@ ], "\uD83D\uDD27 Technicals": [ "technicals-architecture", - "technicals-user-guidance", "technicals-runtime-wrappers", "technicals-messaging" ],
