This is an automated email from the ASF dual-hosted git repository.
zehnder pushed a commit to branch
3885-use-nats-as-default-setup-in-docker-compose
in repository https://gitbox.apache.org/repos/asf/streampipes.git
The following commit(s) were added to
refs/heads/3885-use-nats-as-default-setup-in-docker-compose by this push:
new b2203e6ff6 refactor: Use nats as default and clean up of installation
configurations and readme
b2203e6ff6 is described below
commit b2203e6ff66dd17df6af38f23d069adcb0145681
Author: Philipp Zehnder <[email protected]>
AuthorDate: Tue Oct 28 11:29:41 2025 +0100
refactor: Use nats as default and clean up of installation configurations
and readme
---
installer/compose/README.md | 48 ++++-----
...r-compose.full.yml => docker-compose.kafka.yml} | 33 ++----
installer/compose/docker-compose.nats.yml | 114 ---------------------
installer/compose/docker-compose.pulsar.yml | 109 --------------------
installer/compose/docker-compose.yml | 40 ++------
5 files changed, 35 insertions(+), 309 deletions(-)
diff --git a/installer/compose/README.md b/installer/compose/README.md
index d9cff8bb59..e7e2b982bf 100644
--- a/installer/compose/README.md
+++ b/installer/compose/README.md
@@ -36,24 +36,22 @@ Go to http://localhost to finish the installation in the
browser. Once finished,
* Docker-Compose >= 1.17.0 (Compose file format: 3.4)
* Google Chrome (recommended), Mozilla Firefox, Microsoft Edge
-Tested on: **macOS, Linux, Windows 10** (CMD, PowerShell, GitBash)
+Tested on: **macOS, Linux, Windows** (CMD, PowerShell, GitBash)
-**macOS** and **Windows 10** (Pro, Enterprise, Education) users can easily get
Docker and Docker-Compose on their systems by installing **Docker for
Mac/Windows** (recommended).
+**macOS** and **Windows** users can easily get Docker and Docker-Compose on
their systems by installing **Docker for Mac/Windows** (recommended).
> **NOTE**: On purpose, we disabled all port mappings except of http port
> **80** to access the StreamPipes UI to provide minimal surface for
> conflicting ports.
## Usage
-We provide three options to get you going:
+We provide several options to get you going:
-- **default**: the standard installation, uses Kafka as internal message
broker (recommended)
-- **nats**: the standard installation which uses Nats as message broker
(recommended for new installations)
-- **full**: contains experimental Flink wrappers
-- **quickstart**: contains pre-configured sample assets, including pipelines,
dashboards, and data views. We recommend first-time StreamPipes users to use
the Quickstart mode to experience the convenience of StreamPipes in IIoT!
（Recommended for first-time users)
+- **default**: the standard installation, uses NATS as internal message broker
(recommended for new installations)
+- **kafka**: starts the Kafka-based setup via `docker-compose.kafka.yml`
+- **minimal**: contains only a minimal set of adapters, processors and sinks
for iiot use cases
-The ``nats`` version will become the default version in a later release. You
can already try it for new installations,
-but there's not yet an automatic migration from current Kafka-based
installations to Nats.
+The NATS-based setup is the recommended default. If you previously relied on
Kafka, the Kafka compose file is still available as `docker-compose.kafka.yml`.
-**Starting** the **default** option is as easy as simply running:
+**Starting** the **default (NATS)** option is as easy as simply running:
> **NOTE**: Starting might take a while since `docker-compose up` also
> initially pulls all Docker images from Dockerhub.
```bash
@@ -69,35 +67,27 @@ docker-compose down
# docker-compose down -v
```
-Starting the **nats** option works as follows:
-```bash
-docker-compose -f docker-compose.nats.yml up -d
-# go to `http://localhost` after all services are started
-```
-Stopping the **nats** option:
-```bash
-docker-compose -f docker-compose.nats.yml down
-```
+If you need the Kafka-based setup, start it with the dedicated Kafka compose
file:
-Starting the **full** option is almost the same, just specify the
`docker-compose.full.yml` file:
```bash
-docker-compose -f docker-compose.full.yml up -d
+docker-compose -f docker-compose.kafka.yml up -d
# go to `http://localhost` after all services are started
```
-Stopping the **full** option:
+
+Stopping the **kafka** option:
+
```bash
-docker-compose -f docker-compose.full.yml down
+docker-compose -f docker-compose.kafka.yml down
```
-We introduce quickstart deployment that comes with a set of predefined
StreamPipes assets, to use the quickstart mode, just build the Docker image and
start **quickstart** option:
+Starting the **minimal** option is almost the same, just specify the
`docker-compose.full.yml` file:
```bash
-docker-compose -f docker-compose.quickstart.yml build script-runner
-docker-compose -f docker-compose.quickstart.yml up -d
+docker-compose -f docker-compose.minimal.yml up -d
# go to `http://localhost` after all services are started
```
-Stopping the **quickstart** option:
+Stopping the **minimal** option:
```bash
-docker-compose -f docker-compose.quickstart.yml down
+docker-compose -f docker-compose.minimal.yml down
```
## Update services
@@ -118,7 +108,7 @@ SP_VERSION=<VERSION>
If you've found a bug or have a feature that you'd love to see in StreamPipes,
feel free to create an issue i on
[GitHub](https://github.com/apache/streampipes/issues).
## Get help
-Since we purely levarage Docker Compose, please see their
[documentation](https://docs.docker.com/compose/) in case you want to find out
more about their available
[commands](https://docs.docker.com/compose/reference/overview/).
+Since we purely levarge Docker Compose, please see their
[documentation](https://docs.docker.com/compose/) in case you want to find out
more about their available
[commands](https://docs.docker.com/compose/reference/overview/).
If you have any problems during the installation or questions around
StreamPipes, you'll get help through one of our community channels:
diff --git a/installer/compose/docker-compose.full.yml
b/installer/compose/docker-compose.kafka.yml
similarity index 86%
rename from installer/compose/docker-compose.full.yml
rename to installer/compose/docker-compose.kafka.yml
index aaa3a5613e..cdaa845a3c 100644
--- a/installer/compose/docker-compose.full.yml
+++ b/installer/compose/docker-compose.kafka.yml
@@ -26,6 +26,9 @@ services:
# StreamPipes services
backend:
image: "${SP_DOCKER_REGISTRY}/backend:${SP_VERSION}"
+ environment:
+ - SP_PRIORITIZED_PROTOCOL=kafka
+ - SP_HTTP_SERVER_ADAPTER_ENDPOINT=extensions-all-jvm:8090
depends_on:
- couchdb
volumes:
@@ -37,8 +40,6 @@ services:
ui:
image: "${SP_DOCKER_REGISTRY}/ui:${SP_VERSION}"
- environment:
- - SP_HTTP_SERVER_ADAPTER_ENDPOINT=extensions-all-jvm:8090
ports:
- "80:8088"
depends_on:
@@ -109,31 +110,6 @@ services:
networks:
spnet:
- jobmanager:
- image: fogsyio/flink:1.13.5-scala_2.11
- command: jobmanager
- environment:
- - JOB_MANAGER_RPC_ADDRESS=jobmanager
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
- taskmanager:
- image: fogsyio/flink:1.13.5-scala_2.11
- depends_on:
- - jobmanager
- command: taskmanager
- links:
- - "jobmanager:jobmanager"
- environment:
- - JOB_MANAGER_RPC_ADDRESS=jobmanager
- - TASK_MANAGER_NUMBER_OF_TASK_SLOTS=20
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
# Lite version pipeline elements
extensions-all-jvm:
image: "${SP_DOCKER_REGISTRY}/extensions-all-jvm:${SP_VERSION}"
@@ -142,6 +118,7 @@ services:
networks:
spnet:
+
volumes:
backend:
connect:
@@ -151,6 +128,8 @@ volumes:
influxdb2:
nginx:
+
networks:
spnet:
driver: bridge
+
diff --git a/installer/compose/docker-compose.nats.yml
b/installer/compose/docker-compose.nats.yml
deleted file mode 100644
index 4c3fdefc9d..0000000000
--- a/installer/compose/docker-compose.nats.yml
+++ /dev/null
@@ -1,114 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# global logging
-x-logging:
- &default-logging
- options:
- max-size: '12m'
- max-file: '5'
- driver: json-file
-
-services:
-
- # StreamPipes services
- backend:
- image: "${SP_DOCKER_REGISTRY}/backend:${SP_VERSION}"
- depends_on:
- - couchdb
- environment:
- - SP_PRIORITIZED_PROTOCOL=nats
- volumes:
- - backend:/root/.streampipes
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
- ui:
- image: "${SP_DOCKER_REGISTRY}/ui:${SP_VERSION}"
- environment:
- - SP_HTTP_SERVER_ADAPTER_ENDPOINT=extensions-all-jvm:8090
- ports:
- - "80:8088"
- depends_on:
- - couchdb
- - backend
- volumes:
- - nginx:/etc/nginx/
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
- couchdb:
- image: couchdb:3.3.1
- environment:
- - COUCHDB_USER=admin
- - COUCHDB_PASSWORD=admin
- volumes:
- - couchdb:/opt/couchdb/data
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
- influxdb:
- image: influxdb:2.6
- environment:
- - INFLUXDB_DATA_ENGINE=tsm1
- - INFLUXDB_REPORTING_DISABLED=false
- - INFLUXDB_ADMIN_ENABLED=true
- - DOCKER_INFLUXDB_INIT_USERNAME=admin
- - DOCKER_INFLUXDB_INIT_PASSWORD=sp-admin
- - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=sp-admin
- - DOCKER_INFLUXDB_INIT_ORG=sp
- - DOCKER_INFLUXDB_INIT_BUCKET=sp
- - DOCKER_INFLUXDB_INIT_MODE=${SP_INFLUX_INIT_MODE}
- volumes:
- - influxdb:/var/lib/influxdb
- - influxdb2:/var/lib/influxdb2
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
- nats:
- image: nats
- restart: unless-stopped
- networks:
- spnet:
-
- # Lite version pipeline elements
- extensions-all-jvm:
- image: "${SP_DOCKER_REGISTRY}/extensions-all-jvm:${SP_VERSION}"
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
-volumes:
- backend:
- connect:
- couchdb:
- influxdb:
- influxdb2:
- nginx:
-
-
-networks:
- spnet:
- driver: bridge
-
diff --git a/installer/compose/docker-compose.pulsar.yml
b/installer/compose/docker-compose.pulsar.yml
deleted file mode 100644
index 1366e93990..0000000000
--- a/installer/compose/docker-compose.pulsar.yml
+++ /dev/null
@@ -1,109 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# global logging
-x-logging:
- &default-logging
- options:
- max-size: '12m'
- max-file: '5'
- driver: json-file
-
-services:
-
- # StreamPipes services
- backend:
- image: "${SP_DOCKER_REGISTRY}/backend:${SP_VERSION}"
- depends_on:
- - couchdb
- environment:
- - SP_PRIORITIZED_PROTOCOL=pulsar
- - SP_PULSAR_URL=pulsar://localhost:6650
- volumes:
- - backend:/root/.streampipes
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
- ui:
- image: "${SP_DOCKER_REGISTRY}/ui:${SP_VERSION}"
- environment:
- - SP_HTTP_SERVER_ADAPTER_ENDPOINT=extensions-all-jvm:8090
- ports:
- - "80:8088"
- depends_on:
- - couchdb
- - backend
- volumes:
- - nginx:/etc/nginx/
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
- couchdb:
- image: couchdb:3.3.1
- environment:
- - COUCHDB_USER=admin
- - COUCHDB_PASSWORD=admin
- volumes:
- - couchdb:/opt/couchdb/data
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
- influxdb:
- image: influxdb:2.6
- environment:
- - INFLUXDB_DATA_ENGINE=tsm1
- - INFLUXDB_REPORTING_DISABLED=false
- - INFLUXDB_ADMIN_ENABLED=true
- - DOCKER_INFLUXDB_INIT_USERNAME=admin
- - DOCKER_INFLUXDB_INIT_PASSWORD=sp-admin
- - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=sp-admin
- - DOCKER_INFLUXDB_INIT_ORG=sp
- - DOCKER_INFLUXDB_INIT_BUCKET=sp
- - DOCKER_INFLUXDB_INIT_MODE=${SP_INFLUX_INIT_MODE}
- volumes:
- - influxdb:/var/lib/influxdb
- - influxdb2:/var/lib/influxdb2
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
- # Lite version pipeline elements
- extensions-all-jvm:
- image: "${SP_DOCKER_REGISTRY}/extensions-all-jvm:${SP_VERSION}"
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
-volumes:
- backend:
- connect:
- couchdb:
- influxdb:
- influxdb2:
- nginx:
-
-
-networks:
- spnet:
- driver: bridge
-
diff --git a/installer/compose/docker-compose.yml
b/installer/compose/docker-compose.yml
index f0d00599db..4c3fdefc9d 100644
--- a/installer/compose/docker-compose.yml
+++ b/installer/compose/docker-compose.yml
@@ -26,10 +26,10 @@ services:
# StreamPipes services
backend:
image: "${SP_DOCKER_REGISTRY}/backend:${SP_VERSION}"
- environment:
- - SP_HTTP_SERVER_ADAPTER_ENDPOINT=extensions-all-jvm:8090
depends_on:
- couchdb
+ environment:
+ - SP_PRIORITIZED_PROTOCOL=nats
volumes:
- backend:/root/.streampipes
logging: *default-logging
@@ -39,6 +39,8 @@ services:
ui:
image: "${SP_DOCKER_REGISTRY}/ui:${SP_VERSION}"
+ environment:
+ - SP_HTTP_SERVER_ADAPTER_ENDPOINT=extensions-all-jvm:8090
ports:
- "80:8088"
depends_on:
@@ -63,32 +65,6 @@ services:
networks:
spnet:
- kafka:
- image: apache/kafka:4.1.0
- hostname: kafka
- environment:
- - KAFKA_NODE_ID=0
- - KAFKA_PROCESS_ROLES=controller,broker
- - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
- - KAFKA_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
- - KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,OUTSIDE://:9094
- -
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,OUTSIDE://localhost:9094
- -
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT
- - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
- - KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
- - KAFKA_MESSAGE_MAX_BYTES=5000012
- - KAFKA_REPLICA_FETCH_MAX_BYTES=10000000
- - KAFKA_FETCH_MESSAGE_MAX_BYTES=5000012
- - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
- - KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS=1
- - KAFKA_DEFAULT_REPLICATION_FACTOR=1
- volumes:
- - kafka3:/var/lib/kafka/data
- logging: *default-logging
- restart: unless-stopped
- networks:
- spnet:
-
influxdb:
image: influxdb:2.6
environment:
@@ -109,6 +85,12 @@ services:
networks:
spnet:
+ nats:
+ image: nats
+ restart: unless-stopped
+ networks:
+ spnet:
+
# Lite version pipeline elements
extensions-all-jvm:
image: "${SP_DOCKER_REGISTRY}/extensions-all-jvm:${SP_VERSION}"
@@ -117,12 +99,10 @@ services:
networks:
spnet:
-
volumes:
backend:
connect:
couchdb:
- kafka3:
influxdb:
influxdb2:
nginx:
