This is an automated email from the ASF dual-hosted git repository.
lvshaokang pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-streampark.git
The following commit(s) were added to refs/heads/dev by this push:
new d860f40d7 [Bug] fix docker deployment (#1817)
d860f40d7 is described below
commit d860f40d749ee1ef7abf4592ca820035687a9018
Author: monster <[email protected]>
AuthorDate: Fri Oct 14 09:47:57 2022 +0800
[Bug] fix docker deployment (#1817)
Prefect docker deployment
---
.github/workflows/docker_push.yml | 6 +--
deploy/README.md | 74 -------------------------------
deploy/docker/.env | 27 ++++++++++++
deploy/docker/README.md | 3 ++
deploy/docker/compose/Dockerfile | 52 ----------------------
deploy/docker/console/Dockerfile | 45 ++++++++++---------
deploy/docker/docker-compose.yaml | 92 ++++++++++++++++-----------------------
deploy/docker/mysql/.env | 30 +++++++++++++
deploy/docker/mysql/Dockerfile | 21 ---------
deploy/docker/pgsql/.env | 30 +++++++++++++
10 files changed, 152 insertions(+), 228 deletions(-)
diff --git a/.github/workflows/docker_push.yml
b/.github/workflows/docker_push.yml
index c650ae12c..8c28f0ff4 100644
--- a/.github/workflows/docker_push.yml
+++ b/.github/workflows/docker_push.yml
@@ -56,14 +56,14 @@ jobs:
- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
- username: ${{ secrets.DOCKERHUB_USERNAME }}
- password: ${{ secrets.DOCKERHUB_PASSWORD }}
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Docker meta
id: meta
uses: docker/metadata-action@v4
with:
- images: streamxhub/streampark-console
+ images: apache/streampark
- name: Build and push streamx
uses: docker/build-push-action@v3
diff --git a/deploy/README.md b/deploy/README.md
deleted file mode 100644
index 263e26a51..000000000
--- a/deploy/README.md
+++ /dev/null
@@ -1,74 +0,0 @@
-# StreamPark on Docker and Kubernetes Quick Start
-
-## Docker Quick Start
-
-### Prepare
-
- Docker 1.13.1+
- Docker Compose 1.28.0+
-
-### Using docker-compose to Start Server
-After installed docker-compose, it is recommended to modify some
configurations for better experience. We highly recommended modify
docker-compose's memory up to 4 GB.
-
- Mac：Click Docker Desktop -> Preferences -> Resources -> Memory modified it
- Windows Docker Desktop：
- Hyper-V mode: Click Docker Desktop -> Settings -> Resources -> Memory
modified it
- WSL 2 mode: see WSL 2 utility VM for more detail.
-
-After complete the configuration, we can get the StreamPark file from download
page form its source package, and make sure you get the right version. After
download the package, you can run the commands as below.
-
-1.Environment build via mvn
-```
-mvn clean install -Dscala.version=2.12.5 -Dscala.binary.version=2.12
-DskipTests
-```
-
-2.Execution Docker compose build command
-```
-cd deploy/docker
-docker-compose up -d
-```
-
-3.Set up flink home on the StreamPark web ui
-```
-/streampark/flink/
-```
-
-4.Setting up remote session clusters
-
-5.Complete the above steps and execute the flink task submit
-
-## QuickStart in Kubernetes
-
-### Prerequisites
- Helm version 3.1.0+
- Kubernetes version 1.12+
-### Install StreamPark
-#### Installing the Chart
-
-To install the chart with the release name `streamx`:
-```
-cd helm
-helm install streampark streampark/
-```
-
-#### Uninstalling the Chart
-
-To uninstall/delete the `streampark` deployment:
-
-```
-helm uninstall streampark
-```
-
-The command removes all the Kubernetes components associated with the chart
and deletes the release.
-
-## Build Private Image
-1.Execution of scripts
-```
-cd ./plugins
-bash ./buildPrivateImage
-```
-2.Enter the appropriate parameters
-
- Enter private warehouse address
- Enter username password
- Enter the name of the mirror that matches the private repository
requirements
diff --git a/deploy/docker/.env b/deploy/docker/.env
new file mode 100644
index 000000000..a8d28e1da
--- /dev/null
+++ b/deploy/docker/.env
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+HUB=apache/streampark
+TAG=latest
+
+TZ=Asia/Shanghai
+SPRING_PROFILES_ACTIVE=h2
+FLINK=flink1.14.5
+FLINK_PROPERTIES=jobmanager.rpc.address: jobmanager
+FLINK_BASE_IMAGE=flink:1.14.5-scala_2.12
+COMMAND=/bin/sh -c "bash bin/startup.sh"
\ No newline at end of file
diff --git a/deploy/docker/README.md b/deploy/docker/README.md
new file mode 100644
index 000000000..b75e7a01d
--- /dev/null
+++ b/deploy/docker/README.md
@@ -0,0 +1,3 @@
+# StreamPark for Docker and Kubernetes
+
+* [Start Up StreamPark with
Docker](https://streampark.apache.org/docs/user-guide/docker-deployment)
diff --git a/deploy/docker/compose/Dockerfile b/deploy/docker/compose/Dockerfile
deleted file mode 100644
index 10ce5ead0..000000000
--- a/deploy/docker/compose/Dockerfile
+++ /dev/null
@@ -1,52 +0,0 @@
-###############################################################################
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-###############################################################################
-FROM maven:3.8.5-jdk-8 as deps-stage
-COPY . /
-WORKDIR /
-RUN tar zxvf
streampark-console/streampark-console-service/target/streampark-console-service-*.tar.gz
\
-&& mv streampark-console-service-* streampark
-
-
-FROM maven:3.8.5-jdk-8
-WORKDIR /streampark
-COPY --from=deps-stage /streampark /streampark
-
-ENV NODE_VERSION 16.1.0
-ENV NPM_VERSION 7.11.2
-ARG DB=localhost
-ENV FLINK_VERSION 1.14.5
-ENV SCALA_VERSION scala_2.12
-
-RUN echo Y|apt-get update \
- && echo Y|apt-get install iputils-ping \
- && echo Y|apt-get install vim \
- && echo Y|apt-get install net-tools \
- && echo Y|apt-get install dos2unix
-
-RUN dos2unix bin/*.sh \
- && sed -i -e 's/eval $NOHUP/eval/' bin/streampark.sh \
- && sed -i -e 's/StreamParkConsoleBootstrap \\/StreamParkConsoleBootstrap/'
bin/streampark.sh \
- && sed -i -e 's/>> "$APP_OUT" 2>&1 "&"//' bin/streampark.sh \
- && sed -i -e 's/localhost/'$DB'/' conf/application.yml
-
-RUN wget
"https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz"; \
- && tar zxvf "node-v$NODE_VERSION-linux-x64.tar.gz" -C /usr/local
--strip-components=1 \
- && rm "node-v$NODE_VERSION-linux-x64.tar.gz" \
- && ln -s /usr/local/bin/node /usr/local/bin/nodejs
-
-EXPOSE 10000
-ENTRYPOINT bash bin/startup.sh
diff --git a/deploy/docker/console/Dockerfile b/deploy/docker/console/Dockerfile
index 5ae84eaee..d5697ff92 100644
--- a/deploy/docker/console/Dockerfile
+++ b/deploy/docker/console/Dockerfile
@@ -1,19 +1,20 @@
-###############################################################################
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-###############################################################################
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
FROM alpine:3.16 as deps-stage
COPY . /
WORKDIR /
@@ -25,34 +26,32 @@ FROM docker:dind
WORKDIR /streampark
COPY --from=deps-stage /streampark /streampark
-ARG DB=localhost
ENV NODE_VERSION=16.1.0
ENV NPM_VERSION=7.11.2
RUN sed -i -e 's/eval $NOHUP/eval/' bin/streampark.sh \
&& sed -i -e 's/StreamParkConsoleBootstrap \\/StreamParkConsoleBootstrap/'
bin/streampark.sh \
- && sed -i -e 's/>> "$APP_OUT" 2>&1 "&"//' bin/streampark.sh \
- && sed -i -e 's/localhost/'$DB'/' conf/application.yml
+ && sed -i -e 's/>> "$APP_OUT" 2>&1 "&"//' bin/streampark.sh
RUN apk add openjdk8 \
&& apk add maven \
&& apk add wget \
&& apk add vim \
- && apk add bash
+ && apk add bash \
+ && apk add curl
ENV JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk
ENV MAVEN_HOME=/usr/share/java/maven-3
ENV PATH $JAVA_HOME/bin:$PATH
ENV PATH $MAVEN_HOME/bin:$PATH
-
-
RUN wget
"https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz"; \
&& tar zxvf "node-v$NODE_VERSION-linux-x64.tar.gz" -C /usr/local
--strip-components=1 \
&& rm "node-v$NODE_VERSION-linux-x64.tar.gz" \
- && ln -s /usr/local/bin/node /usr/local/bin/nodejs
+ && ln -s /usr/local/bin/node /usr/local/bin/nodejs \
+ && curl -LO https://dl.k8s.io/release/v1.23.0/bin/linux/amd64/kubectl \
+ && install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
-RUN mkdir -p ~/.kube \
- && cat plugins/config >> ~/.kube/config
+RUN mkdir -p ~/.kube
EXPOSE 10000
diff --git a/deploy/docker/docker-compose.yaml
b/deploy/docker/docker-compose.yaml
index a167dba93..5de0c1f4e 100644
--- a/deploy/docker/docker-compose.yaml
+++ b/deploy/docker/docker-compose.yaml
@@ -1,83 +1,65 @@
-###############################################################################
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-###############################################################################
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
version: '3.8'
services:
streampark-server:
- build:
- context: ../..
- dockerfile: deploy/docker/compose/Dockerfile
- args:
- DB: database
+ image: ${HUB}/streampark:${TAG}
+ command: ${COMMAND}
ports:
- 10000:10000
- depends_on:
- - database
+ env_file: .env
volumes:
- - flink1.14.5:/ streampark/flink/flink1.14.5
+ - flink:/streampark/flink/${FLINK}
+ - /var/run/docker.sock:/var/run/docker.sock
+ - /etc/hosts:/etc/hosts:ro
+ - ~/.kube:/root/.kube:ro
restart: unless-stopped
+ networks:
+ - streampark
- database:
- build:
- context: ../..
- dockerfile: deploy/docker/mysql/Dockerfile
- command: mysqld --character-set-server=utf8mb4
--collation-server=utf8mb4_unicode_ci
- ports:
- - 3306:3306
- volumes:
- - mysql-data:/var/lib/mysql
- environment:
- MYSQL_ROOT_PASSWORD: 123456
- MYSQL_ROOT_HOST: '%'
- MYSQL_DATABASE: streampark
- MYSQL_USER: streampark
- MYSQL_PASSWORD: 123456
- healthcheck:
- test: [ "CMD", "mysqladmin" ,"-uroot","-p123456","ping", "-h",
"localhost" ]
- timeout: 20s
- retries: 10
- restart: unless-stopped
jobmanager:
- image: flink:1.14.5-scala_2.12
+ image: ${FLINK_BASE_IMAGE}
ports:
- "8081:8081"
command: jobmanager
volumes:
- - flink1.14.5:/opt/flink
- environment:
- - |
- FLINK_PROPERTIES=
- jobmanager.rpc.address: jobmanager
+ - flink:/opt/flink
+ env_file: .env
restart: unless-stopped
+ networks:
+ - streampark
+
taskmanager:
- image: flink:1.14.5-scala_2.12
+ image: ${FLINK_BASE_IMAGE}
depends_on:
- jobmanager
command: taskmanager
deploy:
replicas: 1
- environment:
- - |
- FLINK_PROPERTIES=
- jobmanager.rpc.address: jobmanager
- taskmanager.numberOfTaskSlots: 10
+ env_file: .env
restart: unless-stopped
+ networks:
+ - streampark
+
+networks:
+ streampark:
+ driver: bridge
volumes:
- mysql-data:
- flink1.14.5:
+ flink:
diff --git a/deploy/docker/mysql/.env b/deploy/docker/mysql/.env
new file mode 100644
index 000000000..e114998e5
--- /dev/null
+++ b/deploy/docker/mysql/.env
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+HUB=apache/streampark
+TAG=latest
+
+TZ=Asia/Shanghai
+SPRING_PROFILES_ACTIVE=mysql
+SPRING_DATASOURCE_URL=jdbc:mysql://localhost:3306/streampark?useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8
+SPRING_DATASOURCE_USERNAME=root
+SPRING_DATASOURCE_PASSWORD=streampark
+FLINK=flink1.14.5
+FLINK_PROPERTIES=jobmanager.rpc.address: jobmanager
+FLINK_BASE_IMAGE=flink:1.14.5-scala_2.12
+COMMAND=/bin/sh -c "wget
https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.30.tar.gz
&& tar -zxvf mysql-connector-java-8.0.30.tar.gz && cp
mysql-connector-java-8.0.30/mysql-connector-java-8.0.30.jar lib && bash
bin/startup.sh"
diff --git a/deploy/docker/mysql/Dockerfile b/deploy/docker/mysql/Dockerfile
deleted file mode 100644
index d02d79fcf..000000000
--- a/deploy/docker/mysql/Dockerfile
+++ /dev/null
@@ -1,21 +0,0 @@
-###############################################################################
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-###############################################################################
-FROM biarms/mysql:5.7
-
-COPY
streampark-console/streampark-console-service/src/assembly/script/schema/mysql-schema.sql
/docker-entrypoint-initdb.d/01-streampark-init.sql
-COPY
streampark-console/streampark-console-service/src/assembly/script/data/mysql-data.sql
/docker-entrypoint-initdb.d/02-streampark-data.sql
-EXPOSE 3306
diff --git a/deploy/docker/pgsql/.env b/deploy/docker/pgsql/.env
new file mode 100644
index 000000000..dce3a4ba4
--- /dev/null
+++ b/deploy/docker/pgsql/.env
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+HUB=apache/streampark
+TAG=latest
+
+TZ=Asia/Shanghai
+SPRING_PROFILES_ACTIVE=pgsql
+SPRING_DATASOURCE_URL=jdbc:postgresql://localhost:5432/streampark?stringtype=unspecified
+SPRING_DATASOURCE_USERNAME=postgres
+SPRING_DATASOURCE_PASSWORD=streampark
+FLINK=flink1.14.5
+FLINK_PROPERTIES=jobmanager.rpc.address: jobmanager
+FLINK_BASE_IMAGE=flink:1.14.5-scala_2.12
+COMMAND=/bin/sh -c "bash bin/startup.sh"
