This is an automated email from the ASF dual-hosted git repository.
kaihsun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new a6d11c1 SUBMARINE-875. Migrate the Integration test from Travis CI to
GitHub Actions
a6d11c1 is described below
commit a6d11c155bc5967c0359129fd335de637676f64b
Author: KUAN-HSUN-LI <[email protected]>
AuthorDate: Fri Jul 2 10:22:58 2021 +0800
SUBMARINE-875. Migrate the Integration test from Travis CI to GitHub Actions
### What is this PR for?
It creates the integration test in GitHub Actions.
* It first builds the submarine and then uploads the binary
* submarine-e2e and submarine-k8s tests will download the binary before
running tests.
* Build docker image with the downloaded binary and install submarine
through helm which is different from the test in Travis CI.
* Failure status will be on if the test fails.
* It takes about 18 mins and 12 mins in submarine-k8s and submarine-k8s
individually.
### What type of PR is it?
[CI/CD]
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-875
### How should this be tested?
See the GitHub Actions.
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No
Author: KUAN-HSUN-LI <[email protected]>
Signed-off-by: Kai-Hsun Chen <[email protected]>
Closes #632 from KUAN-HSUN-LI/SUBMARINE-875 and squashes the following
commits:
4cca1260 [KUAN-HSUN-LI] SUBMARINE-875. Remove maximize-build-space
d11dcb40 [KUAN-HSUN-LI] SUBMARINE-875. Migrate the Integration test from
Travis CI to GitHub Actions
820e167a [KUAN-HSUN-LI] SUBMARINE-875. Migrate the Integration test from
Travis CI to GitHub Actions
b92d96b5 [KUAN-HSUN-LI] SUBMARINE-875. Migrate the Integration test from
Travis CI to GitHub Actions
---
.github/config/kind-config-kind.yaml | 36 ++++++++
.github/scripts/start-submarine.sh | 27 ++++++
.github/workflows/master.yml | 156 +++++++++++++++++++++++++++++-----
dev-support/database/init-database.py | 79 +++++++++++++++++
4 files changed, 276 insertions(+), 22 deletions(-)
diff --git a/.github/config/kind-config-kind.yaml
b/.github/config/kind-config-kind.yaml
new file mode 100644
index 0000000..a6204a3
--- /dev/null
+++ b/.github/config/kind-config-kind.yaml
@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+nodes:
+- role: control-plane
+ kubeadmConfigPatches:
+ - |
+ kind: InitConfiguration
+ nodeRegistration:
+ kubeletExtraArgs:
+ node-labels: "ingress-ready=true"
+ authorization-mode: "AlwaysAllow"
+ extraPortMappings:
+ - containerPort: 32080
+ hostPort: 80
+ protocol: TCP
+ - containerPort: 443
+ hostPort: 443
+ protocol: TCP
\ No newline at end of file
diff --git a/.github/scripts/start-submarine.sh
b/.github/scripts/start-submarine.sh
new file mode 100644
index 0000000..8d6d361
--- /dev/null
+++ b/.github/scripts/start-submarine.sh
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Fix submarine-database start failed in kind.
https://github.com/kubernetes/minikube/issues/7906
+sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
+sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+helm install --wait submarine ./helm-charts/submarine
+kubectl get pods
+kubectl port-forward svc/submarine-database 3306:3306 &
+kubectl port-forward svc/submarine-server 8080:8080 &
+kubectl port-forward svc/submarine-minio-service 9000:9000 &
+kubectl port-forward svc/submarine-mlflow-service 5001:5000 &
\ No newline at end of file
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index 2fed76f..8dfa2af 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -2,6 +2,11 @@ name: Submarine
on: [push, pull_request]
+env:
+ VERSION: "0.6.0-SNAPSHOT"
+ HADOOP_VERSION: "2.9"
+ PROFILE: "-Phadoop-2.9"
+
jobs:
build:
name: Build
@@ -20,35 +25,142 @@ jobs:
run: |
mvn --version
java -version
- - uses: actions/cache@v1
+ - uses: actions/cache@v2
with:
- path: ~/.m2/repository/com
- key: ${{ runner.os }}-maven-com-${{ hashFiles('**/pom.xml') }}
+ path: ~/.m2/repository
+ key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
- ${{ runner.os }}-maven-com-
- - uses: actions/cache@v1
+ ${{ runner.os }}-maven-
+ - name: Build with Maven
+ env:
+ BUILD_FLAG: "clean install -DskipTests -ntp"
+ run: |
+ echo ">>> mvn ${BUILD_FLAG} ${PROFILE} -B"
+ mvn ${BUILD_FLAG} ${PROFILE} -B
+ - name: Store build binary
+ uses: actions/upload-artifact@v2
with:
- path: ~/.m2/repository/org
- key: ${{ runner.os }}-maven-org-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- ${{ runner.os }}-maven-org-
- - uses: actions/cache@v1
+ name: submarine-bin
+ path: submarine-dist/target/submarine-dist*.tar.gz
+ submarine-e2e:
+ runs-on: ubuntu-latest
+ needs: [build]
+ services:
+ mysql:
+ image: mysql:5.7
+ env:
+ MYSQL_ROOT_PASSWORD: "password"
+ ports:
+ - 3306:3306
+ # wait until mysql is health
+ options: --health-cmd "mysqladmin ping" --health-interval 10s
--health-timeout 10s --health-retries 10
+ steps:
+ - uses: actions/checkout@v2
with:
- path: ~/.m2/repository/net
- key: ${{ runner.os }}-maven-net-${{ hashFiles('**/pom.xml') }}
- restore-keys: |
- ${{ runner.os }}-maven-net-
- - uses: actions/cache@v1
+ fetch-depth: 50
+ - name: Set up JDK 1.8
+ uses: actions/setup-java@v1
with:
- path: ~/.m2/repository/io
- key: ${{ runner.os }}-maven-io-${{ hashFiles('**/pom.xml') }}
+ java-version: "1.8"
+ - name: Set up Maven 3.6.3
+ uses: stCarolas/setup-maven@v4
+ with:
+ maven-version: 3.6.3
+ - name: Check version
+ run: |
+ mvn --version
+ java -version
+ chromedriver --version
+ python3 -V
+ - name: Prepate default mysql-data
+ run: |
+ pip install mysql-connector-python
+ python3 ./dev-support/database/init-database.py
+ - name: Download compiled Submarine binaries
+ uses: actions/download-artifact@v2
+ with:
+ name: submarine-bin
+ - name: Untar binaries
+ run: |
+ mkdir -p
submarine-dist/target/submarine-dist-${VERSION}-hadoop-${HADOOP_VERSION}
+ tar zxvf submarine-dist*.tar.gz -C
submarine-dist/target/submarine-dist-${VERSION}-hadoop-${HADOOP_VERSION}
+ mv submarine-dist*.tar.gz submarine-dist/target/
+ sudo chmod -R a+rwX
submarine-dist/target/submarine-dist-${VERSION}-hadoop-${HADOOP_VERSION}
+ - name: Test
+ env:
+ TEST_FLAG: "verify -DskipRat -ntp -am"
+ TEST_MODULES: "-pl org.apache.submarine:submarine-test-e2e"
+ run: |
+ echo ">>> mvn ${TEST_FLAG} ${TEST_MODULES} ${PROFILE} -B"
+ mvn ${TEST_FLAG} ${TEST_MODULES} ${PROFILE} -B
+ submarine-k8s:
+ runs-on: ubuntu-latest
+ needs: [build]
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ fetch-depth: 50
+ - name: Set up JDK 1.8
+ uses: actions/setup-java@v1
+ with:
+ java-version: "1.8"
+ - name: Set up Maven 3.6.3
+ uses: stCarolas/setup-maven@v4
+ with:
+ maven-version: 3.6.3
+ - uses: actions/cache@v2
+ with:
+ path: ~/.m2/repository
+ key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
- ${{ runner.os }}-maven-io-
- - name: Build with Maven
+ ${{ runner.os }}-maven-
+ - name: Check version
run: |
- export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m
-Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
- mvn --no-transfer-progress -DskipTests install
-
+ mvn --version
+ java -version
+ helm version
+ kind version
+ - name: Create kind cluster
+ run: |
+ kind create cluster --config ./.github/config/kind-config-kind.yaml
--wait 3m --image kindest/node:v1.15.12
+ - name: Show K8s cluster information
+ run: |
+ kubectl cluster-info
+ kubectl version
+ kubectl get pods -n kube-system
+ export KUBECONFIG=~/.kube/kind-config-kind
+ kind export kubeconfig --kubeconfig ${KUBECONFIG}
+ echo "current-context:" $(kubectl config current-context)
+ echo "environment-kubeconfig:" ${KUBECONFIG}
+ - name: Download compiled Submarine binaries
+ uses: actions/download-artifact@v2
+ with:
+ name: submarine-bin
+ - name: Untar binaries
+ run: |
+ mkdir -p
submarine-dist/target/submarine-dist-${VERSION}-hadoop-${HADOOP_VERSION}
+ tar zxvf submarine-dist*.tar.gz -C
submarine-dist/target/submarine-dist-${VERSION}-hadoop-${HADOOP_VERSION}
+ mv submarine-dist*.tar.gz submarine-dist/target/
+ sudo chmod -R a+rwX
submarine-dist/target/submarine-dist-${VERSION}-hadoop-${HADOOP_VERSION}
+ - name: Build Image locally
+ run: |
+ .github/scripts/build-image-locally.sh
+ - name: Start submarine
+ run: bash ./.github/scripts/start-submarine.sh
+ - name: Test
+ env:
+ TEST_FLAG: "verify -DskipRat -ntp -am -Durl=http://127.0.0.1";
+ TEST_MODULES: "-pl
!:submarine-server-api,!:submarine-server-core,!:submarine-server-rpc,!:submarine-commons-cluster,!:submarine-commons-metastore,!:submarine-commons-rpc,!:submarine-commons-runtime,!:submarine-commons-utils,!:submarine-client,:submarine-test-k8s"
+ TEST_PROJECTS: ""
+ run: |
+ echo ">>> mvn ${TEST_FLAG} ${TEST_MODULES} ${PROFILE} -B"
+ mvn ${TEST_FLAG} ${TEST_MODULES} ${PROFILE} -B
+ - name: Failure status
+ run: |
+ kubectl get pods
+ kubectl -n default get events --sort-by='{.lastTimestamp}'
+ kubectl describe nodes
+ if: ${{ failure() }}
rat:
name: Check License
runs-on: ubuntu-latest
diff --git a/dev-support/database/init-database.py
b/dev-support/database/init-database.py
new file mode 100644
index 0000000..f98d509
--- /dev/null
+++ b/dev-support/database/init-database.py
@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import mysql.connector
+
+conn = mysql.connector.connect(
+ user='root', password='password', host='127.0.0.1')
+
+cursor = conn.cursor(buffered=True)
+
+assert conn.is_connected(), "mysql is not connected"
+
+
+def commit(sql):
+ try:
+ # Executing the SQL command
+ cursor.execute(sql)
+ if cursor.with_rows:
+ print(cursor.fetchall())
+ # Commit your changes in the database
+ conn.commit()
+
+ except:
+ # Rolling back in case of error
+ conn.rollback()
+
+def commit_from_file(file_path):
+ with open(file_path) as f:
+ for result in cursor.execute(f.read(), multi=True):
+ if result.with_rows:
+ print(result.fetchall())
+
+
+commit("CREATE DATABASE IF NOT EXISTS submarine_test;")
+commit("CREATE USER IF NOT EXISTS 'submarine_test'@'%' IDENTIFIED BY
'password_test';")
+commit("GRANT ALL PRIVILEGES ON *.* TO 'submarine_test'@'%';")
+commit("use submarine_test;")
+commit_from_file("./dev-support/database/submarine.sql")
+commit("show tables;")
+
+
+commit("CREATE DATABASE IF NOT EXISTS metastore_test;")
+commit("CREATE USER IF NOT EXISTS 'metastore_test'@'%' IDENTIFIED BY
'password_test';")
+commit("GRANT ALL PRIVILEGES ON *.* TO 'metastore_test'@'%';")
+commit("use metastore_test;")
+commit_from_file("./dev-support/database/metastore.sql")
+commit("show tables;")
+
+
+commit("CREATE DATABASE IF NOT EXISTS submarine;")
+commit("CREATE USER IF NOT EXISTS 'submarine'@'%' IDENTIFIED BY 'password';")
+commit("GRANT ALL PRIVILEGES ON *.* TO 'submarine'@'%';")
+commit("use submarine;")
+commit_from_file("./dev-support/database/submarine.sql")
+commit_from_file("./dev-support/database/submarine-data.sql")
+commit("show tables;")
+
+
+commit("CREATE DATABASE IF NOT EXISTS metastore;")
+commit("CREATE USER IF NOT EXISTS 'metastore'@'%' IDENTIFIED BY 'password';")
+commit("GRANT ALL PRIVILEGES ON *.* TO 'metastore'@'%';")
+commit("use metastore;")
+commit_from_file("./dev-support/database/metastore.sql")
+commit("show tables;")
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
