This is an automated email from the ASF dual-hosted git repository.
liuxun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 4d8c814 SUBMARINE-672. Remove unused dependencies from pom.xml
4d8c814 is described below
commit 4d8c814d31850b6537c43ec997a6092cd28447e6
Author: Kevin Su <[email protected]>
AuthorDate: Thu Nov 5 08:32:02 2020 +0800
SUBMARINE-672. Remove unused dependencies from pom.xml
### What is this PR for?
The final submarine binary tar file is too big (8xx MB), because there are
some unused dependencies in submarine-dist/lib/*.
Remove unused dependencies from the pom.xml, and some dependencies don't
need to be packaged to distribution.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-672
### How should this be tested?
* First time? Setup Travis CI as described on
https://submarine.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed
behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
```
➜ lib git:(SUBMARINE-672) ✗ pwd
/home/kobe/git/submarine/submarine-dist/target/submarine-dist-0.5.0-hadoop-2.9/submarine-dist-0.5.0-hadoop-2.9/lib
➜ lib git:(SUBMARINE-672) ✗ du -h .
56M ./submitter/yarn
72K ./submitter/k8s
56M ./submitter
52M ./interpreter/python
291M ./interpreter/spark
342M ./interpreter
615M .
```
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation?No
Author: Kevin Su <[email protected]>
Closes #446 from pingsutw/SUBMARINE-672 and squashes the following commits:
ddaa021 [Kevin Su] Update travis
7503133 [Kevin Su] update
9e9be9e [Kevin Su] update
---
.travis.yml | 41 ++++++++++++-----------
submarine-dist/src/assembly/distribution.xml | 42 ++++++++++++------------
submarine-dist/src/assembly/src-distribution.xml | 2 ++
submarine-workbench/pom.xml | 2 +-
4 files changed, 45 insertions(+), 42 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 2cf5942..3e08a27 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -47,7 +47,7 @@ env:
- EXCLUDE_CLIENT="!:submarine-client"
- EXCLUDE_ALL="!:submarine-all"
- EXCLUDE_WORKBENCH="!:submarine-workbench,!:submarine-workbench-web"
- -
EXCLUDE_INTERPRETER="!:submarine-interpreter,!:submarine-interpreter-core,!:submarine-python-interpreter,!:submarine-spark-interpreter"
+ # -
EXCLUDE_INTERPRETER="!:submarine-interpreter,!:submarine-interpreter-core,!:submarine-python-interpreter,!:submarine-spark-interpreter"
- EXCLUDE_SUBMITTER_K8S="!:submarine-submitter-k8s"
- EXCLUDE_SUBMITTER_YARN="!:submarine-submitter-yarn"
-
EXCLUDE_SUBMITTER="!:submarine-server-submitter,${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_SUBMITTER_YARN}"
@@ -112,7 +112,7 @@ matrix:
- PROFILE="-Phadoop-2.9"
- BUILD_FLAG="clean package install -DskipTests"
- TEST_FLAG="test -DskipRat -am"
- - MODULES="-pl
${EXCLUDE_COMMONS},${EXCLUDE_SUBMITTER},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLIENT},${EXCLUDE_CLOUD},${EXCLUDE_SERVER},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
+ - MODULES="-pl
${EXCLUDE_COMMONS},${EXCLUDE_SUBMITTER},${EXCLUDE_WORKBENCH},${EXCLUDE_CLIENT},${EXCLUDE_CLOUD},${EXCLUDE_SERVER},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
- TEST_MODULES="-pl org.apache.submarine:submarine-commons-cluster"
- TEST_PROJECTS=""
@@ -124,7 +124,7 @@ matrix:
- PROFILE="-Phadoop-2.9"
- BUILD_FLAG="clean package install -DskipTests"
- TEST_FLAG="test -DskipRat -am"
- - MODULES="-pl
${EXCLUDE_COMMONS},${EXCLUDE_SUBMITTER},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLIENT},${EXCLUDE_CLOUD},${EXCLUDE_SERVER},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
+ - MODULES="-pl
${EXCLUDE_COMMONS},${EXCLUDE_SUBMITTER},${EXCLUDE_WORKBENCH},${EXCLUDE_CLIENT},${EXCLUDE_CLOUD},${EXCLUDE_SERVER},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
- TEST_MODULES="-pl org.apache.submarine:submarine-commons-metastore"
- TEST_PROJECTS=""
@@ -136,7 +136,7 @@ matrix:
- PROFILE="-Phadoop-2.9"
- BUILD_FLAG="clean package install -DskipTests"
- TEST_FLAG="test -DskipRat -am"
- - MODULES="-pl
${EXCLUDE_COMMONS},${EXCLUDE_SUBMITTER},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLIENT},${EXCLUDE_CLOUD},${EXCLUDE_SERVER},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
+ - MODULES="-pl
${EXCLUDE_COMMONS},${EXCLUDE_SUBMITTER},${EXCLUDE_WORKBENCH},${EXCLUDE_CLIENT},${EXCLUDE_CLOUD},${EXCLUDE_SERVER},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
- TEST_MODULES="-pl org.apache.submarine:submarine-commons-rpc"
- TEST_PROJECTS=""
@@ -148,7 +148,7 @@ matrix:
- PROFILE="-Phadoop-2.9"
- BUILD_FLAG="clean package install -DskipTests"
- TEST_FLAG="test -DskipRat -am"
- - MODULES="-pl
${EXCLUDE_COMMONS},${EXCLUDE_SUBMITTER},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLIENT},${EXCLUDE_CLOUD},${EXCLUDE_SERVER},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
+ - MODULES="-pl
${EXCLUDE_COMMONS},${EXCLUDE_SUBMITTER},${EXCLUDE_WORKBENCH},${EXCLUDE_CLIENT},${EXCLUDE_CLOUD},${EXCLUDE_SERVER},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
- TEST_MODULES="-pl org.apache.submarine:submarine-commons-runtime"
- TEST_PROJECTS=""
@@ -160,7 +160,7 @@ matrix:
- PROFILE="-Phadoop-2.9"
- BUILD_FLAG="clean package install -DskipTests"
- TEST_FLAG="test -DskipRat -am"
- - MODULES="-pl
${EXCLUDE_COMMONS},${EXCLUDE_SUBMITTER},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLIENT},${EXCLUDE_CLOUD},${EXCLUDE_SERVER},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
+ - MODULES="-pl
${EXCLUDE_COMMONS},${EXCLUDE_SUBMITTER},${EXCLUDE_WORKBENCH},${EXCLUDE_CLIENT},${EXCLUDE_CLOUD},${EXCLUDE_SERVER},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
- TEST_MODULES="-pl
org.apache.submarine:submarine-commons-unixusersync"
- TEST_PROJECTS=""
@@ -172,7 +172,7 @@ matrix:
- PROFILE="-Phadoop-2.9"
- BUILD_FLAG="clean package install -DskipTests"
- TEST_FLAG="test -DskipRat -am"
- - MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
+ - MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_CLOUD},${EXCLUDE_ALL},${EXCLUDE_DIST},${EXCLUDE_TEST}"
- TEST_MODULES="-pl
${EXCLUDE_COMMONS},org.apache.submarine:submarine-server-core"
- TEST_PROJECTS=""
@@ -184,8 +184,8 @@ matrix:
- PROFILE="-Phadoop-2.9"
- BUILD_FLAG="clean package install -DskipTests -DskipRat"
- TEST_FLAG="test -DskipRat -am"
- - MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_DIST}"
- - TEST_MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_COMMONS},${EXCLUDE_DIST},${EXCLUDE_TEST},${EXCLUDE_ALL},${EXCLUDE_SERVER},${EXCLUDE_SPARK_SECURTITY}"
+ - MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_CLOUD},${EXCLUDE_DIST}"
+ - TEST_MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_CLOUD},${EXCLUDE_COMMONS},${EXCLUDE_DIST},${EXCLUDE_TEST},${EXCLUDE_ALL},${EXCLUDE_SERVER},${EXCLUDE_SPARK_SECURTITY}"
- TEST_PROJECTS=""
- name: Test submarine submitter on hadoop-2.10
@@ -196,8 +196,8 @@ matrix:
- PROFILE="-Phadoop-2.10"
- BUILD_FLAG="clean package install -DskipTests -DskipRat"
- TEST_FLAG="test -DskipRat -am"
- - MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_DIST}"
- - TEST_MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_COMMONS},${EXCLUDE_DIST},${EXCLUDE_TEST},${EXCLUDE_ALL},${EXCLUDE_SERVER},${EXCLUDE_SPARK_SECURTITY}"
+ - MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_CLOUD},${EXCLUDE_DIST}"
+ - TEST_MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_CLOUD},${EXCLUDE_COMMONS},${EXCLUDE_DIST},${EXCLUDE_TEST},${EXCLUDE_ALL},${EXCLUDE_SERVER},${EXCLUDE_SPARK_SECURTITY}"
- TEST_PROJECTS=""
- name: Test submarine submitter on hadoop-3.1
@@ -208,8 +208,8 @@ matrix:
- PROFILE="-Phadoop-3.1"
- BUILD_FLAG="clean package install -DskipTests -DskipRat"
- TEST_FLAG="test -DskipRat -am"
- - MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_DIST}"
- - TEST_MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_COMMONS},${EXCLUDE_DIST},${EXCLUDE_TEST},${EXCLUDE_ALL},${EXCLUDE_SERVER},${EXCLUDE_SPARK_SECURTITY}"
+ - MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_CLOUD},${EXCLUDE_DIST}"
+ - TEST_MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_CLOUD},${EXCLUDE_COMMONS},${EXCLUDE_DIST},${EXCLUDE_TEST},${EXCLUDE_ALL},${EXCLUDE_SERVER},${EXCLUDE_SPARK_SECURTITY}"
- TEST_PROJECTS=""
- name: Test submarine submitter on hadoop-3.2
@@ -220,15 +220,16 @@ matrix:
- PROFILE="-Phadoop-3.2"
- BUILD_FLAG="clean package install -DskipTests -DskipRat"
- TEST_FLAG="test -DskipRat -am"
- - MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_DIST}"
- - TEST_MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_COMMONS},${EXCLUDE_DIST},${EXCLUDE_TEST},${EXCLUDE_ALL},${EXCLUDE_SERVER},${EXCLUDE_SPARK_SECURTITY}"
+ - MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_CLOUD},${EXCLUDE_DIST}"
+ - TEST_MODULES="-pl
${EXCLUDE_SUBMITTER_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_CLOUD},${EXCLUDE_COMMONS},${EXCLUDE_DIST},${EXCLUDE_TEST},${EXCLUDE_ALL},${EXCLUDE_SERVER},${EXCLUDE_SPARK_SECURTITY}"
- TEST_PROJECTS=""
- - name: Test submarine interpreter
- language: java
- jdk: "openjdk8"
- dist: xenial
- env: PYTHON="3" PROFILE="-Phadoop-2.9" BUILD_FLAG="clean package install
-DskipTests -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl $(echo
${EXCLUDE_INTERPRETER} | sed 's/!//g')" TEST_MODULES="-pl $(echo
${EXCLUDE_INTERPRETER} | sed 's/!//g')" TEST_PROJECTS=""
+# Temporary disable this test, refer to SUBMARINE-672
+# - name: Test submarine interpreter
+# language: java
+# jdk: "openjdk8"
+# dist: xenial
+# env: PYTHON="3" PROFILE="-Phadoop-2.9" BUILD_FLAG="clean package
install -DskipTests -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl $(echo
${EXCLUDE_INTERPRETER} | sed 's/!//g')" TEST_MODULES="-pl $(echo
${EXCLUDE_INTERPRETER} | sed 's/!//g')" TEST_PROJECTS=""
- name: Test submarine workbench-web Angular
language: node_js
diff --git a/submarine-dist/src/assembly/distribution.xml
b/submarine-dist/src/assembly/distribution.xml
index 8a54573..3e2d56e 100644
--- a/submarine-dist/src/assembly/distribution.xml
+++ b/submarine-dist/src/assembly/distribution.xml
@@ -150,13 +150,13 @@
<include>submarine-submitter-yarn-${project.version}-shade.jar</include>
</includes>
</fileSet>
- <fileSet>
-
<directory>../submarine-server/server-submitter/submitter-yarnservice/target</directory>
- <outputDirectory>/lib/submitter/yarnservice</outputDirectory>
- <includes>
-
<include>submarine-submitter-yarnservice-${project.version}.jar</include>
- </includes>
- </fileSet>
+<!-- <fileSet>-->
+<!--
<directory>../submarine-server/server-submitter/submitter-yarnservice/target</directory>-->
+<!-- <outputDirectory>/lib/submitter/yarnservice</outputDirectory>-->
+<!-- <includes>-->
+<!--
<include>submarine-submitter-yarnservice-${project.version}.jar</include>-->
+<!-- </includes>-->
+<!-- </fileSet>-->
<fileSet>
<directory>../submarine-client/target</directory>
<outputDirectory>/lib</outputDirectory>
@@ -187,20 +187,20 @@
<exclude>netty-*-4.1.27.Final*.jar</exclude>
</excludes>
</fileSet>
- <fileSet>
-
<directory>../submarine-workbench/interpreter/python-interpreter/target</directory>
- <outputDirectory>/lib/interpreter/python</outputDirectory>
- <includes>
-
<include>submarine-python-interpreter-${project.version}-shade.jar</include>
- </includes>
- </fileSet>
- <fileSet>
-
<directory>../submarine-workbench/interpreter/spark-interpreter/target</directory>
- <outputDirectory>/lib/interpreter/spark</outputDirectory>
- <includes>
-
<include>submarine-spark-interpreter-${project.version}-shade.jar</include>
- </includes>
- </fileSet>
+<!-- <fileSet>-->
+<!--
<directory>../submarine-workbench/interpreter/python-interpreter/target</directory>-->
+<!-- <outputDirectory>/lib/interpreter/python</outputDirectory>-->
+<!-- <includes>-->
+<!--
<include>submarine-python-interpreter-${project.version}-shade.jar</include>-->
+<!-- </includes>-->
+<!-- </fileSet>-->
+<!-- <fileSet>-->
+<!--
<directory>../submarine-workbench/interpreter/spark-interpreter/target</directory>-->
+<!-- <outputDirectory>/lib/interpreter/spark</outputDirectory>-->
+<!-- <includes>-->
+<!--
<include>submarine-spark-interpreter-${project.version}-shade.jar</include>-->
+<!-- </includes>-->
+<!-- </fileSet>-->
</fileSets>
</assembly>
diff --git a/submarine-dist/src/assembly/src-distribution.xml
b/submarine-dist/src/assembly/src-distribution.xml
index 94852a5..c3e2a89 100644
--- a/submarine-dist/src/assembly/src-distribution.xml
+++ b/submarine-dist/src/assembly/src-distribution.xml
@@ -44,6 +44,8 @@
<exclude>**/build/**</exclude>
<exclude>**/file:/**</exclude>
<exclude>**/SecurityAuth.audit*</exclude>
+ <exclude>submarine-cloud/bin/**</exclude>
+
<exclude>dev-support/submarine-installer/package/hadoop/yarn/lib/native/**</exclude>
</excludes>
</fileSet>
</fileSets>
diff --git a/submarine-workbench/pom.xml b/submarine-workbench/pom.xml
index 1bbfdd9..5ee7074 100644
--- a/submarine-workbench/pom.xml
+++ b/submarine-workbench/pom.xml
@@ -37,7 +37,7 @@
<description>Submarine Workbench</description>
<modules>
- <module>interpreter</module>
+<!-- <module>interpreter</module>-->
<module>workbench-web</module>
</modules>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
