This is an automated email from the ASF dual-hosted git repository.
roryqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git
The following commit(s) were added to refs/heads/master by this push:
new 3954ce70c [MINOR] fix(pom): Add missing shuffle-server dependencies to
work with -Ptez
3954ce70c is described below
commit 3954ce70c42dfcdc5b65e670db9682c64a3894a0
Author: Enrico Minack <[email protected]>
AuthorDate: Wed Feb 14 03:35:39 2024 +0100
[MINOR] fix(pom): Add missing shuffle-server dependencies to work with -Ptez
### What changes were proposed in this pull request?
Make missing `shuffle-server` dependencies explicit in `server/pom.xml`.
### Why are the changes needed?
The `shuffle-server` module has dependencies that are not explicitly stated
in the `pom.xml`, but transitively available. Notably, these are
`commons-collections` and `hadoop-common`:
https://github.com/apache/incubator-uniffle/blob/866f5ff121948e2144729300a0f5cb03510db29d/server/src/main/java/org/apache/uniffle/server/ShuffleFlushManager.java#L27-L29
Run
```
./mvnw dependency:tree -pl org.apache.uniffle:shuffle-server -am
```
to see the dependency tree:
```
[INFO] -----------------< org.apache.uniffle:shuffle-server
>------------------
[INFO] Building Apache Uniffle Server 0.9.0-SNAPSHOT
[6/6]
[INFO] from server/pom.xml
[INFO] --------------------------------[ jar
]---------------------------------
[INFO] …
[INFO] --- dependency:2.10:tree (default-cli) @ shuffle-server ---
[INFO] org.apache.uniffle:shuffle-server:jar:0.9.0-SNAPSHOT
[INFO] …
[INFO] +- org.apache.hadoop:hadoop-minicluster:jar:2.8.5:test
[INFO] | +- org.apache.hadoop:hadoop-common:test-jar:tests:2.8.5:test
[INFO] …
[INFO] | | +- commons-collections:commons-collections:jar:3.2.2:provided
[INFO] …
[INFO] | +- org.apache.hadoop:hadoop-common:jar:2.8.5:provided
```
Both dependencies are available through
`org.apache.hadoop:hadoop-minicluster` as `provided`.
However, with `-Ptez`, the dependency tree changes in such a way that those
transitive dependency become available only in `test` scope, breaking
compilation of the `shuffle-server` module:
```
./mvnw dependency:tree -pl org.apache.uniffle:shuffle-server -am -Ptez
```
```
[INFO] -----------------< org.apache.uniffle:shuffle-server
>------------------
[INFO] Building Apache Uniffle Server 0.9.0-SNAPSHOT
[6/6]
[INFO] from server/pom.xml
[INFO] --------------------------------[ jar
]---------------------------------
[INFO] …
[INFO] --- dependency:2.10:tree (default-cli) @ shuffle-server ---
[INFO] org.apache.uniffle:shuffle-server:jar:0.9.0-SNAPSHOT
[INFO] …
[INFO] +- org.apache.hadoop:hadoop-minicluster:jar:2.8.5:test
[INFO] …
[INFO] | | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] …
[INFO] | +- org.apache.hadoop:hadoop-common:jar:2.8.5:compile
```
The required dependencies are now in `compile` scope under a `test` scope
dependency. The `mvnw` command line build tool seems robust against this, which
is why CI works just fine. But IntelliJ IDE seems to handle those dependency
trees literally and compilation breaks.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually.
---
pom.xml | 1 +
server/pom.xml | 10 ++++++++++
2 files changed, 11 insertions(+)
diff --git a/pom.xml b/pom.xml
index 82022ca4e..4cbf7567f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -56,6 +56,7 @@
<version.checksum-maven-plugin>1.11</version.checksum-maven-plugin>
<awaitility.version>4.2.0</awaitility.version>
<checkstyle.version>9.3</checkstyle.version>
+ <commons-collections.version>3.2.2</commons-collections.version>
<commons-logging.version>1.2</commons-logging.version>
<commons-lang3.version>3.10</commons-lang3.version>
<commons-codec.version>1.9</commons-codec.version>
diff --git a/server/pom.xml b/server/pom.xml
index d3beb1f1d..4d3dda04f 100644
--- a/server/pom.xml
+++ b/server/pom.xml
@@ -83,6 +83,11 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <scope>${hadoop.scope}</scope>
+ </dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
@@ -113,6 +118,11 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>commons-collections</groupId>
+ <artifactId>commons-collections</artifactId>
+ <version>${commons-collections.version}</version>
+ </dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
