This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 864ff67 [SPARK-35429][CORE] Remove commons-httpclient from Hadoop-3.2
profile due to EOL and CVEs
864ff67 is described below
commit 864ff677469172ca24fdef69b7d3a3482c688f47
Author: Sumeet Gajjar <[email protected]>
AuthorDate: Tue Jun 15 14:43:30 2021 -0700
[SPARK-35429][CORE] Remove commons-httpclient from Hadoop-3.2 profile due
to EOL and CVEs
### What changes were proposed in this pull request?
Remove commons-httpclient as a direct dependency for Hadoop-3.2 profile.
Hadoop-2.7 profile distribution still has it, hadoop-client has a compile
dependency on commons-httpclient, thus we cannot remove it for Hadoop-2.7
profile.
```
[INFO] +- org.apache.hadoop:hadoop-client:jar:2.7.4:compile
[INFO] | +- org.apache.hadoop:hadoop-common:jar:2.7.4:compile
[INFO] | | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | +- commons-httpclient:commons-httpclient:jar:3.1:compile
```
### Why are the changes needed?
Spark is pulling in commons-httpclient as a dependency directly.
commons-httpclient went EOL years ago and there are most likely CVEs not being
reported against it, thus we should remove it.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Existing unittests
- Checked the dependency tree before and after introducing the changes
Before:
```
./build/mvn dependency:tree -Phadoop-3.2 | grep -i "commons-httpclient"
Using `mvn` from path: /usr/bin/mvn
[INFO] +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:provided
```
After
```
./build/mvn dependency:tree | grep -i "commons-httpclient"
Using `mvn` from path:
/Users/sumeet.gajjar/cloudera/upstream-spark/build/apache-maven-3.6.3/bin/mvn
```
P.S. Reopening this since [spark
upgraded](https://github.com/apache/spark/commit/463daabd5afd9abfb8027ebcb2e608f169ad1e40)
its `hive.version` to `2.3.9` which does not have a dependency on
`commons-httpclient`.
Closes #32912 from sumeetgajjar/SPARK-35429.
Authored-by: Sumeet Gajjar <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 1 -
pom.xml | 11 -----------
sql/hive/pom.xml | 4 ----
3 files changed, 16 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
index 8b79d7e5..3482dd2 100644
--- a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
@@ -35,7 +35,6 @@ commons-compiler/3.1.4//commons-compiler-3.1.4.jar
commons-compress/1.20//commons-compress-1.20.jar
commons-crypto/1.1.0//commons-crypto-1.1.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
-commons-httpclient/3.1//commons-httpclient-3.1.jar
commons-io/2.8.0//commons-io-2.8.0.jar
commons-lang/2.6//commons-lang-2.6.jar
commons-lang3/3.12.0//commons-lang3-3.12.0.jar
diff --git a/pom.xml b/pom.xml
index 82a047f..ca038b2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -157,8 +157,6 @@
<!-- org.apache.httpcomponents/httpclient-->
<commons.httpclient.version>4.5.13</commons.httpclient.version>
<commons.httpcore.version>4.4.14</commons.httpcore.version>
- <!-- commons-httpclient/commons-httpclient-->
- <httpclient.classic.version>3.1</httpclient.classic.version>
<commons.math3.version>3.4.1</commons.math3.version>
<!-- managed up from 3.2.1 for SPARK-11652 -->
<commons.collections.version>3.2.2</commons.collections.version>
@@ -593,11 +591,6 @@
<version>${jsr305.version}</version>
</dependency>
<dependency>
- <groupId>commons-httpclient</groupId>
- <artifactId>commons-httpclient</artifactId>
- <version>${httpclient.classic.version}</version>
- </dependency>
- <dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>${commons.httpclient.version}</version>
@@ -1811,10 +1804,6 @@
<artifactId>commons-codec</artifactId>
</exclusion>
<exclusion>
- <groupId>commons-httpclient</groupId>
- <artifactId>commons-httpclient</artifactId>
- </exclusion>
- <exclusion>
<groupId>org.apache.avro</groupId>
<artifactId>avro-mapred</artifactId>
</exclusion>
diff --git a/sql/hive/pom.xml b/sql/hive/pom.xml
index 729d3f4..67a9854 100644
--- a/sql/hive/pom.xml
+++ b/sql/hive/pom.xml
@@ -134,10 +134,6 @@
<artifactId>avro-mapred</artifactId>
</dependency>
<dependency>
- <groupId>commons-httpclient</groupId>
- <artifactId>commons-httpclient</artifactId>
- </dependency>
- <dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</dependency>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
